Pre-testing of Evaluation Surveys
Information Collection Request
0970 - 0355
Supporting Statement
Part A
January 2015
Updated March 2015
Submitted By:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
7th Floor, West Aerospace Building
370 L’Enfant Promenade, SW
Washington, D.C. 20447
A1. Necessity for the Data Collection
The Office of Planning, Research and Evaluation (OPRE), in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks renewal of this generic survey development clearance to allow us to use samples of more than nine participants in applying methods useful for identifying questionnaire and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions.
OPRE studies ACF programs, and the populations they serve, through rigorous research and evaluation projects. These include evaluations of existing programs, evaluations of innovative approaches to helping low-income children and families, research syntheses and descriptive and exploratory studies. This generic clearance allows us to identify if and when a survey may be simplified for respondents, respondent burden may be reduced, among other possible improvements. The research completed under this generic is intended to be informative in nature; the studies may be iterative, as variation in questions or procedures are proposed, evaluated, and retested. The core methodology and target populations will be consistent and burden caps and incentive structure are proposed in this request. The pretesting of surveys will inform future ACF/OPRE information collections, resulting in higher quality studies.
This generic clearance (0970-0355) was originally approved for use in October 2008 and renewed in January 2012. During the first three years of use, information was collected from 3,605 respondents, for 1,336 hours of burden. Since January 2012, ACF/OPRE received approval to collect information from 3395 respondents, for 1742 hours of burden. The increased use is indicative of how pretesting surveys has improved data collection efforts on behalf of the ACF Office of Planning, Research and Evaluation. For a list of approved generic clearances since January 2012, see Attachment B.
In the past, ACF has received approval for seven types of activities: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests.
Example ACF projects that have benefited from this generic IC include survey activities as part of the Head Start Family and Child Experiences Survey (FACES), and the National Survey of Early Care and Education, among others.
ACF is undertaking the collection at the discretion of the agency.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Approach
ACF/OPRE will use the information collected to develop and test questionnaires and procedures to evaluate and improve the quality of the data gathered through ACF/OPRE’s research and evaluation studies. Evaluation of questionnaires and/or procedures is the main objective of the activities in this clearance. The goal of developing these surveys and refining procedures is to improve the evaluations of ACF programs and demonstrations. ACF/OPRE will use the results internally to inform subsequent information collection requests. Results of these methodological studies may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications describing findings of methodological interest. When necessary, results will be labeled as exploratory in nature. The data collected will not be presented as findings on the substantive topic under study.
All of the methods proposed for use in questionnaire development will be used with either purposive or statistically representative samples of participants in ACF programs being evaluated; participants in ACF demonstrations, many of which are supported by ACF program grants; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services. All information collection activities conducted under this generic clearance will be voluntary and low-burden.
Study Design
ACF will test a variety of instruments and procedures under this clearance. The exact nature of the surveys and the samples is currently undetermined and details will be provided for each individual information collection request. The particular samples included in future generic information collection requests will vary based on the content of the survey being tested. The proposed methods for use under this clearance include the following:
Cognitive and Usability Laboratory and Field Techniques: A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted one-on-one with respondents. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand the physical features of a survey, for instance, its display and navigational features. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. ACF has used cognitive interviews in previous generic clearance projects to revise questionnaires and has broadened the methodology request at OMB’s suggestion in 2012.
Behavior Coding: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.
Exploratory Interviews: A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign.
Respondent debriefing questionnaires: In this method, standardized debriefing questionnaires are administered to respondents who participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.
Split sample experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 100 persons in a split sample test.
Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.
Pilot Studies/Pretests: These methodologies are used to test a preliminary version of the data collection instrument. Pretests are used to gather data to refine questionnaire items and scales and assess reliability or validity. Pilot studies are also used to test aspects of implementation procedures in addition to testing survey measurement issues. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.
Procedures for Clearance
Since the types of surveys included under the umbrella of the clearance are so varied, we cannot specify at this point what kinds of activities would be involved in any particular test. At a minimum, however, one of the types of testing described above would be incorporated into the development testing program for each generic information collection (IC). With each generic IC, we will provide OMB with a copy of questionnaires, supplementary materials, and a brief justification package in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. When iterative testing is conducted, initial instruments will be submitted for review and approval and any revised materials will be uploaded to ROCIS as a nonsubstantive change between each round of testing. A memo will detail any changes.
ACF understands that OMB will make every effort to review materials for individual generic information collection requests within 10 working days of submission. All information gathered from these testing activities will be for the purpose of improving data collection instruments, procedures, not for the purpose of generating findings on the substantive topic under study. ACF will make separate information collection requests for full, non-developmental data collection efforts.
ACF will provide a report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance with subsequent overarching generic information collection renewals. Attachment B provides an overview of ACF/OPRE’s use of this generic information collection between January 2012 and the submission date of this renewal request.
A3. Improved Information Technology to Reduce Burden
ACF/OPRE and its contractors will employ information technology as appropriate to reduce the burden of respondents who agree to participate in its research. We will provide specific information about the use of technology for each individual generic information collection request.
A4. Efforts to Identify Duplication
This research does not duplicate any other questionnaire design work being done by ACF. The purpose of this clearance is to better inform and improve the quality of ACF’s research and evaluation. Pre-testing of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pre-testing plan. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.
A5. Involvement of Small Organizations
The information collection activities proposed under this clearance are not expected to impact small organizations. If an individual collection involves small organizations, the justification package will include a discussion to address this involvement.
A6. Consequences of Less Frequent Data Collection
This clearance involves questionnaire development activities for each survey that is connected with the clearance. This may include one-time collections or iterative testing, based on the specific situation. In all cases, without the proposed information collection activities, the quality of the data collected for our studies would suffer.
A7. Special Circumstances
There are no special circumstances for the proposed data collection efforts.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on September 15, 2014; Volume 79, Number 178, page 54985, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment C. During the notice and comment period, no comments were) received.
Consultation with staff from ACF contractors carrying out research and evaluation surveys will occur in preparation for and in conjunction with the fielding of these data collections under this request. Information about consultation for individual development activities will be provided in individual information collection requests.
A9. Incentives for Respondents
Gifts of appreciation may be provided when appropriate to respondents for activities conducted under this clearance. The type and amount will depend on the types of data collection and the specific population contacted. Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance may receive a small gift of appreciation. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by other Federal cognitive laboratories. For participation in a cognitive interview participants may receive up to $40, and for participation in a focus group it is up to $75 unless otherwise specifically justified. Respondents for methods that are relatively low in burden will not receive a gift in appreciation unless there are extenuating circumstances that warrant it, in which case this will be discussed in the individual justification package.
Not all individual information collections under this generic clearance will provide incentives. If an incentive is proposed, a detailed justification based on the type of collection, population of respondents, and other circumstances will be provided in the individual information collection request.
A10. Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
Individual statements will be included with each generic information collection request submitted under this clearance, but in general, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor will ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. If evaluation staff must sign privacy pledges, these will be referenced in the individual information collection requests.
As necessary, contractors shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
A11. Sensitive Questions
Most of the questions included in these pre-testing activities will not be of a sensitive nature. However, it is possible that some potentially sensitive questions may be included in questionnaires tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered. Information about and justification for any sensitive questions will be included in the justification statement for each individual generic information collection request.
A12. Estimation of Information Collection Burden
Previously Approved Information Collections
The last revision of the generic clearance was approved for 2000 total burden hours.
Newly Requested Information Collections
The estimated burden for this renewal is based on previous experience, consultation with research contractors with whom we have partnered and also on advice from the OMB Office of Information and Regulatory Affairs. A variety of instruments and procedures will be used in conducting the research under this clearance, and the exact time for a respondent to complete a collection will vary. We request approval of 3,825 total burden hours, which is a slight increase from the previous approval. The estimates provided are based primarily on the use between 2012-2015. For more information about past use, see Attachment B.
Total Burden Requested Under this Information Collection
Instrument |
Total Number Respondents |
Annual Number Respondents |
Number Responses Per Respondent |
Average Burden Hours Per Response |
Total Burden hours |
Annual Burden Hours |
Hourly Wage Rate |
Annualized Hourly Costs |
Survey development field tests, respondent debriefing questionnaires, cognitive interviews, split sample experiments, focus groups |
5100 |
1700 |
1 |
.75 |
3,825 |
1,275 |
$16.64 |
$52,33,280 |
To calculate the annualized cost to respondents for the hour burden, we assume an average household income of $36,568, or 200 percent of the poverty threshold of $18,284 for a family of three1. OPRE projects are expected to study low-income populations. This figure translates to an hourly rate of $17.51.
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
Although we cannot anticipate the actual number of participants, length of interview,
and/or mode of data collection for the surveys to be conducted under this clearance, we estimate cost to the Federal Government based on costs incurred between January 2012 and submission of this current request.
Based on previous costs, we estimate the annual costs to the Federal Government to be around $175,000. Costs will be covered by the individual research and evaluation projects, from their data collection budgets. These costs will be described in individual information collection requests.
A15. Change in Burden
This request is to renew the use of the ACF/OPRE generic clearance for another three years. Based on actual and desired use over the past three year clearance period, ACF requests burden level to increase, as compared to the previously approved burden under 0970-0355.
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
This clearance is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the primary subject of any published ACF reports; however, information may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications describing findings of methodological interest. When necessary, results will be labeled as exploratory in nature. The results of this pre-testing research may be prepared for presentation at professional meetings or publication in professional journals.
Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.
ACF/OPRE will develop individual timelines for projects involving generic clearances based on an understanding that OMB/OIRA will review within 10 working days of receiving the information collection request.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 As estimated by the US Census Bureau in “Poverty Thresholds for 2012 by Size of Family…” http://www.census.gov/hhes/www/poverty/data/incpovhlth/2012/index.html
| File Type | application/msword |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| Last Modified By | Molly |
| File Modified | 2015-03-23 |
| File Created | 2015-03-23 |