Pre-testing of Evaluation Data Collection Activities
Information Collection Request
0970 - 0355
Supporting Statement
Part A
February 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
330 C Street, SW, 4th Floor
Washington, DC 20201
The goal of this generic information collection request (ICR) is to develop and test information collection instruments and procedures.
Intended use of the resulting data is to evaluate and improve the quality of the data gathered through ACF/OPRE’s research and evaluation studies.
These pretesting and piloting efforts will collect data using well established methodologies, including: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests.
The populations to be studied include participants in ACF programs being evaluated; participants in ACF demonstrations; recipients of ACF grants and individuals served by ACF grantees; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services.
Data will be analyzed using well established methods, such as data tabulations to evaluate the results of instrument testing.
A1. Necessity for the Data Collection
The Office of Planning, Research, and Evaluation (OPRE), in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks renewal of this generic survey development clearance to allow us to use samples of more than nine participants in applying methods useful for identifying instrument and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions.
OPRE studies ACF programs, and the populations they serve, through rigorous research and evaluation projects. These include evaluations of existing programs, evaluations of innovative approaches to helping low-income children and families, research syntheses and descriptive and exploratory studies. This generic clearance allows us to identify if and when an instrument may be simplified for respondents, respondent burden may be reduced, among other possible improvements. The research completed under this generic is intended to be informative in nature; the studies may be iterative, as variation in questions or procedures are proposed, evaluated, and retested. The core methodology and target populations will be consistent and burden caps and incentive structure are proposed in this request. The pretesting of instruments will inform future ACF/OPRE information collections, resulting in higher quality studies.
This generic clearance (0970-0355) was originally approved for use in October 2008 and renewed in January 2012. During the first three years of use, information was collected from 3,605 respondents, for 1,336 hours of burden. Between January 2012 and March 2015, ACF/OPRE received approval to collect information from 3395 respondents, for 1742 hours of burden. The increased use is indicative of how pretesting surveys has improved data collection efforts on behalf of the ACF Office of Planning, Research and Evaluation. Since approval in March 2015, information has been collected from 1573 respondents for 1178 burden hours. For a list of approved generic clearances since March 2015, see Attachment A.
In the past, ACF has received approval for seven types of activities: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests. We expect future activities to fall within these or similar categories.
Example ACF projects that have benefited from this GenIC include activities as part of the Head Start Family and Child Experiences Survey (FACES), and the National Survey of Early Care and Education, among others. See Reginfo.gov (https://www.reginfo.gov/public/do/PRAOMBHistory?ombControlNumber=0970-0355) for examples of instruments previously approved under this generic clearance.
ACF is undertaking the collections at the discretion of the agency.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Approach
ACF/OPRE will use the information collected to develop and test data collection instruments and procedures to evaluate and improve the quality of the data gathered through ACF/OPRE’s research and evaluation studies. Evaluation of information collection instruments and/or procedures is the main objective of the activities in this clearance. The goal of developing these instruments and refining procedures is to improve the evaluations of ACF programs and demonstrations. ACF/OPRE will use the results internally to inform subsequent information collection requests. Results of these methodological studies may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. When necessary, results will be labeled as exploratory in nature. The data collected will not be presented as findings on the substantive topic under study.
All of the methods proposed for instrument and procedure development will be used with either purposive or statistically representative samples of participants in ACF programs being evaluated; participants in ACF demonstrations, many of which are supported by ACF program grants; recipients of ACF grants and individuals served by ACF grantees; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services. All information collection activities conducted under this GenIC will be voluntary and low-burden.
Study Design
ACF will test a variety of instruments and procedures under this clearance. The exact nature of the instruments and the samples is dependent on each individual project and details will be provided for each GenIC request. The particular samples included in future generic information collection requests will vary based on the content of the instrument being tested. The proposed methods for use under this clearance include the following:
Cognitive and Usability Laboratory and Field Techniques: A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted one-on-one with respondents. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand the physical features of a survey, for instance, its display and navigational features. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. ACF broadened the methodology request to include cognitive interviews at OMB’s suggestion in 2012.
Behavior Coding: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.
Exploratory Interviews: A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign.
Respondent debriefing questionnaires: In this method, standardized debriefing questionnaires are administered to respondents who participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.
Split sample experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 100 persons in a split sample test.
Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.
Pilot Studies/Pretests: These methodologies are used to test a preliminary version of the data collection instrument. Pretests are used to gather data to refine questionnaire items and scales and assess reliability or validity. Pilot studies are also used to test aspects of implementation procedures in addition to testing survey measurement issues. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.
Procedures for Clearance
Since the types of instruments included under the umbrella of the clearance are so varied, we cannot specify at this point the exact activities that will be involved in any particular GenIC. With each generic IC, we will provide OMB with a copy of instruments, supplementary materials, and a brief justification package in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. When iterative testing is conducted, initial instruments will be submitted for review and approval and any revised materials will be uploaded to ROCIS as a nonsubstantive change between each round of testing. A memo will detail any changes.
ACF understands that OMB will make every effort to review materials for individual generic information collection requests within 10 working days of submission. All information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. ACF will make separate information collection requests for full, non-developmental data collection efforts.
ACF will provide a report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance with subsequent overarching generic information collection renewals. Attachment A provides an overview of ACF/OPRE’s use of this generic information collection between March 2015 and the submission date of this renewal request.
A3. Improved Information Technology to Reduce Burden
ACF/OPRE and its contractors will employ information technology as appropriate to reduce the burden on respondents who agree to participate in its research. We will provide specific information about the use of technology for each GenIC request.
A4. Efforts to Identify Duplication
This research does not duplicate any other data collection design work being done by ACF. The purpose of this clearance is to better inform and improve the quality of ACF’s research and evaluation. Pre-testing of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pre-testing plan. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.
A5. Involvement of Small Organizations
The information collection activities proposed under this clearance are not expected to impact small organizations. If an individual collection involves small organizations, the GenIC justification package will include a discussion to address this involvement.
A6. Consequences of Less Frequent Data Collection
This generic clearance involves instrument and procedure development activities for each study that is connected with the clearance. This may include one-time collections or iterative testing, based on the specific situation. In all cases, without the proposed information collection activities, the quality of the data collected for ACF studies would suffer.
A7. Special Circumstances
There are no special circumstances for the proposed data collection efforts.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on October 20, 2017; Volume 82, Number 202, page 48820, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment B. During the notice and comment period, no substantive comments were received.
Consultation with staff from ACF contractors carrying out research and evaluation surveys will occur in preparation for and in conjunction with the fielding of these data collections under this request. Information about consultation for individual development activities will be provided in GenIC requests.
A9. Incentives for Respondents
Gifts of appreciation may be provided when appropriate to respondents for activities conducted under this clearance. The type and amount will depend on the types of data collection and the specific population contacted. Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance may receive a small gift of appreciation. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by other Federal cognitive laboratories. For participation in a cognitive interview participants may receive up to $40, and for participation in a focus group it is up to $75 unless otherwise specifically justified. Respondents for methods that are relatively low in burden will not receive a gift in appreciation unless there are extenuating circumstances that warrant it, in which case this will be discussed in the individual justification package.
Not all individual information collections under this generic clearance will provide incentives. If an incentive is proposed, a detailed justification based on the type of collection, population of respondents, and other circumstances will be provided in the individual information collection request.
A10. Privacy of Respondents
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.
Individual statements will be included with each GenIC request submitted under this clearance, but in general, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor will ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. If evaluation staff must sign privacy pledges, these will be referenced in the individual information collection requests.
As necessary, contractors shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
A11. Sensitive Questions
Most of the questions included in these pre-testing activities will not be of a sensitive nature. However, it is possible that some potentially sensitive questions may be included in instruments tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them as much as possible before the actual survey is administered. Information about and justification for any sensitive questions will be included in the justification statement for each GenIC request.
A12. Estimation of Information Collection Burden
Previously Approved Information Collections
The last revision of the generic clearance was approved for 3,825 total burden hours.
Newly Requested Information Collections
The estimated burden for this renewal is consistent with the previously approved level of burden. The estimated number of respondents has been reduced, but the average hours per response has been increased to an average of one hour per response. This change is based on experiences over the past three years.
Total Burden Requested Under this Information Collection
Instrument Type |
Estimated Total Number Respondents |
Estimated Number Responses Per Respondent |
Average Burden Hours Per Response |
Estimated Total Burden hours |
Hourly Wage Rate |
Annualized Hourly Costs |
Survey development field tests, respondent debriefing questionnaires, cognitive interviews, split sample experiments, focus groups |
3,825 |
1 |
1 |
3,825 |
$18.37 |
$23,421.75 |
To calculate the annualized cost to respondents for the hour burden, we assume an average household income of $38,218, or 200 percent of the poverty threshold of $19,109 for a family of three1. OPRE projects are expected to study low-income populations. This figure translates to an hourly rate of $18.37.
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
Although we cannot anticipate the actual number of participants, length of responses,
and/or mode of data collection for the surveys to be conducted under this clearance, we estimate cost to the Federal Government based on costs incurred between January 2012 and submission of this current request.
Based on previous costs, we estimate the annual costs to the Federal Government to be around $300,000. Costs will be covered by the individual research and evaluation projects, from their data collection budgets. These costs will be described in GenIC requests.
A15. Change in Burden
This request is to renew the use of the ACF/OPRE generic clearance for another three years. Based on historical use of this umbrella generic, ACF requests burden level to remain consistent with the previously approved burden under 0970-0355.
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
This clearance is for instrument and procedure development purposes. Data tabulations will be used to evaluate the results of instrument testing. The information collected in this effort will not be the primary subject of any published ACF reports; however, information may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. When necessary, results will be labeled as exploratory in nature. The results of this pre-testing research may be prepared for presentation at professional meetings or publication in professional journals.
Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.
ACF/OPRE will develop individual timelines for projects involving generic clearances based on an understanding that OMB/OIRA will review within 10 working days of receiving the information collection request.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1 As estimated by the US Census Bureau in “Preliminary Estimate of Weighted Average Poverty Thresholds for 2016 ” https://www.census.gov/data/tables/time-series/demo/income-poverty/historical-poverty-thresholds.html
| File Type | application/msword |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| Last Modified By | SYSTEM |
| File Modified | 2018-02-09 |
| File Created | 2018-02-09 |