Supporting Statement A - Umbrella Generic for Pretesting_Extension 2021_final Aug 2021

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Pre-testing of Evaluation Data Collection Activities

OMB Information Collection Request

0970 - 0355

Supporting Statement

Part A

March 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Part A

Executive Summary

• Type of Request: This Information Collection Request is for an extension with no changes to an umbrella generic: Pre-testing of Evaluation Data Collection Activities. We are requesting 3 years of approval.

• Progress to Date: This umbrella generic clearance (0970-0355) was originally approved for use in October 2008 and renewed in January 2012, January 2015, and again in May 2018. The Administration for Children and Families has consistently used this mechanism to pretest instruments and procedures to inform research and evaluation work.

• Description of Request: This request is to renew ACF’s umbrella generic for Pre-testing of Evaluation Data Collection Activities. No changes are proposed.

  • The goal of this umbrella generic information collection request is to develop and test information collection instruments and procedures.

  • Intended use of the resulting data is to evaluate and improve the quality of the data gathered through ACF’s research and evaluation studies.

  • These pretesting and piloting efforts will collect data using well established methodologies, including: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments,(f) focus groups, and (g) pilot studies/pretests.

  • The populations to be studied include participants in ACF programs being evaluated; participants in ACF demonstrations; recipients of ACF grants and individuals served by ACF grantees; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services.

  • Data will be analyzed using well established methods, such as data tabulations to evaluate the results of instrument testing.

We do not intend for the information collected under this umbrella generic to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Office of Planning, Research, and Evaluation (OPRE), in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks renewal of this generic survey development clearance to allow us to use samples of more than nine participants in applying methods useful for identifying instrument and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions.

OPRE studies ACF programs, and the populations they serve, through rigorous research and evaluation projects. These include evaluations of existing programs, evaluations of innovative approaches to helping low-income children and families, research syntheses and descriptive and exploratory studies. This generic clearance allows us to identify if and when an instrument may be simplified for respondents, respondent burden may be reduced, among other possible improvements. The research completed under this generic is intended to be informative in nature; the studies may be iterative, as variation in questions or procedures are proposed, evaluated, and retested. The pretesting of instruments is necessary to improve future ACF information collections, resulting in higher quality studies. The core methodology and target populations will be consistent and burden caps and token of appreciation structure are proposed in this request.

Study Background

This generic clearance (0970-0355) was originally approved for use in October 2008 and renewed in January 2012, January 2015, and again in May 2018. Table A shows the number of respondents and total burden requested for each approval period. For a list of approved generic clearances since May 2018, see Attachment A.

Table A

Timeframe	Respondents	Total Burden
October 2008-October 2011	3,605	1,336
January 2012-January 2015	2,584	1,701
May 2018-present	538	1,114

In the past, ACF has received approval for seven types of activities: (a) cognitive and usability laboratory and field techniques, (b) behavior coding (c) exploratory interviews (d) respondent debriefing questionnaires, (e) split sample experiments, (f) focus groups, and (g) pilot studies/pretests. We expect future activities to fall within these or similar categories.

Example ACF projects that have benefited from this GenIC include activities as part of the Head Start Family and Child Experiences Survey (FACES), and the National Survey of Early Care and Education, among others. See Reginfo.gov (https://www.reginfo.gov/public/do/PRAOMBHistory?ombControlNumber=0970-0355) for examples of instruments previously approved under this generic clearance.

Legal or Administrative Requirements that Necessitate the Collection

ACF is undertaking the collections at the discretion of the agency.

A2. Purpose

Purpose and Use

ACF will use the information collected to develop and test data collection instruments and procedures to evaluate and improve the quality of the data gathered through ACF’s research and evaluation studies. Evaluation of information collection instruments and/or procedures is the main objective of the activities in this clearance. The goal of developing these instruments and refining procedures is to improve the evaluations of ACF programs and demonstrations.

ACF will use the results internally to inform subsequent information collection requests.

Results of these methodological studies may be made public through methodological appendices or footnotes, reports on instrument development, instrument user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. The results of these pre-testing activities may be prepared for presentation at professional meetings or publication in professional journals. Results will be labeled as exploratory in nature and any limitations will be described.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

Individual GenIC submissions under this umbrella generic will include the guiding questions for the specific proposed activities.

Study Design

All of the methods proposed for instrument and procedure development will be used with either purposive or statistically representative samples of participants in ACF programs being evaluated; participants in ACF demonstrations, many of which are supported by ACF program grants; recipients of ACF grants and individuals served by ACF grantees; comparison group members; and other relevant populations, such as individuals at risk of needing ACF services. Data tabulations will be used to evaluate the results of instrument testing. All information collection activities conducted under this GenIC will be voluntary and low-burden.

ACF will test a variety of instruments and procedures under this clearance. The exact nature of the instruments and the samples is dependent on each individual project and details will be provided for each GenIC request. The particular samples included in future generic information collection requests will vary based on the content of the instrument being tested. The proposed methods for use under this clearance include the following:

• Cognitive and Usability Laboratory and Field Techniques: A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted one-on-one with respondents. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand the physical features of a survey, for instance, its display and navigational features. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. ACF broadened the methodology request to include cognitive interviews at OMB’s suggestion in 2012.

• Behavior Coding: This test serves as the vehicle for conducting standardized behavior coding of the interaction between the respondent and the interviewer. It involves applying a standardized coding scheme at the completion of a field interview, either by a coder using a tape-recording of the interview or by an observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire. For example, if respondents frequently interrupt the interviewer before the question is completed, the question may be too long. If respondents frequently give incomplete answers, this suggests there may be some other problems with the question. An objective of standardized field tests is to collect data derived from standardized coding schemes to identify problem areas in a questionnaire in an objective and reliable manner.

• Exploratory Interviews: A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey. They may cover discussions related to administrative records, subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign.

• Respondent debriefing questionnaires: In this method, standardized debriefing questionnaires are administered to respondents who participated in a field test. The debriefing form is administered at the end of the questionnaire being tested and contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a sample of respondents to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers. Interviewer debriefing enhances a standardized field test since it utilizes the knowledge of the survey staff that have the closest contact with respondents.

• Split sample experiments: This method involves testing alternative versions of questionnaires, some of which may be designed to address problems identified in draft questionnaires or questionnaires from previous survey waves. The use of multiple questionnaires is a critical component in this type of data collection, which can include mail, telephone, or personal visit interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a randomly assigned control version facilitates statistical evaluation of the performance of alternative versions of the questionnaire. In any split sample experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than approximately forty persons or more than 100 persons in a split sample test.

• Focus groups: This method involves group sessions guided by a moderator who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring a range of issues that may be relevant to development and administration of a survey.

• Pilot Studies/Pretests: These methodologies are used to test a preliminary version of the data collection instrument. Pretests are used to gather data to refine questionnaire items and scales and assess reliability or validity. Pilot studies are also used to test aspects of implementation procedures in addition to testing survey measurement issues. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.

Procedures for Clearance

Since the types of instruments included under the umbrella of the clearance are so varied, we cannot specify at this point the exact activities that will be involved in any particular GenIC. With each generic IC, we will provide OMB with a copy of instruments, supplementary materials, and a brief justification package in advance of any testing activity. When split sample experiments are conducted, either in small group sessions or as part of a field test, all the questionnaires to be used will be provided. When iterative testing is conducted, initial instruments will be submitted for review and approval and any revised materials will be uploaded to ROCIS as a nonsubstantive change between each round of testing. A memo will detail any changes.

ACF understands that OMB will make every effort to review materials for individual generic information collection requests within 10 working days of submission. All information gathered from these testing activities will be for the purpose of improving data collection instruments and procedures, not for the purpose of generating findings on the substantive topic under study. ACF will make separate information collection requests for full, non-developmental data collection efforts.

ACF will provide a report summarizing the number of hours used, as well as the nature and results of the activities completed under this clearance with subsequent overarching generic information collection renewals. Attachment A provides an overview of ACF’s use of this generic information collection between May 2018 and the submission date of this renewal request.

Other Data Sources and Uses of Information

Individual GenIC submissions under this umbrella generic will include information, as appropriate, about how the information collected may be used in concert with other sources of information (e.g., administrative data sources, prior data collections).

A3. Use of Information Technology to Reduce Burden

ACF and its contractors will employ information technology as appropriate to reduce the burden on respondents who agree to participate in its research. We will provide specific information about the use of technology for each GenIC request.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The proposed activities under this umbrella generic will not duplicate any other data collection design work being done by ACF. The purpose of this clearance is to better inform and improve the quality of ACF’s research and evaluation. Pre-testing of the scale envisioned here would not be done under other circumstances due to the time constraints of seeking clearance for each individual survey’s pre-testing plan. To the maximum extent possible, we will make use of previous information by reviewing results of previous evaluations of survey data before we attempt to revise questionnaires using additional field work sought under this clearance.

A5. Impact on Small Businesses

The information collection activities proposed under this clearance are not expected to impact small organizations. If an individual collection involves small organizations, the GenIC justification package will include a discussion to address this involvement.

A6. Consequences of Less Frequent Collection

This generic clearance involves instrument and procedure development activities for each study that is connected with the clearance. This may include one-time collections or iterative testing, based on the specific situation. In all cases, without the proposed information collection activities, the quality of the data collected for ACF studies would suffer.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on January 5, 2021; Volume 86, Number 2, page 308, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

Consultation with staff from ACF contractors carrying out research and evaluation surveys will occur in preparation for and in conjunction with the fielding of these data collections under this request. Information about consultation for individual development activities will be provided in GenIC requests.

A9. Tokens of Appreciation

Tokens of appreciation may be provided when appropriate to respondents for activities conducted under this clearance. The type and amount will depend on the types of data collection and the specific population involved. Respondents for activities conducted in the laboratory (that is, cognitive interviews and focus groups) under this clearance may receive a token of appreciation. For participation in a cognitive interview participants may receive up to $40, and for participation in a focus group it is up to $75 unless otherwise specifically justified. Respondents for methods that are relatively low in burden will not receive a gift in appreciation unless there are extenuating circumstances that warrant it, in which case this will be discussed in the individual justification package. For any collection over 90 minutes, participants may be offered to token of appreciation to account for incidental expenses (transportation, child care, lost wages, etc.).

Not all individual information collections under this generic clearance will provide tokens of appreciation. If a token of appreciation is proposed, a detailed justification based on the type of collection, population of respondents, and other circumstances will be provided in the individual information collection request.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

Individual GenICs under this umbrella generic may request personally identifiable information (PII), as appropriate. This is most commonly used for contacting individuals. If a GenIC proposes to collect PII, the individual justification package will include information about what specific PII will be requested and the proposed uses of the PII.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.

Individual statements will be included with each GenIC request submitted under this clearance, but in general, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor will ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. If evaluation staff must sign privacy pledges, these will be referenced in the individual information collection requests.

Data Security and Monitoring

As necessary, contractors shall use Federal Information Processing Standard (currently, FIPS 140-2) compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.

A11. Sensitive Information ¹

Most of the questions included in these pre-testing activities will not be of a sensitive nature. However, it is possible that some potentially sensitive questions may be included in instruments tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them as much as possible before the actual survey is administered. Information about and justification for any sensitive questions will be included in the justification statement for each GenIC request.

A12. Burden

Explanation of Burden Estimates

The last revision of the generic clearance was approved for 3,825 total burden hours. The estimated burden for this renewal is consistent with the previously approved level of burden. For this extension request, the burden has been broken out to provide more detailed estimates, based on prior use. Table 12.2 table below is illustrative. While we will not exceed the total burden cap for this generic (3,825), we may use more or less burden within each instrument or activity type.

At the time of this submission, there are two approved GenICs with ongoing information collections. The burden associated with these collections, 292 hours, is detailed in table 12.1.

Table 12.1: Burden Approved for Ongoing Generic Information Collections
GenIC Title	Instrument	No. of Respondents	No. of Responses per Respondent	Avg. Burden per Response (in hours)	Total Burden (in hours)
Touchpoints for Addressing Substance Use Issues in Home Visiting: Performance Measurement Pilot	Instrument 1: SUD-1 and SUD-2 Measures Reporting Tool	6	10	0.25	15
	Instrument 2: Interview Protocol: MIECHV State Awardees	2	1	1.5	3
	Instrument 3: Interview Protocol: LIA Managers and Data Managers, Home Visiting Supervisors, and Home Visitors	42	1	1.2	50
	Instrument 4: Interview Protocol: Home-Visiting-Model Representatives	6	1	.75	5
Mother and Infant Home Visiting Program Evaluation: Kindergarten Follow-Up (MIHOPE-K)	Instrument 1: Recruitment Screener	70	1	0.11	8
	Instrument 2: Virtual Visit (Direct assessments of children, Direct assessments of caregivers, Videotaped caregiver-child interactions, Debrief)	110 (55 caregivers and 55 children)	1	1.92	211
Total					292

Estimated Annualized Cost to Respondents

To calculate the annualized cost to respondents for the hour burden, we assume an average household income of $41,172, or 200 percent of the poverty threshold of $20,586 for a family of three². OPRE projects are expected to study low-income populations. This figure translates to an hourly rate of $20.42. The total cost over 3 years is $78,106.50 and the annual cost is $26,035.50.

Table 12.2: Burden Request for New Generic Information Collections
Instrument or Activity Type	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Average Hourly Wage Rate	Total Respondent Cost
Interviews/Debriefings	267	3	1.5	1,200	$20.42	$24,504.00
Questionnaires	280	10	.5	1,400	$20.42	$28,588.00
Focus Groups	600	1	1.5	900	$20.42	$18,378.00
Usability Tests	65	5	1	325	$20.42	$6,636.50
Totals	1212			3,825		$78,106.50

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Although we cannot anticipate the actual number of participants, length of responses, and/or mode of data collection for the surveys to be conducted under this clearance, we estimate cost to the Federal Government based on costs incurred on previously approved GenICs.

Based on previous costs, we estimate the annual costs to the Federal Government to be around $325,000. Costs will be covered by the individual research and evaluation projects, from their data collection budgets. These costs will be described in GenIC requests.

A15. Reasons for changes in burden

This request is to renew the use of the ACF generic clearance for another three years. Based on historical use of this umbrella generic, ACF requests burden level to remain consistent with the previously approved burden under 0970-0355.

A16. Timeline

Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

ACF will develop individual timelines for projects involving generic clearances based on an understanding that OMB/OIRA will review within 10 working days of receiving the information collection request.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Attachment A: Use of Pretesting Generic Clearance (0970-0355) – 2018-2021

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

2 As estimated by the US Census Bureau in “Preliminary Estimate of Weighted Average Poverty Thresholds for 2020 ” https://www.census.gov/data/tables/time-series/demo/income-poverty/historical-poverty-thresholds.html

11