Attachment A -Use of Pretesting Generic Clearance (0970-0355) – 2021-2024

Attachment A: use of pretesting generic CLEARANCE (0970-0355) – 2021-2024

Administration for Children and Families

U.S. Department of Health and Human Services

330 C Street, SW, 4^th Floor

Washington, DC 20201

In August 2021, OMB approved ACF’s request to renew the generic information collection for pre-testing of evaluation surveys (0970-0355). This report describes the use of the generic over the three years of approval (2021-2024), including the number of hours used, as well as the nature and results of the activities completed under this generic clearance.

Since this umbrella generic was renewed in 2021, ACF requested eight (8) information collections (IC) for pretesting. The use of the pre-testing generic has been beneficial to the development of instruments for these research studies. Over the three years, ACF collected information from 3,293respondents, for a total of 1,827 hours of burden. The continued use of the generic is indicative of the usefulness of pretesting our surveys and procedures and ACF expect the generic to be very beneficial for ACF as a whole in the future.

Approval Date	IC Title	Respondents	Responses	Burden Hours
9/14/2021	Success Sequence Interviews – Pre-Test	120	160	50
3/14/2022	Understanding Children's Transitions from Head Start to Kindergarten	43	43	75
8/19/2022	Supporting and Strengthening the Home Visiting Workforce (SAS-HV)	60	162	105
3/13/2023	The Home-Based Child Care Toolkit for Nurturing School-Age Children Pilot Study	990	990	323
4/7/2023	Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs Phase 1	450	900	153
4/27/2023	Feasibility Testing of Online Annual Survey of Refugees Mode	20	20	30
6/12/2023	Supporting and Strengthening the Home Visiting Workforce (SAS-HV): Online Pretest of Draft Reflective Supervision Measure	125	163	61
1/30/2024	Supporting and Strengthening the Home Visiting Workforce (SAS-HV): Testing and Validation of a Draft Measure of Reflective Supervision for Home Visiting	785	1,570	810
DATE	Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs	700	910	220
Total Over 3 Years:		3,293	4,918	1,827

Example uses of the Pre-testing Generic Clearance

Annual Survey of Refugees (ASR) Mode Testing

The Office of Refugee Resettlement (ORR) funds an annual survey of newly arrived refugee households to collect information on refugees’ progress toward self-sufficiency and integration during their first five years of living in the United States. The survey data are used to meet reporting requirements for ORR’s Annual Report to Congress and inform refugee resettlement policy, by providing a unique source of nationally representative information on refugees’ experiences of adjustment to life in the United States. The Annual Survey of Refugees (ASR) collects information on a range of information about refugees, including language proficiency, training and education, employment and income, public assistance, social connections, and access to health care. The annual survey is conducted by telephone in up to twenty languages.

ORR is interested in collecting evidence to inform a potential transition to a multi-mode version of the survey, in which respondents could participate either by telephone or online. However, it is imperative that ORR explores whether the quality of responses differs between survey modality – telephone interview versus online self-administration. The ASR project team conducted a feasibility test of fielding the ASR as a self-administered, web survey. Twenty interviews were conducted with refugees, five in each of the following languages: Dari, Arabic, Swahili, and Ukrainian. In addition to exploring the feasibility of the online survey for this refugee population, the interviews included cognitive probing to assess participants’ interpretation of selected questions.

The limited feasibility and cognitive test of the online version of the ASR showed that a fraction of refugees arrived in the last five years would be able to respond to the survey online and that they would also prefer to do so, if given the option. Given the possibility to offer multiple modes for completion, any reduction of the number of cases requiring a telephone interview would positively impact the survey cost. Additionally, the data gathered from the pretest informs the translation of surveys as the project gained valuable information on where the telephone survey did not translate well in the online version. The ASR project team gathered valuable information to reformulate questions based on specific language and cultural translations as well as explore ways to structure an online version of the ASR.

The project team benefitted from the pretesting generic clearance in several ways. The ability to conduct the pretest allowed the project team to begin assessing the capacity and willingness of respondents to complete an online survey quickly and efficiently. The speed of the generic clearance process enabled the project team to conduct the mode testing within the contract timeline and allowed the testing to overlap with the routine administration of the annual survey to maximize cost effectiveness. Had the project team been required to seek a full OMB clearance, the mode test would not have been able to align with the routine administration of annual survey.

Supporting and Strengthening the Home Visiting Workforce

The Supporting and Strengthening the Home Visiting Workforce (SAS-HV) project developed, pre-tested, refined, and is currently conducting a developmental validation study of a measure of reflective supervision practices. Reflective supervision is a specific type of professional development that is intended to help home visitors (1) develop knowledge, skills, and key competencies to carry out their roles and (2) support and help restore home visitor professional well-being. Despite strong theoretical support for reflective supervision, there is limited understanding of how it is implemented in practice and limited evidence of effectiveness. This is due, in part, to a lack of valid, reliable measures of reflective supervision. Valid, reliable measures of reflective supervision are important for advancing research on the role of reflective supervision in supporting home visitors’ work with families.

The SAS-HV project sought a pretesting generic clearance, approved in 8/19/2022, to conduct a concept mapping study design to inform the development of the reflective supervision measure. The data collected from a survey and group interpretation meeting identified what end users of the measure view as key elements of reflective supervision in the home visiting context. The study team used these results to assemble a pool of items and then sought a second pretesting generic clearance, approved on 6/12/2023, to conduct a preliminary pretest of the measure with home visiting supervisors. Using the results from the pretest, the Study Team refined the reflective supervision measure and sought a third pre-testing generic clearance, approved 1/30/2024, to conduct a mixed methods testing and validation study of the current draft of the reflective supervision measure. The data collected from this study will help assess the extent to which the measure is valid and reliable for use in home visiting contexts. We will disseminate the final measure along with a manual and technical report.

The pre-testing generic clearance enabled the team to develop, iterate, and fully test the reflective supervision quickly after it had been identified as a need by home visiting grantees early in the contract. It also enabled the team to maximally use the information gained from home visiting staff at each data collection point to refine the measure. The information we learned will help us create materials and disseminate the measure to end users. If we had had to seek a full OMB clearance, we would not have been able to work with home visiting supervisors at three time points to inform the measure development and refinement, and instead may have been only able to pilot the measure one time. The timing has made it such that we will have more confidence in the validity of the measure and more time to disseminate it to home visiting researchers and grantees for the benefit of the field.

Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) Pilot Study

The Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit) Pilot Study was completed as part of the Home-Based Child Care Supply and Quality (HBCCSQ) project. Home-based child care (HBCC) is child care and early education (CCEE) offered in a provider’s or child’s home. The HBCCSQ project intended to fill gaps in the understanding of HBCC supply and address challenges defining and measuring quality in HBCC settings. The team sought to develop a toolkit that would account for features of HBCC that most or no existing measures of child care quality recognize, such as caring for school-age children.^[1] In addition, few measures were developed to account for features that may be more likely to take place in or to be implemented differently in HBCC settings, such as children’s cross-age peer interactions, racial and ethnic socialization, and nontraditional-hour care.^[2] These features that are missing from existing measures may meet the CCEE needs of families who use HBCC and might support positive outcomes for these children and families. To fill these gaps, the HBCCSQ project developed and pretested the Home-Based Child Care Toolkit for Nurturing School-Age Children (HBCC-NSAC Toolkit).

The instruments in the HBCC-NSAC Toolkit included a provider questionnaire and a family questionnaire, and both were available in English and Spanish. The HBCC-NSAC Toolkit is intended for HBCC providers who are licensed or regulated by states to provide CCEE, commonly referred to as family child care (FCC) providers, and those who are unlicensed, or legally exempt from state licensing or other state regulations for CCEE, commonly referred to as family, friend, and neighbor (FFN) providers. The toolkit’s primary purpose is to help HBCC providers who regularly care for at least one school-age child identify and reflect on their strengths and areas of growth related to their caregiving practices and how they partner with children’s families. Research about and with this group of CCEE educators has been very limited, therefore the team sought to develop and refine the toolkit and research procedures to sensitively and effectively center the experiences of HBCC providers. After a robust process of developing the toolkit relying on existing research, consultation with academic, professional development, policy and community experts, the team sought to test the HBCC-NSAC Toolkit instruments and data collection procedures and materials with a diverse group of HBCC providers and the families they care for before conducting a larger validation study. OPRE sought a pretesting generic clearance, which was approved in March 2023, to conduct a pilot study.

The team benefitted from the pretesting generic clearance in a few ways. The speed of the generic clearance process enabled us to begin work quickly. If we had to seek a full OMB clearance, we would not have been able to refine the toolkit instruments and to improve data collection procedures with input from providers, families and community organizations prior to implementing a rigorous validation study within the timeframe of the current contract.  For example, the pilot study produced initial psychometric evidence leading to reductions in items and changes to the response scales. Specifically, based on analysis of pilot study data (i.e., item-total correlations, coefficient alphas, Rasch scores, interfactor descriptive statistics and correlations, exploratory analysis of potential differential item functioning (DIF) for subgroups), we reduced the number of items in the provider questionnaire from 223 to 152 items. Without sufficient sample size to support these analyses, it would not have been possible to revise the instruments prior to the full validation study. In addition, pilot study findings led to wording changes in outreach materials, instructions to respondents, and items in the Spanish and English versions of the provider and family questionnaires. As a result, the team has been able to collect data on the current contract to support rigorous psychometric analysis of the HBCC-NSAC Toolkit.

Understanding Children's Transitions from Head Start to Kindergarten

As part of a larger project, the Understanding Children’s Transitions from Head Start to Kindergarten (HS2K) project team developed and conducted initial testing of a new set of survey measures to assess Head Start and K-12 administrators’ perspectives on systems-level approaches to Head Start-to-kindergarten transitions. Foundational work of the HS2K project, including a review of the literature and a scan of existing measures, had revealed that there were gaps in knowledge about systems-level supports for kindergarten transitions and a lack of tools to measure this information. The development of the new measures was intended to support future research regarding the degree of coordination, collaboration, and alignment of kindergarten transition policies, perspectives, professional supports, and practices both within and across Head Start and elementary school systems. 

OPRE sought a pretesting generic clearance, approved in March 2022, to test and refine new measures of kindergarten transition strategies by conducting cognitive interviews with Head Start and K-12 administrators. The measures were intended to assess the content, quality, and quantity of kindergarten transition strategies. The HS2K project team conducted cognitive interviews in two iterative phases, where findings from the first phase of interviews and analysis informed changes to the survey tested during the second phase. In both phases, the team followed a think-aloud protocol that included prompts to determine comprehension, retrieval, judgment, and response for survey questions and response categories. The items and response options underwent significant revision from initial development to the last round of cognitive interviews.

OPRE has benefitted from using the generic clearance in this case in several ways. Because we were able to conduct pre-testing as part of the HS2K project, we are now able to plan for a new national descriptive study of Head Start-to-kindergarten transitions with a set of measures that has already gone through initial testing, thus reducing the time and effort needed for measure development and testing under a new contract. Additionally, the reduced time for review and approval under a generic allowed us to save time and effort while still complying with the requirements of the Paperwork Reduction Act. And, because the pre-testing generic is designed for specific purposes and methods, the shorter review and approval process was relatively streamlined.

[1]^[1] A. R. Datta, C. Milesi, S. Srivastava, and C. Zapata-Gietl. “NSECE Chartbook – Home-Based Early Care and Education Providers in 2012 and 2019: Counts and Characteristics.” OPRE Report No. 2021-85, Washington DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021.

[2]^[2] Bromer, Juliet, Toni Porter, Chris Jones, Marina Ragonese-Barnes, and Jaimie Orland. “Quality in Home-Based Child Care: A Review of Selected Literature.” OPRE Report # 2021-136. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021a.

Doran, Elizabeth, Ann Li, Sally Atkins-Burnett, Jasmine Forde, Jaimie Orland, Marina Ragonese-Barnes, Nathan Mix, Natalie Reid, and Ashley Kopack Klein (2022). Quality in Home-Based Child Care: Summary of Existing Measures and Indicators. OPRE Report #2022-27. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Doran, Elizabeth, Ann Li, Sally Atkins-Burnett, Jasmine Forde, Jaimie Orland, Marina Ragonese-Barnes, Nathan Mix, Natalie Reid, and Ashley Kopack Klein. “Compendium of Measures and Indicators of Home-Based Child Care Quality.” OPRE Report #2022-28. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2022.