VIQI-FS Supporting Statement A - CLEAN Revised draft Jan2022

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Variations in Implementation of Quality Interventions

OMB Information Collection Request

0970 - 0508

Supporting Statement

Part A

April 2021

Updated January 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Ivelisse Martinez-Beck
Amy Madigan

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a nonsubstantive change request.

• Progress to Date: In February 2017, the Variations in Implementation of Quality Interventions (VIQI): Examining the Quality-Child Outcomes Relationship in Child Care and Early Education Project received approval under the Generic for Pretesting Activities (OMB #0970-0356) to collect information on existing services and populations served from staff responsible for Head Start and child care programs to better understand the landscape of early care and education programs and to aid in the refinement of the study design. The project subsequently received approval for a pilot study, impact study, and process study in May 2018 (OMB #0970-0508) and conducted a year-long pilot study in 2018-2019. Lessons learned from the pilot have informed the study design and updates to data collection instruments and installation activities focused on teacher professional development.

In September 2019, the study team began landscaping and recruitment activities for the Impact Evaluation and Process Study. However, in March 2020 the COVID-19 pandemic resulted in the closure of many businesses and organizations, including Head Start and community-based child care centers, which are the target of these landscaping and recruitment activities. In light of this, the study team decided to postpone the Impact Evaluation and Process Study to the 2021-2022 school year. The team received approval for an extension with changes in 2021.

• Timeline: The follow-up data collection for the Impact Evaluation and Process Study is planned for winter through spring 2022.

• Summary of changes requested: The following changes are being requested:

  • to finalize the items (i.e., remove placeholder items) in the follow-up teacher survey and teacher reports for questions about children in classroom.

  • to adjust some language in instruments to accommodate the use of virtual coaching due to restrictions from the COVID-19 pandemic

  • to confirm the list of child assessments that will be collected in the Follow-up Protocol for Child Assessments; and

  • to adjust the total burden associated with the Follow-up Protocol for Child Assessments.

We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: Due to the timing of the 2021-2022 school year and the need to collect follow-up data by March 2022, the approval is time sensitive.

A1. Necessity for Collection

The early childhood education (ECE) literature has identified several basic dimensions of classroom quality – such as structural, process (which we refer to as “interactional” henceforth) and instructional quality – that are hypothesized to promote child outcomes. Nonexperimental evidence portrays an intriguing pattern of correlational findings suggesting that quality may need to reach certain levels before effects on child outcomes become evident and that different dimensions of quality may interact with each other in synergistic ways to affect child outcomes. But, existing evidence has not pinpointed the exact levels that are consistently linked with child outcomes. Further, there is relatively little causal evidence showing that efforts to strengthen ECE quality will yield improvements in child outcomes. Without such rigorous evidence, it is difficult to draw policy and practice implications.

The field needs a stronger, causal evidence base that provides a better understanding of the quality-child outcomes relationship, the dimensions of quality that are most related to child outcomes, and the program and classroom factors that aid delivery of quality teaching and caregiving in ECE settings. VIQI aims to fill these gaps in the ECE literature. Primary data collection of implementation, classroom quality, child outcome measures, and drivers of implementation and classroom quality is needed, as no existing data sources reliably and uniformly capture these constructs.

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) launched the Variations in Implementation of Quality Interventions (VIQI): Examining the Quality-Child Outcomes Relationship in Child Care and Early Education Project in 2016. VIQI is a research study sponsored by the Office of Planning, Research, and Evaluation (OPRE) and conducted through a contract with MDRC and its subcontractors, Abt Associates, RTI International, University of Virginia, and MEF Associates. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The literature and theory point to classroom quality – or the quality of children’s learning opportunities and experiences in the classroom – as being potentially influential for promoting child outcomes. Yet, there is considerable variation in the overall quality of ECE services, with instructional quality – a hypothesized key driver of children’s gains – often being low across ECE programs nationally despite a focus on quality improvement at national, state and local levels. Further, these relationships may vary with children of different ages. Indeed, there are still many open questions about how best to design and target investments to ensure that children, particularly low-income children, receive and benefit from high-quality, ECE programming on a large scale.

There is a growing, but imperfect, knowledge base about which dimensions of quality are most important to strengthen, and what levels of quality need to be achieved to promote child outcomes across ECE settings. The ECE literature has identified several basic dimensions of classroom quality – such as structural, process and instructional quality – that are hypothesized to promote child outcomes. Nonexperimental evidence portrays an intriguing pattern of correlational findings suggesting that quality may need to reach certain levels before effects on child outcomes become evident and that different dimensions of quality may interact with each other in synergistic ways to affect child outcomes. But, existing evidence has not pinpointed the exact levels that are consistently linked with child outcomes. Further, there is relatively little causal evidence showing that efforts to strengthen ECE quality will yield improvements in child outcomes. Without such rigorous evidence, it is difficult to draw policy and practice implications.

The purpose of the VIQI project is to provide causal evidence on the quality-child outcomes relationship, the dimensions of quality that are most related to child outcomes, and the program and classroom factors that aid delivery of quality teaching and caregiving in ECE settings. To do so, the study employs a rigorous experimental study design testing two promising interventions that consist of curricular and professional development supports and target different dimensions of classroom quality to build evidence about the effectiveness of the interventions and investigate the relationship between classroom quality (and its dimensions) and children’s outcomes in mixed-aged ECE classrooms that serve both three- and four-year-old children.

The information collected will be published online in a final report available to researchers, policymakers, and practitioners within 12 months of the end of the evaluation. At the time of publication, the report will be emailed to participating centers. The information may be used to better understand the types of interventions that can improve preschool classroom quality in Head Start and community-based programs and the quality-outcomes relationship. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests
The impact evaluation and process study of the VIQI project aims to address the following research questions:

Impact Evaluation Research Questions

• What are the effects of the interventions on different dimensions of quality, teacher, and child outcomes? For whom and under what circumstances are the interventions more or less effective?

• What are the causal effects of different dimensions of quality (structural and interactional quality, and instructional quality) on children’s outcomes?

• Are there thresholds in the effects of quality on child outcomes?

• Do the effects of quality on child outcomes differ, depending on child, staff and center characteristics, including centers that vary in their initial levels of quality?
  
  

Process Study Research Questions

• What are the characteristics of the participants at the center, staff, and child levels in the Impact Evaluation? Which of these characteristics are drivers of fidelity of implementation? How do these drivers relate with each other?

• What are the implementation systems (e.g., professional development, training, coaching, assessments) that support the delivery of the interventions in classrooms? How much variation is there in participation of these supports? What drivers seem to support or inhibit participation?

• To what degree are the interventions delivered in the classrooms as intended? How much variation is there in fidelity of implementation of the interventions? What drivers seem to facilitate or inhibit successful implementation and fidelity to the intended intervention model(s)?

• What is the relative treatment contrast achieved in teacher practices targeted by the intervention(s)?

Study Design

VIQI is a three-group randomized controlled study where ECE centers are randomly assigned to one of two intervention conditions or a business-as-usual control condition. Intervention 1 takes a whole-child, global approach and targets structural and interactional quality, and Intervention 2 takes an integrated, domain-specific approach with scope and sequence and targets instructional quality. If the interventions improve quality as intended, this design will create random (experimentally-induced) variation in the two quality dimensions (structural and interactional quality, and instructional quality) that can be used to rigorously estimate their effect on children’s outcomes. Because of the design, we are only able to assess the effects of two dimensions of quality on children’s outcomes, so structural and interactional quality will be analyzed together. See sections B1 and B6 in Supporting Statement B for more details.

A conceptual framework (See Appendix A: Conceptual Framework) underlies the VIQI study design, highlighting (from left to right): there are varied setting, center, classroom and teacher characteristics that drive or influence not only how curricula and professional development supports are implemented but also how quality might affect child outcomes. We expect that implementation of these two interventions primarily will improve the targeted dimension of quality (bold line from an intervention to a quality dimension) and, in turn, will lead to improvements in children’s developmental outcomes. In line with this framework, data collection instruments aim to collect data on: implementation drivers and inputs, fidelity of implementation, classroom quality, and child outcomes. With the exception of information on fidelity of implementation, which will be collected throughout the school year, all other data collection instruments will be collected at baseline (fall) and follow-up (spring). Data collection protocols are detailed in Table A2.1. For a description of changes in data collection protocols since our prior OMB approval, see Table B3.2 in Supporting Statement B.

A2.1 Data Collection Protocols

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode, Duration, and OMB approval status
Screening and Recruitment of ECE Centers	Landscaping protocol with stakeholder Agencies Screening protocol for phone calls Protocol for follow-up calls/in-person visits for screening and recruitment activities	Respondents: Head Start grantee and community-based child care agency staff; Head Start and community-based child care center staff Content: Program, center, classroom characteristics; changes in programming due to COVID-19 Purpose: Assess eligibility of localities, programs, centers for inclusion in study	Mode: Phone or video conference calls, in –person if feasible Duration: 1.2 – 2 hours OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline administrator survey	Respondents: Center administrators Content: Demographics, background, characteristics of centers Purpose: Capture implementation drivers, influences on intervention impacts; describe sample	Mode: web-based, paper-pencil Duration: 36 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline teacher/ assistant teacher survey	Respondents: Teachers Content: Demographics, background characteristics of teachers, classroom instruction Purpose: Capture implementation drivers, influences on intervention impacts; describe sample	Mode: web-based, paper-pencil Duration: 36 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline coach survey	Respondents: Coaches Content: Demographics, background characteristics of coaches Purpose: Capture implementation drivers	Mode: web-based, paper-pencil Duration: 36minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline classroom observation protocol (teacher burden)	Respondents: Teachers Content: Staffing, curricula used, how typical day was; classroom quality dimensions Purpose: Capture initial levels of quality and background information about observation day	Mode: verbal questions Duration: 18 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline parent/ guardian information form	Respondents: Parents/guardians Content: Demographics, background of children and families Purpose: Describe sample, capture implementation drivers, influences on intervention impacts	Mode: web-based, paper-pencil Duration: 6 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Baseline protocol for child assessments (child burden)	Respondents: Children Content: Children’s skills Purpose: Examine baseline equivalence, use as covariates in impact analyses, define subgroups of interest	Mode: in-person direct assessment, remote if needed Duration: 30 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Teacher reports to questions about children in classroom (administered as part of the baseline teacher survey)	Respondents: Teachers Content: Children’s skills Purpose: Examine baseline equivalence, use as covariates in impact analyses, define subgroups of interest	Mode: web-based, paper-pencil Duration: 10 minutes/child OMB Approval Status: No changes since previous approval
Baseline data collection	Administrator/teacher COVID-19 supplemental survey questions (administered as part of administrator and teacher survey, to contextualize findings from impact evaluation and process study due to circumstances surrounding COVID-19 at the time of data collection)	Respondents: Administrators, teachers Content: Center and classroom programming, admin/ teacher function given COVID-19 Purpose: Contextualize findings from the impact evaluation and process study	Mode: web-based, paper-pencil Duration: 15 minutes OMB Approval Status: No changes since previous approval
Baseline data collection	Parent/guardian reports to questions about children (administered as part of the baseline parent/guardian information form)	Respondents: Parents/guardians Content: Children’s skills Purpose: Examine baseline equivalence, use as covariates in impact analyses, define subgroups of interest	Mode: web-based, paper-pencil Duration: 6 minutes OMB Approval Status: No changes since previous approval
Follow-up data collection	Follow-up administrator survey	Respondents: Administrators Content: Characteristics of administrators and centers Purpose: Capture implementation drivers, influences on intervention impacts	Mode: web-based, paper-pencil Duration: 30 minutes OMB Approval Status: No changes since previous approval changes since previous approval
Follow-up data collection	Follow-up teacher/ assistant teacher survey	Respondents: Teachers Content: Characteristics of teachers and classrooms Purpose: Capture implementation drivers, influences on intervention impacts	Mode: web-based, paper-pencil Duration: 45 minutes OMB Approval Status: Some changes since previous approval
Follow-up data collection	Follow-up coach survey	Respondents: Coaches Content: characteristics of coaches Purpose: Capture implementation drivers	Mode: web-based, paper-pencil Duration: 30 minutes OMB Approval Status: No changes since previous approval
Follow-up data collection	Follow-up classroom observation protocol (teacher burden)	Respondents: Teachers Content: Staffing, curricula used, how typical day was; classroom quality dimensions Purpose: examine intervention impacts on quality; understand context around classroom observations	Mode: verbal questions Duration: 18 minutes OMB Approval Status: No changes since previous approval
Follow-up data collection	Follow-up protocol for child assessments (child burden)	Respondents: Children Content: Children’s skills Purpose: examine intervention impacts on child skills	Mode: in-person direct assessment, remote if needed Duration: 57 minutes OMB Approval Status: Some changes since previous approval
Follow-up data collection	Teacher reports to questions about children in classroom (administered as part of the follow-up teacher survey)	Respondents: Teachers Content: Children’s skills Purpose: examine intervention impacts on child skills	Mode: web-based, paper-pencil Duration: 10 minutes/child OMB Approval Status: Some changes since previous approval
Follow-up data collection	Parent/guardian reports to questions about children	Respondents: Parents/guardians Content: Children’s skills Purpose: examine intervention impacts on child skills	Mode: web-based, paper-pencil Duration: 6 minutes OMB Approval Status: No changes since previous approval
Fidelity of Implementation	Teacher/ assistant teacher log	Respondents: Teachers Content: curriculum implementation Purpose: examine fidelity of implementation from teacher perspective	Mode: web-based Duration: 15 minutes/log OMB Approval Status: No changes since previous approval
Fidelity of Implementation	Coach log	Respondents: Coaches Content: coaching and curriculum implementation by classroom on caseload Purpose: examine fidelity of implementation from coach perspective	Mode: web-based Duration: 15 minutes/log OMB Approval Status: No changes since previous approval
Fidelity of Implementation	Implementation fidelity observation protocol (teacher burden)	Respondents: Teachers Content: curriculum implementation Purpose: examine fidelity of implementation from external perspective	Mode: verbal questions Duration: 18 minutes OMB Approval Status: No changes since previous approval
Fidelity of Implementation	Interview/ Focus group protocol (administrator, teacher/ assistant teacher and coach burden)	Respondents: Administrators, teachers, coaches Content: perceptions and experiences with training, coaching, curriculum Purpose: understand different staff experiences with the interventions	Mode: phone or video conference call, in-person if feasible Duration: 1.5 hours OMB Approval Status: No changes since previous approval

Other Data Sources and Uses of Information

When available, we will collect administrative data (e.g., children’s attendance records) from participating centers that cannot be collected through active data collection with respondents as outlined above. This information will be used to understand the amount of curriculum dosage or exposure that children experience to inform the interpretation of impact results.

A3. Use of Information Technology to Reduce Burden

This study will use information technology, when possible, to minimize respondent burden and to collect data efficiently. Electronic data collection methods (e.g., emails to contact study participants, web-based survey instruments and logs) will be used to reduce burden on study participants when possible. Conducting the surveys and logs in this manner means that the respondent can answer questions on their own without coordinating with a member from the study team to complete the instrument. It also allows for efficient administration of a survey/log by using skip logic to quickly move to the next appropriate question, depending upon a respondent’s previous answer. We also anticipate using mixed-mode delivery systems for surveys to allow for ease of completion by permitting respondents to complete the surveys using a paper and pencil format or a web-based system. In both cases, the surveys will be self-administered by the participant.

Electronic informed consent forms (ICFs) will be available to reduce burden on center staff in the distribution and collection of completed ICFs for teachers and parents/guardians. If email addresses are available, participants will receive an email with a link to an electronic version of the consent form. Otherwise, participants will receive cover letters with a link to an electronic version of the consent form that explains the research study and voluntary nature of participation. Participants will then be able to sign the ICF electronically indicating whether they agree to participate in the study.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

In the design of the planned data collection instruments and activities, attention has been paid to leveraging existing or administrative data sources whenever possible. However, as is often the case in ECE settings, there are limited or nonexistent administrative data sources that can reliably and consistently inform the constructs and processes of interest as delineated in the conceptual model underlying the design of the VIQI project. This is because extant data are collected at differing times of the year with methodologies and approaches that vary across states, localities, programs, and centers, making it difficult to collect consistent information across the pooled group of participating centers to address the guiding questions of the VIQI project. As such, we do not make many assumptions about our ability to fruitfully gather crucial information to successfully achieve the research aims and goals of the VIQI project without unique data collection activities.

Further, much of the information collected about centers during the recruitment process is informative for the process study. Our team has worked together to design protocols that will be complementary and informative for both recruitment efforts and the process study, which reduces the degree to which center staff are asked the same or very similar questions at both recruitment and the baseline data collection. We intend to streamline data collection from center staff to reduce duplicity across multiple data collection activities. For instance, if information about center or staff characteristics is collected during recruitment activities, that information will be fed into other protocols such as the baseline surveys or center administrator interviews. That way the same information will not be requested multiple times assuming the information remains the same.

A5. Impact on Small Businesses

We expect some of the participating programs will be independent, small organizations. To minimize the burden of the study on staff, the study team will provide resources for each center to facilitate the designation of liaisons for the study. The study team also will work in partnership with the center staff to identify the best opportunities for administering and collecting information with the data collection instruments to minimize interruption of their routine programming by scheduling and planning visits and activities in conjunction with center leadership.

A6. Consequences of Less Frequent Collection

The planned data collection activities aim to gather information only as frequently as needed to achieve the aims of the study, typically being collected once or twice during the course of the study. Eliminating any of the proposed data collection items would compromise our ability to address key research questions. For instance, coach and teacher logs will be collected on an ongoing basis (up to weekly for teachers and after every coaching session for coaches) to monitor implementation of the interventions and to inform the process study. More frequent collection of logs allows for the strongest intervention possible because the real-time data can inform technical assistance, training, and coaching efforts. In addition, the inclusion of several data points allows for a more robust study of implementation, indicating how implementation changed over time as a result of technical assistance, training, coaching or environmental factors. Providing this level of detail increases the value added of the VIQI project to the ECE field.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on January 6, 2021, Volume 86, Number 3, page 543, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

An expert advisory panel has provided foundational guidance that informed the study design and analysis as well as data collection instruments, particularly the classroom observation tools and child assessments. Members of the expert panel are listed in Table A8.1

Table A8.1 VIQI Expert Panel
Name	Affiliation
Sandra Barrueco	Professor, Catholic University
Karen Bierman	Professor, Pennsylvania State University
Howard Bloom	Consultant, MDRC
Iheoma Iruka	Professor, University of North Carolina at Chapel Hill
Luke Miratrix	Associate Professor, Harvard University
Martha Zaslow	Interim Executive Director, SRCD and Senior Scholar, Child Trends

A9. Tokens of Appreciation

Participation in VIQI will place some burden on families and children. To offset this, we are requesting nominal monetary tokens of appreciation for parents/guardians and children. For parents/guardians, we propose a $10 token of appreciation to be included with the parent/guardian baseline information form in Fall 2021 and a $10 token of appreciation for completing the report to questions about their child in Fall 2021 and Spring 2022. In addition, we propose offering children stickers for attempting the child assessment in Fall 2021 and Spring 2022. Table A9.1 provides an overview of the proposed tokens of appreciation for data collection.

Table A9.1. Tokens of Appreciation by Research Activity

Research Activity	Avg. Length Per Instrument	Amount	Timing
Baseline Parent/Guardian Information Form	6 min	$10	Fall 2021
Parent/Guardian Reports to Questions about Children	6 min	$10	Fall 2021, Spring 2022
Baseline Protocol for Child Assessments	30 min	Stickers	Fall 2021
Follow-up Protocol for Child Assessments	57 min	Stickers	Spring 2022

The goal of these data collections is to capture information regarding the skills and background information on as many children as possible to ensure that participating children and their families are representative of the children being served in participating centers. Thus, given the complex study design, high levels of participation among families and children in study activities are essential to maintain statistical power to detect meaningful effects when measuring participant outcomes. In addition, the integrity of the study’s estimates requires maintaining similar response rates for the randomly assigned intervention and control groups and across demographic groups of interest to the study. Maintaining high response rates will be especially difficult in VIQI because a focus of this study is on children from families with low incomes and minority populations.

In the VIQI pilot, we did not provide a token of appreciation to families, and the response rate was much lower than the rate that will needed to have adequate statistical power in the impact evaluation. MDRC and its subcontractors engaged many of the same centers and localities from the VIQI Pilot in the ExCEL Quality Study, funded by Arnold Ventures, to provide a second year of implementation in the 2019-2020 school year. In that study, families received $10 for completing the baseline parent/guardian information form, which substantially increased the response rates over those seen in the VIQI pilot, from 42% in the VIQI pilot to 68% without differential response across research conditions and demographic characteristics of the sample in ExCEL Quality.

Other similar studies of low-income families and children have successfully used tokens of appreciation to improve survey response. For example, FACES (OMB #0970-0151), which was also conducted in Head Start centers, included $35 tokens of appreciation for 30-minute parent/guardian baseline information forms and 30-minute parent reports to questions about their children to participating families in the 2006, 2009, and 2019 cohorts. In FACES 2014-2018, parent tokens of appreciation were reduced to $15 as a base (with add-ons for a potential of $25 total), which resulted in a large decrease in response rates (from 93.1% to 77.5%) and differential response rates among different demographic groups. As a result, OMB approved an increase back to $35 for FACES 2019. Given that the proposed instruments for the VIQI impact evaluation and process study are similar in nature, but shorter, a proportionately smaller token of appreciation is proposed. Similarly, the Project LAUNCH Cross-Site Evaluation (OMB #0970-0373) had a sample that included early care and education and preschool settings, and did not offer an tokens of appreciation to parents of young children completing a 30-minute, web-based survey, finding that early respondents were not representative of their communities. Minorities, individuals with lower incomes and those who worked part-time or were unemployed were underrepresented. Following OMB approval of a $25 post-pay tokens of appreciation after data collection had started, completion rates and representativeness both improved (LaFauve et al., 2018).

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

Personally identifiable information (PII) will be collected, such as contact information (e.g., name, address, phone numbers, e-mail address) for administrators, teachers, coaches, and parents and children (and in the case of the parent consent, additional contacts that could help the study team find the child/family). The collection of personal identifiers is necessary for participant tracking for follow-up surveys and to allow us to access and match administrative records data.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information (including any audio recording of interviews/focus groups) will be kept private to the extent permitted by law. See Section B4 in Supporting Statement B for more information on how this information will be conveyed. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Due to the sensitive nature of this research (see A.11 for more information), the evaluation will obtain a Certificate of Confidentiality. The study team has been approved for this Certificate and it is attached in Appendix B. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law. The study team has also received IRB approval for this study.

Data Security and Monitoring

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. 

As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.  

A11. Sensitive Information ¹

Our baseline and follow-up surveys of teachers and administrators will contain questions on some sensitive topics, like salary, feelings about the workplace, and work-related stress and burnout. These questions will be answered in a self-administered format, which should minimize discomfort. The introductions to each survey will state that their participation is voluntary, they may skip any questions they do not wish to answer, their answers will be protected to the extent permitted by law, and that their responses will not affect their job. The sensitive questions included in the surveys are necessary for understanding the variability in center and staff characteristics that potentially influence implementation and quality levels achieved in the course of the study. The organizational climate and staff stress and burnout have been linked to lower levels of implementation and classroom quality in previous empirical studies (Han & Weiss, 2005) and, therefore, are critical constructs to measure as implementation drivers.

A12. Burden

Explanation of Burden Estimates

The estimated annual burden for respondents is shown in Table A12.1. We estimated burden for the landscaping and screening instruments by considering the number and type of questions in the protocol and by including some time to allow for respondents to ask questions. We estimated burden for all other protocols by considering the number and type of questions in the protocols. Estimates for the parent and teacher baseline instruments include time for respondents to read through the informed consent form. The estimated annual burden is 7,624 hours.

Estimated Annualized Cost to Respondents

The estimated annual cost for respondents is shown in Exhibit A12.1. The source for the mean hourly wage information for each respondent type is Bureau of Labor Statistics, Occupational Employment and Wages, May 2019.

• For administrators and coaches, the mean hourly wage of $25.81 was used, based on the wage for education and childcare administrators in preschool and child care centers and programs (11-9031 Education and Childcare Administrators, Preschool and Daycare; https://www.bls.gov/oes/current/oes119031.htm).

• For teachers/assistant teachers, we averaged the mean hourly wage for preschool teachers ($16.66) and childcare workers ($12.27), which yielded an estimated teacher/assistant teacher hourly wage of $14.47 (25-2011 Preschool Teachers, Except Special Education; https://www.bls.gov/oes/current/oes252011.htm and 39-9011 Childcare Workers; https://www.bls.gov/oes/current/oes399011.htm). Further, this hourly wage has been adjusted for overtime pay 1.5 times the estimated average teacher/assistant teacher hourly wage to account for the fact that we anticipate teachers and assistant teachers to complete the data collection instruments in hours that fall outside of their typical, standard work hours, assuming that they are working full-time schedules. This brings the estimated average overtime teacher/assistant teacher hourly wage to be $21.71/hour, which is the estimate that is used to determine the estimated pay for time required to complete each instrument.

• The federal minimum wage was used to calculate the hourly wage for parents/guardians.

The total estimated annual cost amount is $124,974.30.

Table A12.1. Annual Burden and Cost Estimates

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instruments for Screening and Recruitment of ECE Centers for the Impact Evaluation and Process Study
Landscaping protocol with stakeholder agencies (staff burden in Head Start (HS) grantee and community-based child care agencies)	120	1	1.50	180	60	$25.81	$1,548.60
Screening protocol for phone calls (staff burden in HS grantees and community-based child care agencies)	132	1	2.0	264	88	$25.81	$2,271.28
Screening protocol for phone calls (HS and community- based child care center staff burden)	336	1	1.2	403	134	$25.81	$3,458.54
Protocol for follow-up calls/in-person visits for screening and recruitment activities (staff burden in HS grantees and community-based child care agencies)	610	1	1.5	915	305	$25.81	$7,872.05
Protocol for follow-up calls/in-person visits for screening and recruitment activities (HS and community- based child care center staff burden)	950	1	1.2	1140	380	$25.81	$9,807.80
Baseline Instruments for the Impact Evaluation and Process Study
Baseline administrator survey with COVID-19 supplemental questions	175	1	0.85	149	50	$25.81	$1,279.75
Baseline teacher/ assistant teacher survey with COVID-19 supplemental questions	1,050	1	0.85	893	298	$21.71	$6,458.73
Baseline coach survey	59	1	0.6	35	12	$25.81	$309.72
Baseline classroom observation protocol	420	1	0.3	126	42	$21.71	$911.82
Baseline parent/ guardian information form	6,300	1	0.1	630	210	$7.25	$1,522.50
Baseline protocol for child assessments	4,200	1	0.5	2,100	700	--
Baseline teacher reports to questions about children in classroom	420	10	0.17	714	238	$21.71	$5,166.98
Parent/guardian reports to questions about children	6,300	1	0.1	630	210	$7.25	$1,522.50
Follow-Up Instruments for Impact Evaluation and Process Study
Follow-up administrator survey	140	1	0.5	70	23	$25.81	$593.63
Follow-up teacher/ assistant teacher survey	840	1	0.75	630	210	$21.71	$4,559.10
Follow-up coach survey	47	1	0.5	24	8	$25.81	$206.48
Follow-up classroom observation protocol	420	3	0.3	378	126	$21.71	$2,735.46
Follow-up protocol for child assessments	3,200	1	0.95	3,040	1,013	--
Follow-up teacher reports to questions about children in classroom	420	10	0.17	714	238	$21.71	$5,166.98
Parent/guardian reports to questions about children	6300	1	0.1	630	210	$7.25	$1,522.50
Fidelity of Implementation Instruments for the Process Study
Teacher/ assistant teacher log	840	36	0.25	7,560	2,520	$21.71	$54,709.20
Coach log	47	108	0.25	1,269	423	$25.81	$10,917.63
Implementation fidelity observation protocol	80	1	0.3	24	8	$21.71	$173.68
Interview/Focus group protocol	236	1	1.5	354	118	$24.44*	$2,883.92
Total				22,872	7,624		$125,598.85

*Note: Interview/focus group will be conducted with administrators, teachers, and coaches and therefore the estimated average hourly wage rate is averaged across those 3 estimated wage categories.

A13. Costs

OMB previously approved honoraria for administrators and teachers for the VIQI project under our approved package (OMB #0970-0508). As in our prior approval, we will provide payments to centers based on the amount of time that staff are expected to coordinate schedules with the research team, support and facilitate data collection activities, and complete research activities, and estimated wages for staff at different levels. The total estimated payment per center is $3,085 (an increase from the approved amount of $2,798).

The honoraria under our approved package (OMB #0970-0508) have been updated to reflect increases in the estimated wages of program administrators and teachers since 2016. Using estimates from the 2019 Bureau of Labor Statistics National Occupational Employment and Wage Estimates, the program or center administrator’s wage is $25.81/hour, so we are requesting increases to honoraria in accordance with that amount (in 2016, the estimate for administrators was $22.83/hour). As in our prior approval, we estimate that one 8-hour day a month for 10 months, for a total of 10 days (estimated to be about $2,065/administrator) will be required to coordinate with the research team.

Up to $1,020 per center would be paid in installments upon completion of certain teacher research activities as denoted in Table A13.1. As in our prior approval, we estimate that a maximum of 7.33 hours per lead teacher and 4 hours per assistant teacher will be required to complete research activities. All of the proposed amounts are either the same as those that were previously approved or based on the same formulas for new instruments. The total amount provided to any given center would be adjusted depending on which teacher research activities are completed and the total number of teachers that completed them. Centers can decide how to distribute these honoraria.

Table A13.1. Honoraria Provided to Respondents

Research Activity	Length	Honorarium Amount	Timing	Estimated Pay for Time Required to Complete Instrument (Based on an Estimated Hourly Wage of $21.71/hour)	Previously Approved?
Baseline Teacher/ Assistant Teacher Survey	30 min	$10/survey	Fall 2021	$10.86/survey	Yes
Baseline Lead teacher reports to questions about children in classroom	100 min (10 min/child)	$4/child (est. 10 children per classroom)	Fall 2021	$3.62/child	No (new instrument)
Teacher/Assistant Teacher COVID-19 Supplemental Survey Questions (incorporated into Baseline Teacher/Assistant Teacher Survey)	15 min	$5/survey	Fall 2021 or Winter 2021/2022, depending on circumstances surrounding COVID-19	$5.43/survey	No (new instrument)
Teacher/Assistant Teacher log	15 min per log	$10/month	Monthly (assumed to be collected for 10 months)	$10.86/month	Yes
Follow-up Teacher/ Assistant Teacher Survey	45 min	$15/survey	Spring 2022	$16.28/survey	Yes
Lead teacher reports to questions about children in classroom	100 min (10 min/child)	$4/child (est. 10 children per classroom)	Spring 2022	$36.18	Yes

A14. Estimated Annualized Costs to the Federal Government

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$200,000
Field Work	$16,000,000
Analysis	$600,000
Publications/Dissemination	$900,000
Total costs over the request period	$17,700,000
Annual costs	$5,900,000

A15. Reasons for changes in burden

The following are key reasons for program changes in burden for the Follow-up Protocol for Child Assessments. These adjustments are being made due to COVID-19 circumstances and reduce the total burden associated with this instrument:

• A change in the amount of time the follow-up child assessment will take, on average. Due to the pandemic, we must train assessors virtually, which limits the types of assessments we can choose to administer at the follow-up timepoint. The language assessment chosen (QUILS) requires that children who speak Spanish at home receive both the English and Spanish versions. Therefore, for the approximately 18 percent of our sample that we anticipate will be Spanish speaking, we will need to assess them on the QUILS in both languages. This increases their assessment time to ~70 minutes. We have updated the burden estimates accordingly to account for 18 percent of the sample receiving a slightly longer assessment time (70 minutes) and the rest of the sample receiving the originally assumed average assessment time (54 minutes); and

• A smaller sample size of classrooms is participating in the impact evaluation and process study, resulting in fewer children eligible for assessments (assuming ~10.6 children/classroom and 301 classrooms);

A16. Timeline

Table A16.1 provides the anticipated time schedule for this information request. Additional time may be required, if data collection is delayed or additional cohorts of data are needed to fulfill the study aims, due to COVID-19. As such, this request is for a three-year extension.

Exhibit A16.1 Anticipated Timeline

Activity	Anticipated Start Date	Anticipated Duration after OMB approval
Baseline instruments	August 15, 2021	3 months
Follow-up instruments	March 1, 2022	3 months
Fidelity of implementation instruments	August 15, 2021	9 months
Conduct analyses	March 1, 2022	9 months
Report/brief	July 1, 2022	12 months
PUF/RAF	January 1, 2023	16 months

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendices

Appendix A: Conceptual Framework

Appendix B: Certificate of Confidentiality

Instruments

Instrument 1: Landscaping Protocol with Stakeholder Agencies and Related Materials 

Instrument 2: Screening Protocol for Phone Calls and Related Materials 

Instrument 3: Protocol for In-person Visits for Screening and Recruitment Activities and Related Materials 

Instrument 4: Baseline Administrator Survey 

Instrument 5: Baseline Teacher Survey 

Instrument 6: Baseline Coach Survey

Instrument 7: Baseline Protocol for Classroom Observations 

Instrument 8: Baseline Parent/Guardian Information Form

Instrument 9: Baseline Protocol for Child Assessments

Instrument 10: Baseline teacher reports to questions about children in classroom

Instrument 11: Administrator/teacher COVID-19 supplemental survey questions

Instrument 12: Parent/guardian reports to questions about children

Instrument 13: Follow-up Administrator Survey

Instrument 14: Follow-up Teacher Survey

Instrument 15: Follow-up Coach Survey

Instrument 16: Follow-up Classroom Observation Protocol

Instrument 17: Follow-up Protocol for Child Assessments

Instrument 18: Follow-up Teacher Reports to Questions about Children in Classroom

Instrument 19: Teacher Log

Instrument 20: Coach Log

Instrument 21: Implementation Fidelity Observation Protocol

Instrument 22: Interview/Focus Group Protocol

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

13