SSA_VIQI Formative Generic

Variations in Implementation of Quality Interventions (VIQI): Examining the Quality-Child Outcomes Relationship in Child Care and Early Education

Generic OMB Information Collection Request for Formative Data Collection

0970-0356

Supporting Statement Part A

February 2017

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Ivelisse Martinez-Beck

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks permission for a series of tailored, semi-structured discussions under ACF’s Generic Clearance (GC) for Information Collection (IC) to inform the design of the Variations in Implementation of Quality Interventions (VIQI): Examining the Quality-Child Outcomes Relationship in Child Care and Early Education study. VIQI is a large-scale study sponsored by the Office of Planning, Research and Evaluation (OPRE) that aims to build rigorous evidence about the nature of the quality-child outcome relationship.

This Information Collection Request (ICR) includes the collection of preliminary information about the landscape of child care and early childhood education (CCEE) programs and policy context at national, regional, state, and local levels. The IC is expected to begin upon OMB approval and continue for approximately 13 months. The IC will inform the research design, recruitment and sampling strategies used for the pilot and full-scale phases of VIQI. Table 1 provides a summary of the ICR.

Table 1. Summary of VIQI Generic Clearance Information Collection Request

IC goal	To develop a detailed understanding of the current CCEE landscape, as well as identify potential feasibility and practical challenges and considerations for carrying out VIQI
Intended use of resulting data	To inform the design of the pilot and full-scale phases of VIQI, including its research design, recruitment and sampling strategies, and selection of interventions to be tested. This request does not involve site recruitment for the pilot or full scale phases of VIQI.
Data collection methods	Semi-structured discussions will take place on the phone and in-person. Discussions will be conducted in small group formats when possible, will be facilitated by members of the study team, and will be tailored based on informant expertise, context, and the study team’s extant knowledge gaps about the CCEE landscape.
Populations to be studied	National and regional stakeholders, such as agency administrators, researchers, policymakers and other leaders in the CCEE field State and local stakeholders, such as early childhood administrators and practitioners, local leaders in early childhood education, and local CCEE administrators and practitioners in several large metropolitan areas
Data analysis plan	Data will be systematically reviewed and synthesized by the study team for content and themes including, but not limited to: CCEE program structure and population served Measures of quality and variation in levels of quality across CCEE settings Curricula and professional development CCEE data infrastructure Feasibility of design options

Study Background

Overview of VIQI

Note: This section goes beyond a description of this current ICR by providing background on and plans for the full VIQI project. This ICR is an initial step, with subsequent ICRs expected to be submitted in the future to cover the activities in the remaining phases of the VIQI project.

VIQI is a study sponsored by OPRE through a task order to MDRC and its subcontractors, Abt Associates/Abt SRBI, Frank Porter Graham Child Development Institute, and MEF Associates.

VIQI aims to fill a particular gap in the CCEE literature. To date, consensus in the CCEE field is that classroom quality matters for promoting children’s developmental outcomes. But, there are still many questions about how to design and target investments to ensure that children, particularly low-income children, can benefit from such programs on a large scale.

Notably, the CCEE literature has identified several basic dimensions of classroom quality that are hypothesized to improve children’s outcomes, such as structural quality – how centers and classrooms are designed and configured; process quality – the nature of children’s interactions with teachers and others in the classroom, including warmth, sensitivity, and overall classroom management and organization; and, instructional quality – the intentional teaching of children through organized activities, including scope and sequence of activities within specific content domains. A vast body of nonexperimental evidence portrays an intriguing pattern of correlational findings suggesting that the quality-child outcome relationship may be non-linear and there may be interactions among different dimensions of quality that have synergistic effects on child outcomes. But, the nonexperimental evidence to date has not pinpointed exact thresholds or consistently linked specific interactions with child outcomes.

At the same time, while there are a number of nonexperimental studies exploring the quality-child outcome relationship, there is a relative dearth of rigorous, causal evidence demonstrating that efforts to strengthen CCEE quality will yield improvements in child outcomes. Indeed, to draw policy and practice implications from the literature, these linkages require causal evidence and a stronger evidence base that provides a better understanding of the relationship between classroom quality and child outcomes, the dimensions of quality that are most related to children’s outcomes, and the program and classroom factors that aid delivery of quality teaching and caregiving in CCEE settings. In line with this literature, VIQI aims to tackle these issues by unpacking the “black box” of quality and its drivers in CCEE settings by addressing the following research questions:

• Quality Dimensions: Do different dimensions of quality yield differential effects on child outcomes?

• Quality Thresholds: Do critical thresholds in quality need to be met to promote gains in child outcomes?

• Quality Moderators: Do the effects of quality – different dimensions or thresholds – on child outcomes differ, depending on child, staff and center characteristics?

• Quality Supports: Do initial levels of readiness/quality in centers and classrooms require varying levels of support to benefit from quality improvement efforts?

To address these questions, VIQI aims to test the effectiveness of quality improvement interventions in CCEE settings to rigorously examine how different levels and dimensions of classroom quality relate to children’s developmental outcomes by exploring (1) the causal relationship of different levels and dimensions of quality on child outcomes and (2) how these relationships are moderated by initial levels of quality through a rigorous experimental study design. VIQI will also include an in-depth implementation study to understand the conditions necessary to plan, install, and implement an intervention that will produce changes in the content of instruction, quality of teacher practice, instruction and interactions with children, and child outcomes. CCEE settings may include classrooms in Head Start and child care and public-pre-K programs serving children ages two through four, not yet in kindergarten.

Current phase

VIQI is currently in the planning and design phase where decisions regarding the age of children, child care and early education setting types, intervention(s) to be tested, and other aspects of the design will be finalized. In addition, the recruitment and sampling strategies will be formulated during this phase of the study to ensure that we are able to efficiently and successfully recruit CCEE programs that vary along key characteristics and initial levels of readiness and quality to explore the quality moderators and supports questions laid out above. This planning and design phase will be followed by a year-long pilot study to inform implementation feasibility and further refine the research design for the full-scale evaluation, where multiple measures of classroom quality and child outcomes will be collected.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The IC includes tailored, semi-structured discussions in which the study team will ask a series of questions (using semi-structured protocols) designed to help the team develop a detailed understanding of the current CCEE landscape at national, regional and state levels and in several large metropolitan areas. This IC does not include site recruitment for the pilot- or full-scale studies in VIQI. Subsequent packages requesting clearance for information collected as part of the ensuing pilot and full-scale phases of VIQI will be submitted at later dates.

Purpose

As discussed earlier, VIQI aims to rigorously examine how different levels and dimensions of classroom quality relate to children’s developmental outcomes by exploring (1) the causal relationship of different levels and dimensions of quality on child outcomes and (2) how these relationships are moderated by initial levels of quality through a rigorous experimental study design. To ensure the success of VIQI, the study design and recruitment and sampling strategies need to take into account not only the variation in quality and fragmentation of CCEE programs across most communities and how the current CCEE landscape might present opportunities, but also feasibility and practical challenges and considerations.

Across most localities, CCEE systems are still comprised of a web of federal, state, and private funding streams with multiple program types and delivery mechanisms. There is no uniform service delivery platform providing CCEE services. More often than not, CCEE in any given locality is provided across a mix of organized settings (e.g., Head Start programs, community-based organization, churches and public schools). Moreover, there is considerable variation in quality of CCEE services being provided (Burchinal et al., 2015).

Complicating the portrait of the CCEE landscape is that the characteristics of the settings (such as funding levels, initial levels of quality, quality improvement efforts and use of curricular and professional development models, etc.); characteristics of the workforce (such as teacher and administrator qualifications, training levels, and ability or readiness to deliver specific interventions and developmentally appropriate practices, etc.); and, characteristics of children being taught or cared for in the settings (such as age, race/ethnicity, poverty status, etc.), are linked with each other, and vary systematically by state, locality and setting type. Thus, any study aimed at making headway on disentangling key drivers of quality and types of implementation supports needed to successfully deliver interventions with fidelity in different contexts to causally inform the nature of the quality-child outcome relationship requires a careful and detailed understanding of the landscape of CCEE programming. This is so the research design can be tailored to these operational realities and a sample CCEE programs can be successfully recruited and selected that appropriately captures variation on key drivers of quality.

As such, the purpose of the current ICR is to gather information from national, regional, state and local informants about the current CCEE program and policy context to inform the research design, recruitment and sampling strategies used for the pilot and full-scale phases of a research and evaluation study.

In line with the goals of the ICR outlined above, this IC will focus on the following topics:

• CCEE landscape, including program and center characteristics and structures

• Characteristics of populations served in CCEE settings

• Curricula and professional development

• Measures of quality and variation in levels of quality across CCEE settings

• CCEE data infrastructure

• Feasibility of design options

The remainder of this section describes the study team’s plans for contacting informants and how the information will be used.

Approach

This IC will consist of semi-structured phone/in-person discussions with informants to gather information about the CCEE landscape, program and policy context at the national, regional, state and local levels. The discussions will be conducted in small group format when possible, will be facilitated by members of the VIQI study team (in “facilitator teams”), and will be tailored to the specific background and expertise of informants, depending upon the study team’s remaining gaps in knowledge about the CCEE landscape. We propose to include two groups of informants: (national/regional informants and state/local informants) using two separate, but similar, protocols. We intend to use the information gathered through these discussions to develop a broad understanding of the CCEE landscape, with a focus on large metropolitan areas, to inform the research design, recruitment and sampling strategies used for the pilot and full-scale phases of VIQI.

• National and regional stakeholders such as agency staff at the national and regional levels, researchers, practitioners and leaders in the CCEE field including experts in CCEE policy, early childhood interventions, curricula, implementation of quality-enhancing initiatives, contextual influences, professional development of the early childhood workforce, and quality of CCEE programs; and

• State and local stakeholders such as CCEE agency staff at the state level who are connected to CCEE programs in localities, local agency staff, and local-level CCEE program administrators/practitioners.

We will use two separate protocols to obtain information about similar overall themes, but protocols are differentiated by the level of detail appropriate for the two groups of informants. The specific subset of questions asked will depend on the informant’s expertise and background and the needs of the study at the time of the discussion. For example, a national-level informant might be asked about extant data sources at the national level, and about national or regional initiatives to improve quality, while a local-level informant might be asked about their local CCEE data infrastructure and/or professional development models that are being implemented as part of their local quality enhancement initiatives. We intend to use the information gathered through these discussions to develop a broad understanding of the CCEE landscape, with a focus on large metropolitan areas, to inform the research design, recruitment and sampling strategies used for the pilot and full-scale phases of VIQI.

Research Questions

This IC will explore the following questions:

1. What is the landscape and characteristics of CCEE programs at national, state and local levels, including: a) scale; b) structure and size of programs; c) characteristics of workforce; d) characteristics of population served; e) availability and structure of services for two, three and four year olds; f) setting and community contexts; g) variability in initial quality; h) use of quality improvement efforts, curricula and professional development models; and g) CCEE data infrastructure?

2. How might operational realities of CCEE programs at national, state and local levels present opportunities and challenges for the pilot and full-scale study designs and recruitment and sample selection strategies for the VIQI project, including: a) the feasibility of using existing data to identify and target sample of CCEE programs for recruitment; b) the feasibility of implementing various curricular and professional development models; c) the feasibility of focusing VIQI on a particular age range of children, such as three-year-olds; and d) the feasibility of recruiting and stratifying a sample of CCEE programs that represent a broad spectrum of initial quality levels?

Study Design

The study team will utilize information from two primary sources: (1) semi-structured phone discussions with national and regional CCEE informants (Attachment A); and, (2) semi-structured phone/in-person discussions with state and local informants (Attachment B). The topics to be discussed are described in Table 2. VIQI Design Phase: Information Collection for Landscape Activities and Informant Contribution.

The study team will identify informants through a purposeful, snowball sampling process that draws from recommendations from the Contracting Officer’s Representative (COR) for VIQI, organizations recognized as experts in the CCEE field, other recommendations for state and local experts within large metropolitan areas, and independent internet searches and reviews of existing and publicly available information (such as reports, websites) about the landscape of CCEE programming at national, state and local levels.

Informants will be selected and contacted in waves with opportunities to take stock of the information learned from each wave to guide and refine the priorities and directions for the next wave of conversations. As such, the sampling process will be conducted in an iterative fashion where, after a small set of discussions, the study team will take stock to assess what has been learned from the discussions and reviews of any existing materials (such as reports, agency websites) gathered through internet searches independently conducted by the study team or referred to the study team by informants. At these take stock points, we will identify remaining gaps in our knowledge base in order to identify additional potential informants. The next set of informant candidates will then be selected to participate based on their expertise and the needs of the study and in consultation with OPRE.

We expect that each wave of discussions will include approximately 20 informants, so that the study team can use the information obtained in previous waves to refine its selection of informants on an ongoing basis. We envision a combination of phone and in-person discussions conducted in one-on-one or small group formats, though we will opt to conduct these discussions with small groups of informants (2 – 3 informants per discussion) whenever possible.

We plan to talk with up to 100 state and local informants and up to 40 national and regional informants as part of this IC. In early waves, we plan to prioritize conversations with national and regional informants. We will prioritize conversations with state and local informants in later waves. This prioritization will allow the study team to first understand the landscape at a national and regional level before we gather CCEE landscape information specific to any particular locality or state.

In identifying and selecting informants for each wave of conversations, we will contact individuals who will add new information, based on our extant knowledge base and gaps where we hope to gain further insights, in order to minimize potential burden on IC participants. Priority candidates will be identified based on their field of expertise, purpose of engagement, geographic/locality representation, role in CCEE, type of organization they represent, and other key information.

There are no quantitative components to this study.

Table 2. VIQI Design Phase: Information Collection for Landscape Activities and Informant Contributions

	National and Regional Informants	State and Local Informants
CCEE Program/Center Structure and Population Served	Strategies for obtaining detailed information about CCEE program/center structure in localities	CCEE structure and population served in particular localities
Quality and Quality Measures	National, state and local trends in quality improvement Variations in levels of quality by type of CCEE Variation among commonly- used measurement tools	Current/upcoming initiatives to improve quality Variations in levels of quality by type of CCEE Local participation, utilization and penetration of initiatives to improve quality
Curricula and Professional Development	Common curricula and professional development initiatives Metropolitan areas using various interventions	Details about state/local implementation of curricula and professional development initiatives
Data Infrastructure	National and regional monitoring Monitoring and penetration Linkages between CCEE MIS and other human service MIS systems.	Mandated vs. voluntary monitoring Other monitoring via MIS Dimensions of measurement tools in use Linkages between CCEEMIS and other human service MIS systems
Feasibility of Design Options	Relevance of research questions in the current CCEE context Opportunities and challenges (e.g. with respect to recruitment or implementation of quality interventions)	Relevance of research questions in the current CCEE context Opportunities and challenges (e.g. with respect to recruitment or implementation of quality measures)

Universe of Data Collection Efforts

This ICR includes two data collection protocols:

• Semi-Structured Protocol for VIQI Landscaping Discussions (National/Regional Version): Phone discussions with national and regional informants will be conducted using the semi-structured discussion protocol that is tailored depending upon the expertise and background of the informants and the gaps in the study team’s knowledge base about the CCEE landscape. (See Attachment A.)

• Semi-Structured Protocol for VIQI Landscaping Discussions (State/Local Version): Phone or in-person discussions will be conducted with state and local informants using the semi-structured discussion protocol that is tailored depending upon the expertise and background of the informants and the gaps in the study team’s knowledge base about the CCEE landscape. (See Attachment B.)

For both protocols, the study team will also utilize email templates (see Attachment C: VIQI Phone Discussion Email Templates), a project description (see Attachment D: VIQI Project Description), and an agenda template (see Attachment E: VIQI Landscaping Discussions Sample Agenda) to contact potential informants.

In addition, for both protocols, information is expected to be gathered in a discussion that should last approximately one hour. We may also conduct a second discussion to follow-up and clarify information emerging from the initial discussion. The maximum burden to any one informant is expected to be 1.5 hours, even if two discussions are necessary. As such, the estimate of 1.5 hours per informant and instrument is used to calculate the burden of this IC in Section A12.

A3. Improved Information Technology to Reduce Burden

The study team will conduct internet searches and use publicly available information to update our knowledge base of CCEE structures, key policies, and other contextual information about states and localities, including larger metropolitan areas prior to conducting any discussions. When available, this information will be used to provide context for the study team in advance of semi-structured discussions.

We will also use technology to reduce burden by initially reaching out to potential informants via e-mail, providing information about VIQI and the purpose of the call (See Attachments C and D).

A4. Efforts to Identify Duplication

The IC requirements for this study have been carefully reviewed to determine what information about the CCEE landscape is already available from existing studies and program documents and what will need to be collected. However, to our knowledge, systematic, current information about the CCEE landscape is not readily available. The study team has reviewed the information that is available through the Quality Rating and Improvement System (QRIS) and through other federal, state and local resources on child care and early education. The information needed to inform the final design and methodology of VIQI was not identified.

A5. Involvement of Small Organizations

We expect that up to one-half of the informants will be employees of CCEE centers or umbrella agencies. These programs are often small to mid-size, non-profit organizations. Burden will be minimized for respondents by restricting the discussion length to the extent possible to answer the study team’s questions (maximum of 1.5 hours total), and by conducting telephone and in-person discussions at times convenient for the respondents.

A6. Consequences of Less Frequent Data Collection

The formative data collected through this IC will include preliminary information about the CCEE program and policy context to inform the research design, recruitment, and sampling strategies used for the pilot and full-scale phases of the study. Without the information requested for this phase of the study, it would be difficult to move forward with the design and planning stages (e.g., finalize study sampling design and methodology, select an intervention, select measures and instruments, recruit programs for the pilot study).

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this IC. This notice was published on September 15, 2014, Volume 79, Number 178, page 54985, and provided a sixty-day period for public comment. During the notice and comment period, no comment(s) were received.

Consultation with Experts Outside of the Study

Below is a list of key staff at the Administration for Children and Families who the research team has consulted with to-date.

Office of Child Care
Ellen Wheatley, Deputy Director

Andrew Williams, Director, Policy Division

Megan Campbell, Program Specialist

Office of Early Childhood Development
Amanda Bryans, Director of Research, Analysis & Communication, ECD

Office of Head Start
Beth Meloy, Social Science Analyst, Office of Head Start
Theresa Rowley, Program Specialist/Data Analyst, OHS
Jesse Escobar, Data Analyst, OHS

A9. Incentives for Respondents

No incentives for respondents are proposed for this IC.

A10. Privacy of Respondents

All respondents who participate in the IC under this GC will be read a statement that will explain the study and will inform individuals that their participation is voluntary and of the extent of their privacy as respondents (See Attachments A and B). Participants will be told verbally that their conversations will not be shared outside the study team in a form that identifies them. As ACF’s prime contractor, MDRC plans to implement all data collection activities. If data collection activities are performed by a subcontractor, that subcontractor will maintain the same standards of privacy as required by MDRC. Information will be kept private to the extent permitted by law and in accordance with current federal information security standards and other applicable regulations.

MDRC employees are required to maintain and process qualitative data in designated project folders on the MDRC network. With the exception of the temporary storage of data during onsite collection, MDRC employees are not allowed to download, keep, or process individual-level data on the hard drives of their MDRC work stations or any other storage. Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by an individuals’ personal identifier.

The project Data Manager will organize VIQI project folders and will supervise storage of VIQI data files on a “need-to-know” basis. Data files that contain PII (personal identifying information) data will reside in password-protected secure folders with restricted access. Only the Data Manager and a small number of staff will have access to them. Thereafter, following standard MDRC practice, any data collected will be stripped of PII, if shared outside of the study team. All reports, tables and printed materials will be limited to presentation of aggregate numbers. MDRC will destroy all paper records and electronic records containing PII when no longer needed for research purposes in accordance with funder and contractual requirements, as well as MDRC retention policies.

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

Total Burden Requested Under this Generic Information Collection

Instrument	Total/ Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Semi-Structured Protocol for VIQI Landscaping Discussions (National/ Regional Version)	40	1	1.5	60	$28.54	$1,712.40
Semi-Structured Protocol for VIQI Landscaping Discussions (State/Local Version)	100	1	1.5	150	$28.54	$4,281.00
	140			210		$5,993.40

Total Annual Cost

To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for employees over age 25 with a bachelor’s degree or higher. To calculate the estimated average hourly wage, the median usual weekly earnings of full-time wage and salary workers was divided by the average hours of work per week for persons who usually work full time. According to the Bureau of Labor Statistics’ Current Population Survey 2015, the median usual weekly earnings of employed full-time wage and salary workers over age 25 with a bachelor’s degree or higher is $1,230 (Table A-17. Usual weekly earnings of employed full-time wage and salary workers by educational attainment, age, sex, race, and Hispanic or Latino ethnicity and Non-Hispanic ethnicity, Annual Average 2015). The average hours at work for people who usually work full time over age 25 with a bachelor’s degree or higher is 43.1 (Table 42. Persons at work by actual hours of work at all jobs during the reference week, educational attainment, and sex, Annual Average 2015). The estimated hourly wage is $28.54. The estimated total cost is $5,993.40.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $78,697. Annual costs to the Federal government will be $72,643 for the proposed data collection.

A15. Change in Burden

This is an additional request under generic clearance 0970-0356.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

The information collected will be used to inform the research design, recruitment, and sampling strategies used for the pilot and full-scale phases of the study. Initial calls to the first wave of informants for the purpose of information gathering will take place starting when OMB approval is received and continuing through subsequent waves for approximately 13 months. . The data collected under this IC may be published if it is of methodological interest. Information may be made public if helpful to contextualize research findings from later information collections for the VIQI project. Plans for use of data collected during the study will be further explained in a subsequent package.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this ICR.

13