Variations in Implementation of Quality Interventions (VIQI): Examining the Quality-Child Outcomes Relationship in Child Care and Early Education

Attachment B: Semi-Structured Protocol for VIQI Landscaping Discussions

State/Local Version

Guidelines for Use of this Protocol in Landscaping Discussions for the Variations in Implementation of Quality Interventions (VIQI) Study. This protocol will be used to guide a series of semi-structured landscaping discussions with state and local child care and early education (CCEE) informants. In each module, illustrative questions and prompts provide examples of the types of questions that will be asked.

Modules will cover the following topics:

Module A. CCEE Program/Center Structure and Population Served
Module B. Quality and Quality Measures

Module C. Curricula and Professional Development

Module D. Data Infrastructure

Module E. Feasibility of Design Options
Module F. Follow-up/Next Steps

Facilitators will not cover all of the questions in this protocol during every discussion. We are asking informants for a discussion that is expected to last approximately one hour. We may also conduct a second discussion to follow-up and clarify information emerging from the initial discussion. In total, we expect that any given participant will be engaged in a maximum of 1.5 hours of discussion. Facilitators will select and edit questions/prompts from different modules as needed, depending on: 1) the expertise of the informants with whom they are speaking; and 2) the study team’s extant knowledge base accumulated through earlier discussions. Additional questions and prompts (not included in this protocol) may be asked as non-standardized follow-up prompts and to clarify responses provided by informants.

Members of the VIQI study team will serve as facilitators, and will:

1. Introduce members of the study team and ask informants to introduce themselves and describe their roles and responsibilities.

2. Provide a brief overview of the VIQI study and summarize the purpose of the call/visit. (See Attachment E for agenda).

Standard introductory language to be used by facilitators (not for distribution):

• The Administration for Children and Families’ Office of Planning, Research and Development is working with MDRC and its partners on a major new study that will examine how different dimensions and levels of quality in early childhood classrooms relate to children’s developmental outcomes. We plan to partner with multiple localities to conduct this study.

• In the initial planning stages for this study, we are reaching out to leaders in the early childhood field to:

• share more details about the study;

• ask questions to help us better understand the landscape of CCEE services; and

• discuss potential opportunities and challenges that our study team should consider in planning for this study.

• Participation is voluntary, and responses will be kept private to the extent permitted by law.

• An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.
  
  

3. Answer clarifying questions about the purpose of the discussion and study plans.

4. Select questions and prompts from the following modules as needed based on the expertise of the informants
   
   
   

Module A. CCEE Program/Center Structure and Population Served. This module focuses on the structural characteristics of CCEE services in specific localities. Illustrative questions and possible prompts are provided below. Note: Some questions and prompts in this module also appear in Module D (“Data Infrastructure”), but the prompts serve different purposes across the two modules. In Module A, the prompts serve to gather high-level, first-hand insights from experts regarding the landscape of CCEE services in different localities. In Module D, the prompts serve to learn more about the type and depth of information being stored in existing data sources. Asking the prompts in Module D allows us to identify specific data sources that might be helpful to explore to develop a more detailed understanding about the characteristics and publicly-funded child care programs and centers nationally and in different localities.

Introduction to Module A. The VIQI study will be conducted in collaboration with multiple localities. By “locality,” we mean “metropolitan areas.” We will be looking for localities where there are large numbers of Head Start and publicly-funded child care centers serving children from low-income backgrounds, and where there is a documented wide range of quality among centers across settings. In this initial planning phase, we would like to develop a better understanding of the landscape of CCEE services in [locality X], including the structural characteristics of CCEE programs and centers, and the populations of children and families served.

Illustrative Questions and Prompts

• Who are the key stakeholders involved in oversight of publicly-funded child care and Head Start programs in [locality X]?

  • Possible prompts:

    • What are the major nonprofit organizations that operate multiple CCEE centers?

    • Who determines which curricula are used?

    • Who determines what professional development teachers receive?

    • Who are the other key decision-makers or influencers? (e.g., funders, teachers’ unions, heads of early childhood coalitions or advocacy groups)
      
      

• Are there large concentrations of certain types of CCEE programs in [locality X]?

  • Possible prompts: large, multi-site non-profit organizations, for-profit organizations, faith-based organizations
    
    

• What is the age make-up of most CCEE classrooms?

  • Possible prompts:

    • Are two’s, three’s, and four’s in mixed age classrooms or separated based on birth date?

    • Is this different across different CCEE settings? (Head Start, publicly-funded child care?)

    • Are three’s included in public pre-k programs?
      
      

• We need to develop a detailed understanding of the characteristics of Head Start and publicly-funded child care programs and centers in [locality X]. Do you have answers to the following questions, or can you tell us how to obtain the answers?

  • Number of Head Start and publicly-funded child care programs and centers

    • Number of centers per program

    • Number of classrooms per center

  • Length of day (What percentage are half-day vs. full-day?)

  • Demographic data about children served by CCEE programs and centers in a given locality

    • The number of children ages two, three, and four

    • The proportion of children who are from low-income families

    • The proportion of children who are English language learners

    • The proportion of children who have special needs

Module B. Quality and Quality Measures. This module aims to understand how states and localities might assess and measure CCEE quality on an ongoing basis. It further aims to understand how this information is stored and maintained. Note that some of the questions in this module will overlap with the questions in Module D (Data Infrastructure), but this module is designed to focus the discussion on measures of quality, while Module D is designed to gather information about a broader set of data that can inform the VIQI study. Illustrative questions and possible prompts are provided below.

Introduction to Module B. We would like to ask for your insights about common ways in which the quality of CCEE programs and centers is described and measured, and about state/local trends in quality improvement.

Illustrative Questions and Prompts

• The field has identified several basic dimensions of classroom quality that are hypothesized to influence children’s outcomes – namely structural quality (how centers and classrooms are designed and configured) , process quality (quality of children’s interactions with teachers and others in the classroom, including warmth, sensitivity, and overall classroom management and organization, and instructional quality (in terms of intentional teaching of children through organized activities, including scope and sequence of activities within specific domains).

  • Possible prompt: Are there other aspects of quality that locality X (or the state) is particularly interested in?
    
    

• What are important recent, current, or upcoming initiatives to improve CCEE quality at the state and local levels?

  • Possible prompts:

    • Any major recent/upcoming changes to CCEE curricula?

    • Other initiatives led by foundations or regional/state/local early childhood coalitions?
      
      

• Which extant data sources (Quality Rating and Improvement Systems [QRIS], other) can be used to determine levels of quality among CCEE programs and centers in [locality X]?

  • Possible prompts:

    • What data/management information systems (MIS) are used to facilitate monitoring of child care and Head Start programs and centers in [locality X]?

      • Tracking as part of community-based organizations’ services

      • Tracking as a part of a local CCEE initiative

      • Other tools and measures used to monitor curriculum implementation

    • How would you describe the variation in levels of quality among CCEE programs (and centers within those programs) that are included in these monitoring systems? Are there many high AND low quality classrooms?

      • What about the programs and centers that are NOT included in these monitoring systems?

      • How would you describe the variation in levels of quality by type of CCEE program (e.g. Head Start, publicly-funded child care center)?

      • What are the reasons for the variation in quality?

Module C. Curricula and Professional Development. This module focuses on common curricula, curricular enhancements, professional development (PD) initiatives, and other interventions that [locality X] is using in Head Start and publicly-funded child care centers. Illustrative questions and possible prompts are provided below.

Introduction to Module C. In the planning and design phase of the VIQI study, we are gathering information about the curricula, curricular enhancements, professional development initiatives, and other interventions that localities are using with three and four year-olds in Head Start and publicly-funded child care centers. We would like to ask for your thoughts about this topic, and any advice you can offer regarding other stakeholders or resources that might help us with this effort.

Illustrative Questions and Prompts

Curricula and curricular enhancements

• What curricula and curricular enhancements are commonly used within Head Start centers in [locality X]? within publicly-funded child care centers?

  • Possible prompts:

    • Different curricular/interventions for ages two, three and four?

    • Curricular enhancements focused on particular domains (e.g., math, language and literacy, social-emotional skills)?
      
      

• What factors were considered when choosing these curricula/interventions?

  • Possible prompts:

    • Ages of children?

    • Evidence base?

    • Professional development model?

    • Alignment with other existing standards?

    • Cultural context?
      
      

• [For State-level stakeholders] How does choice of intervention/curricula vary from locality to locality and why?
  
  
  
  

Professional Development (PD)

• We are exploring how professional development (or PD) might influence the level of quality in a CCEE center or classroom. What are important recent, current, or upcoming initiatives that focus on teacher PD as a way to improve quality in [locality X]?
  
  

• What does in-service training for CCEE teachers entail?

  • In Head Start? In publicly-funded child care settings?

  • How often, how long, and on what topics?
    
    

• Do teachers receive mentoring or coaching?

  • How often, how long, by whom, and on what topics?

  • Does it involve a classroom visit and/or a sit-down meeting?
    
    

• Do teachers have shared planning time?

  • How often, how long, and with whom?
    
    

• What are some of the challenges that states and localities face with respect to PD for CCEE teachers?

Module D. Data Infrastructure. This module is designed to explore the types of data collected on the state and local levels that can inform the VIQI study. Illustrative questions and possible prompts are provided below. Note: Some questions and prompts in this module also appear in Module A (“CCEE Program/Center Structure and Population Served”), but the prompts serve different purposes across the two modules. In Module A, the prompts serve to gather high-level, first-hand insights from experts regarding the landscape of CCEE services in different localities. In Module D, the prompts serve to learn more about the type and depth of information being stored in existing data sources. Asking the prompts in Module D allows us to identify specific data sources that might be helpful to explore to develop a more detailed understanding about the characteristics and publicly-funded child care programs and centers nationally and in different localities.

Introduction to Module D. One of the goals of the VIQI study is to understand the effects of different dimensions and thresholds of quality on child outcomes and to understand how these effects might vary depending upon the characteristics of the participating CCEE programs and the populations they serve. In the pilot and full-scale phases of this study, we would like to work with localities that have a large number of CCEE centers that represent a wide range of quality. We would like to understand, from your perspective, what types of data already exist that might help our team to better understand the landscape of these programs (and centers within those programs) in [locality X].

Illustrative Questions and Prompts

• What data sources do you think would be helpful in this endeavor? (At the state level? At the local level?)

  • Possible prompts:

    • Quality Rating and Improvement Systems (QRIS)

    • Monitoring data collected by local agencies with oversight of CCEE programs

    • Are there any initiatives or efforts underway to build a CCEE data infrastructure?
      
      

  • Possible prompts for each data source mentioned:

    • Does this data source include names and characteristics of Head Start or publicly-funded child care centers?

• At what level: Individual centers? Programs?

• Does the data source provide information on the following features of programs, centers and population characteristics?

  • Structural characteristics

    • Number of centers in a given locality

    • Number of classrooms per center

    • Length of day (what percentage of centers are half-day vs. full-day?)

    • Grouping of two-four year olds: are they usually in mixed classrooms, or segregated by age?

  • Demographic data about children served by CCEE programs in [locality X]

    • The number of children ages two, three, and four

    • The proportion of children who are from low-income families

    • The proportion of children who are English language learners

    • The proportion of children who have special needs

  • Characteristics of administrators, teachers, and other staff

    • By center or program

  • Curricula and Professional Development

    • Curricula and curricular enhancements used by individual centers

    • Information about professional development (how long, how often, on what topic, etc.)

• How are these data maintained?

  • How frequently are the data updated in this data source?

  • What is the catchment area covered by this data source?

  • What is the level of penetration for this data source?

    • Does the level of penetration differ for Head Start or publicly-funded child care programs?

    • Is participation in this data source voluntary or mandatory?

  • How complete are these data? Are there concerns about missing fields of information in this data source? What might these missing fields be?

  • How accurate is the information included in the data source?

  • How are these data maintained?

  • How can these data sources be accessed? What is the process for accessing the data?

  • Are CCEE management information systems (MIS) within localities usually linked to each other? To other MIS systems (e.g., k-12)? If so, for what purposes?

    • Possible prompt: Are there unique IDs that are used to follow children’s progress from child care through the K-12 system?

• How are MIS data structured to describe quality (e.g., at the center or program level? At the state or local level?)

  • What are the primary measures that are used to assess classroom quality, teacher quality, and child outcomes in CCEE programs and centers in [locality X]?

  • Possible prompts:

    • What are the key dimensions of these measures?

    • On classroom quality, is there anything collected besides the ECERS or the CLASS?

Module E. Feasibility of Design Options. The questions in this module are designed to elicit additional advice from informants about any topics or issues that have not already been covered in the discussion. Illustrative questions and possible prompts are provided below.

Introduction to Module E. We would like to know whether there are any other words of advice you might have for us as we embark on this study – including any additional thoughts about opportunities or challenges we should have on our radar screen, or any other questions you think we should consider.

Illustrative Questions and Prompts

• We are considering a focus on three year-olds in center-based care. We would appreciate your thoughts about focusing on this age group. What are the opportunities/challenges that would come with a focus on threes?
  
  

• We have a particular interest in recruiting a sample of CCEE programs and centers that includes good representation of programs and centers that are both high and low in initial quality to stratify the sample. What opportunities/challenges in [locality X] would come with trying to do so?

  • Possible prompts:

    • How might the existing distribution of initial quality across CCEE programs and centers affect our ability to do so?

    • Might this vary by setting type?

    • Might this vary specifically for CCEE programs and centers serving 3 year olds?

• What are your thoughts about the challenges we may encounter (either in site recruitment, or implementation of quality interventions)?

  • Possible prompt:

    • Do you have any advice for us regarding possible solutions to those challenges?
      
      

• Is [locality X] (or the CCEE programs/centers within [locality X]) participating (or planning to participate) in any other research projects?

Module F. Follow-up/Next Steps. This module focuses on wrapping up the discussion and laying out next steps. Illustrative questions and possible prompts are provided below.

Introduction to Module F. In the last few minutes of this discussion, we would like to discuss next steps, make plans to follow up on items that we have discussed today, and address any other questions you might have for us.

Illustrative Questions and Prompts

• Are there other particular localities that you think might have the following characteristics:

  • A range of program quality

  • Large numbers of Head Start and publicly-funded child care centers serving children from low-income backgrounds

  • Localities that have strong commitment to data collection and quality
    
    

• Do you have suggestions for key contacts in these localities?
  
  

• We want to make sure we speak with the people who can best inform our landscaping and study design. Based on the nature of our questions and interests, who else would you suggest we speak with to inform the study?

  • Possible prompts:

    • Other stakeholders in the CCEE system

    • Other key staff in state/local child care/Head Start programs

    • Experts on curricula and interventions, quality and quality improvement, professional development

    • Experts on CCEE data infrastructure
      
      

• (If appropriate): If a follow-up call or visit is needed, discuss timelines and identify a point person for follow-up.
  
  

• (If appropriate): Would your office be willing to sign a letter of introduction addressed to (insert names of stakeholders as appropriate) describing the VIQI study and asking them to speak with someone from our team about the VIQI study?