PPSS_T018_OMBClearance_2.PartB_063015X

JANUARY 2015

Task Order 18: Study on Sustaining the Positive Effects of Preschool

Draft 2: OMB Clearance Request

Prepared for

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

June 2015

Contents

Page

Description of Statistical Methods (Part B) 2

B1. Sampling Design 2

B2. Procedures for Data Collection 3

B3. Methods to Maximize Response Rate 7

B4. Expert Review and Piloting Procedures 7

B5. Individuals and Organizations Involved in the Project 8

Supporting Statement for Paperwork Reduction Act Submission

Description of Statistical Methods (Part B)

B1. Sampling Design

The overall purpose of the case studies is to provide detailed information regarding how sites have succeeded in implementing their successful programs. The study team will therefore select a purposive sample of five sites that show evidence of successful practice for the selected topic(s). Sites will be districts or systems so that it is possible to learn from practices that are implemented more widely than in just one school or program. A site must demonstrate success among its students by showing positive outcomes in cognitive, social-emotional, or academic domains.

Site Selection Procedures. To select sites for case studies, the study team considered sites or programs that emerge from the following sources:

• Descriptions and findings from the research literature

• Recommendations from early childhood experts, including

• American Institutes for Research (AIR) staff working on projects related to preschool and early elementary school programs

• Technical working group (TWG) members

• Program and Policy Studies Service (PPSS), Office of Early Learning (OEL), and Administration for Children and Families (ACF) partners

• State offices of early learning

• National-level associations, including the National Association of Elementary School Principals, the National Association for the Education of Young Children, and the National Institute for Out-of-School Time

To cull a list of potential sites from these sources, AIR first acquired any publicly available documents about the program (e.g., information from Web searches, public reports, and brochures) and summarized information about the programs (e.g., key activities, target grade levels, reported outcomes). Any site that did not meet initial criteria—including topic focus, age group targeted, or length of time program has been in place—was eliminated from consideration. Meanwhile, AIR and the U.S. Department of Education (ED) prioritized a list of potential sites by targeting programs that (a) focused on a larger range of grades, not just prekindergarten and kindergarten; (b) were districts implementing policies or programs across multiple schools or were schools that served as examples of a larger state policy; and (c) had some evidence that the policies or practices were intentionally focused on sustaining the effects of preschool.

The study team looked for performance data and any program implementation information about the prioritized sites from extant sources. When unavailable, the team requested them from the principal or district administrator.¹ Using this information, AIR conducted additional screening to eliminate sites that have not been able to continue the program or policy, shown grade-to-grade decreases in achievement (based on available district or school assessment [e.g., progress monitoring or screening data]), , or have shown obvious marked inconsistency in program implementation. From this list of finalists, AIR will select five final case study sites that will best answer the study’s research questions, working closely with the TWG, PPSS, OEL, and ACF. A back-up list of sites will be developed in case any first-choice sites do not wish to participate in the study. Across the sites, the study team will aim to include variation in approach, geographic location, urbanicity, and other factors that will be informed by the topic(s) of selected focus. Although this sample will be purposive and in no way representative, diversifying the cases as much as possible can provide insights into implementation issues that arise within and across particular contexts.

In order to recruit all five sites, AIR has developed recruitment materials, including an overview of the study that describes activities requested of participants, an introductory letter to district superintendents from ED, and an introductory letter to principals from AIR. These materials will be sent to sites when they are invited to participate. AIR staff will then follow up by phone. Copies of the study’s recruitment materials are included in Appendix A. The study team will complete district research requirements, as needed, before collecting data.

B2. Procedures for Data Collection

Discussions of the data collection procedures of each of the main components of the case study (document reviews and interviews) follow.

Data Collection Activities

Exhibit 8 presents an overview of the data collections planned for this study, with details following.

Exhibit 8. Overview of Data Collection Activities

	Document Review	Interviews	Program Activity Observations
Program characteristics		
Context		
Participants		
Program’s philosophy and model		
Staffing and expenditures		
Program activities and engagement			
Classroom characteristics			
Implementation challenges and solutions		
Staff training		
Program sustainability		
Student outcomes	

Document Review

The study team will collect and review documents in stages as follows:

• During the process of identifying and recruiting potential sites, as described earlier, we will have acquired some publicly available documents describing the program and will have requested from sites some nonpublic documents, including more recent reports or activity descriptions.

• During site visits, site visitors will request other relevant documents available on-site, including training materials, student work samples, organizational charts, and any other information about the program’s goals and results.

Review of all of these documents will be directed by a structured protocol in which analysts will respond to guided questions about information related to the program’s goals and activities, focusing on strategies and successful implementation for the site in sustaining positive outcomes.

Semistructured Interviews

At each site, site visitors will interview up to five program staff, five program teachers, and the program funder (e.g., from a foundation or private funding agency) and program evaluator, if applicable. AIR has designed interview protocols to be semistructured, guiding respondents to comment on questions of interest, but also maintaining a conversational tone to capture breadth of information. All interviews will be audio recorded and later transcribed to facilitate coding.

Interview protocols (included in Appendices C–I) include questions about key constructs of interest, including characteristics of the program; the resources, personnel, staff characteristics, and/or training that facilitate implementation of the program; and program sustainability. AIR has developed a separate set of interview protocols for each respondent type. There is a pre-interview survey for district staff and principals (Appendix C), as well as interview protocols to be used with (Appendix D) district staff, (Appendix E) principals, (Appendix F) elementary teachers or other program staff, (Appendix G) preschool teachers, (Appendix H) funders, and (Appendix I) program evaluators. Teacher interviews may be conducted as individual interviews or focus groups, as determined by the schedules and roles of the selected interviewees. Timing will be determined in consultation with each school’s principal based on the teachers’ schedules. Each protocol will include questions that apply across all both topics of focus—differentiated instruction and PK–3 alignment—in addition to questions that focus specifically on each of the topics to be asked as appropriate depending on the program.

Program Activity Observations

Case study site visits may include observations of classroom instruction, teacher meetings, and other activities relevant to the program being implemented. These observations will be guided by structured observation rubrics, but these are not included in the OMB package because they do not create additional burden for participants. Observation forms will be customized to each of the five programs and depend on the age-range and setting of the program.

Institutional Review Board (IRB) Approval

All data collection protocols will be submitted to AIR’s IRB for approval. AIR’s IRB, which is registered with the Office of Human Research Protection and operates under a federalwide assurance comparable to many universities, is responsible for reviewing all research conducted by the organization and its subcontractors. The IRB ensures that projects involving human subjects comply with professional standards and government regulations designed to safeguard participants and that research team members, including subcontractor staff, are adequately trained. The criteria that a study must meet for AIR’s IRB approval include assurances that risks to participants are minimized and balanced by benefits, subject selection is equitable, participants are informed about risks and give uncoerced consent, privacy of the subjects is adequately protected, and the rights and welfare of populations that might be vulnerable to coercion or undue influence are protected.

Conducting Site Visits

Data collection will take place over approximately a one-month period in the fall of the 2015–16 school year. Before data collection, AIR will train site visitors and carefully plan with each site the schedule and logistics of the visit.

Training

Before site visits are conducted, all members of the case study team will participate in a one-day in-person or webinar-based training to ensure that they thoroughly understand the content of the protocols and site visit procedures. Staff members will review best practices for interviews, in addition to conducting practice interviews. Staff will discuss strategies for (1) avoiding leading questions, (2) ensuring consistency, and (3) conducting interviews in a way that is conversational, yet still directed toward collecting the intended information systematically. Important procedural issues to be addressed include guidelines for ensuring respondent privacy. For observations, we will thoroughly introduce and practice the protocol to be used, making sure that observers understand the goal of each item and section. Observers will use video clips to practice taking high-quality observation notes and making summaries or ratings.

Logistics

Each site visit will likely take two days. Two researchers will visit each site. This approach will allow two interviews to be conducted simultaneously, as needed, to minimize scheduling burden on respondents. The two site visitors will both be present for any focus groups, with one researcher serving as the lead interviewer and the other researcher serving as the secondary interviewer, asking probes or encouraging additional participants to respond. All interviews and focus groups will be recorded for later transcription. The two researchers will co-observe in order to compare and debrief about activity observations. Site visit schedules will be developed in concert with the appropriate staff at the selected sites. The same pair of researchers will be responsible for scheduling and conducting visits, thus developing a rapport with staff at each site. A sample site visit schedule, reflecting a visit to one school, is presented in Exhibit 9.

Exhibit 9. Sample Site Visit Schedule

Day/Time	Site Visitor 1	Site Visitor 2
Before site visit, by phone	Conduct interview with superintendent	Conduct interview with district curriculum director
Day 1, 8:30–9:30	Conduct interview with principal at School 1	Support principal interview; ask follow-up questions as needed, take brief notes
Day 1, 9:30-10:30	Conduct interview with program evaluator	Support program evaluator interview; ask follow-up questions as needed, take brief notes
Day 1, 10:30- 12:00	Observe Classroom Activities
Day 1, 12:00–1:00	Lunch break
Day 1, 1:00–2:00	Conduct focus group with preschool teachers at/affiliated with School 1	Conduct interview with preschool center director
Day 1, 2:50–3:50	Conduct focus group with elementary teachers at School 1	Support teacher focus group; ask follow-up questions as needed, take brief notes
Day 2, 9:00–10:00	Conduct interview with principal at School 2	Support principal interview; ask follow-up questions as needed, take brief notes
Day 2, 10:00–11:00	Conduct interview with elementary teacher 1	Conduct interview with elementary teacher 2
Day 2, 11:00–12:00	Conduct interview with elementary teacher 3	Conduct interview with other key program staff (e.g., coach)
Day 2, 12:00–1:00	Lunch break
Day 2, 1:00–2:00	Conduct focus group with preschool teachers at or affiliated with School 2	Conduct interview with preschool center director
Day 2, 2:15-3:15	Observe Teacher Meeting
After site visit, by phone	Conduct interview with funder	Support funder interview; ask follow-up questions as needed

B3. Methods to Maximize Response Rate

Data collection is a complicated process that requires careful planning. The research team has developed interview and observation protocols that are appropriately tailored to the respondent group and are designed to place as little burden on respondents as possible. The team will also pilot core data collection instruments with similar respondents in schools local to AIR’s offices to ensure that they are user‑friendly and easily understandable, all of which increases participants’ willingness to participate in the data collection activities and thus increases response rates.

In addition to careful instrument design, a high response rate among the case study sites may be ensured through careful recruitment materials. These recruitment materials emphasize the social incentive to respondents by stressing the importance of the data collections as part of a study of innovative practices that will provide much-needed information to districts and schools. AIR’s experience in past evaluations has demonstrated the importance and value of building a consensus of support with participating districts. This level of consensus leads to districts and schools that have the capacity, willingness, and commitment to cooperate fully with the research and data collection responsibilities. Investing in site development at the front end reduces problems at the back end, helping to ensure smooth implementation of the study.

B4. Expert Review and Piloting Procedures

Following the conclusion of 60-day public comment, the study team pilot tested case study protocols with seven participants. Through phone conversations and/or email exchanges in which piloters provided written feedback on questions and/or the pre-interview survey, we asked for feedback on questions that were confusing and incorporated changes in final versions. In response to these pilots, the research team has made several changes to the interview protocols and pre-interview survey to ensure questions are clear to interviewees:

• Clarifying what we mean by “preschool” on the pre-interview survey by adding in parentheses the specific types of programs we are requesting information about

• Reminding interviewees earlier in the interview protocol what the purpose of the interview is

• Adding additional language to clarify what we are interested in regarding communications with and questions from parents

• Breaking up questions about alignment of standards, curriculum, and other instructional materials into separate questions

• Adding a question regarding gains for particular student groups, in addition to the question about outcomes for all students

• Making additional small wording changes to ensure clarity

B5. Individuals and Organizations Involved in the Project

AIR is the contractor for Task Order 18: Sustaining the Positive Effects of Preschool. The project director is Dr. Kathryn Drummond, who is supported by an experienced team of researchers that will guide and conduct the case study data collection and analysis (see Exhibit 10 for a list of key staff involved in the project, their roles, and contact information).

Exhibit 10. Key Staff Involved in the Project

Role	Organization	Contact Name	Telephone Number
Project Director	AIR	Dr. Kathryn Drummond	541-521-8517
Case Studies Task Leader	AIR	Karen Manship	650-843-8198
Site Visitor	AIR	Jennifer Anthony	650-843-8101
Site Visitor	AIR	Connie Chandra	650-843-8170
Quality Assurance Monitor	AIR	Dr. Eboni Howard	312-588-7339
Quality Assurance Monitor	AIR	Dr. Kerstin LeFloch	202-403-5649

1 In keeping with Office of Management and Budget requirements, we limited requests for additional information to fewer than 10 respondents.

American Institutes for Research
1000 Thomas Jefferson Street NW
Washington, DC 20007-3835
202.403.5000 | TTY 877.334.3499

www.air.org

Copyright © 2015 American Institutes for Research. All rights reserved.

1875_06/15