1875-new 4763

Contract # ED-CFO-10-A-0110-0001

21^st Century Community Learning Centers: Lessons Learned Guides

Deliverable 3.5 (2 of 2): First Draft OMB Package

Supporting Statement for Paperwork Reduction Act “B”

January 16, 2011

Revised January 31, 2012

Revised February 6, 2012

Revised February 7, 2012

Prepared for

Prepared by

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The sampling approach for the study is designed to support three key study objectives. The first objective is to identify 21^st Century Community Learning Centers (21^st CCLC) sites that are implementing curricula, enrichment activities, and strategies in four topic areas. These topic areas are Science, Technology, Engineering, and Math (STEM), English Learners (EL), Career and Technical Education (CTE), and structures to increase learning time (ILT). The second objective is to conduct site visits to those programs to investigate and document the practices. The third objective is to write “Lessons Learned Guides” for practitioners that include site descriptions and cross-site analyses of practices. The study will be conducted using a case study approach, with the goal of providing an in-depth description of program practices, rather than a statistical approach, which would require random sampling. To that end, we have chosen a sampling process that will identify 15 21^st CCLC sites with a demonstrated focus on each topic area and that are diverse with regard to geographic region and program characteristics.

Sampling Procedure

The sampling procedure will include two steps: (1) identifying the most promising 21^st CCLC sites for the study and (2) selecting interview respondents from among site personnel.

Site Selection

The research team will use six interrelated channels for nominations or vetting to identify potential sites:

1. Nominations from the TWG

2. Nominations from state coordinators

3. Review of information from recent 21^st CCLC and afterschool conference agendas / Contact up to 9 national intermediary organizations for nominations

4. Vetting against available PPICS data

5. Consultation with state coordinators

6. Screening tool completed by project directors/site coordinators

The initial channel for identifying sites consists of nominations from the members of the technical working group (TWG). Based on a U.S. Department of Education-generated list of practices and an initial review and synthesis of What Works Clearinghouse materials, the research team will create a brief synopsis of the main practices to be studied in each of the four topic areas. These lists will be reviewed with the TWG, and additions or alterations to each list will be made based on their input. After the lists are discussed with the TWG, the research team will ask the members of the TWG whether they would like to nominate 21^st CCLC programs that demonstrate the practices described. TWG members will have the opportunity to make nominations directly in the meeting or in subsequent correspondence after TWG members have had time to consider further.

A second channel for identifying sites will consist of nominations from the state coordinators. The 21^st CCLC program office will distribute to the state coordinators the synopsis of main practices in each topic area created with the TWG, and the coordinators will be invited to nominate programs that demonstrate the practices described. To ensure a broad array of nominations, state coordinators will be encouraged to supplement their own program knowledge in determining site nominations and to reach out broadly to their networks, including communication with state technical assistance providers, evaluators, and other non–21^st CCLC afterschool networks, such as the C.S. Mott Statewide Afterschool Networks.

To supplement these nominations, the research team will pursue a third channel for program identification consisting of a review of recent 21^st CCLC and afterschool conference agendas as well as nominations from nine or fewer national afterschool intermediary organizations, who provided technical assistance and professional development to 21^st CCLC and other afterschool programs. Recent afterschool conference agendas might be a valuable source of program identification, as those subgrantees accepted for presentations at conferences often have developed innovations, program activities, or curricula in the areas of focus. The research team will review programs that were presented at recent 21^st CCLC Summer Institutes and similar state-level conferences to identify those that appear to be providing strong and/or innovative programming in the topic areas. Likewise, national intermediary organizations may be a valuable source of program identification as they often work closely with individual programs and can offer a useful window into where some of the most innovative program work is occurring.

Based on the combination of sites nominated by the TWG, sites nominated by the state coordinators, and sites identified through conference agendas and national intermediary organizations, the research team anticipates identifying at least 20-30 potential programs in each topic area. Researchers will examine data from the 21^st CCLC Profile and Performance Information Collection System (PPICS) on each of these sites. Specifically, researchers will focus on information that grantees have entered in the activity section of their profile, program partners, outside funding beyond 21^st CCLC, and the number of total program enrollees and percentages of regular participants, among other categories. The research team will judge whether the profile reflects an active program dedicated to the topic area in question. At this point, programs that seem not to meet this criterion may be eliminated.

Upon completion of the PPICS vetting, a revised list of sites will be developed consisting of sites nominated by state coordinators or national organizations or identified in the review of conference agendas and press reports, all of which have suitable PPICS profiles. This list of sites will be shared with the appropriate state coordinators for consultation through the 21^st CCLC program office. The state coordinators will be asked whether each program is currently in good standing. In cases where the state coordinator indicates that a program is not in good standing, that program will be eliminated from the list.

The research team will initiate direct program contact as the final channel for the selection of sites for visitation. A screening tool will be sent to program directors or site coordinators, as appropriate, which will include a limited number of questions about the kinds of practices that are in place and any outcomes the program has documented. Pending approval from the Office of Management and Budget (OMB), the survey will be made available online. After reviewing the survey and interview responses, the research team will select 15 sites for visitation and 5 sites as alternates in case of changes in scheduling or availability. The research team’s goal is to select sites that demonstrate a strong programmatic focus or interest in one of the topic areas; an ability to demonstrate or describe deliberate practices, strategies, and goals specific to that area; identifiable innovative or original practices in that area; and potential evidence or documentation of success in that area.

The following diagram summarizes the recruitment of 21^st CCLC sites as described above:

Figure 1

Summary of Recruitment of 21^st CCLC Sites

Interviewee selection

During two-day site visits, researchers will conduct interviews with a selection of 21^st CCLC and affiliated personnel, including (1) the project director, (2) the site coordinator, (3) 21^st CCLC staff members, (4) the school principal, (5) classroom teachers, and (6) community partners (where applicable). The exact individuals to be interviewed (i.e., which 21^st CCLC staff members and how many or which classroom teachers and how many) will be determined in coordination with the 21^st CCLC project director and the school principal prior to the site visit and will vary depending on the conditions and circumstances at each site. In general, researchers will work with the project director to select a diverse array of staff for interviews, i.e., staff who serve in differing roles, who work with different aged participants, who bring differing instructional backgrounds, or who have served in the program for differing lengths of time, depending on the individual program circumstances. The resulting interview schedule will be designed to facilitate understanding of how the topic area is implemented in the 21^st CCLC site with triangulation of data from multiple informant perspectives. It will not support a probability sample that would provide accurate estimates for the entire nation or for the states or districts from which the sample is drawn. Because this is a case study to document practices and engage in an in-depth analysis of program activities rather than a statistical estimation of population parameters, power estimates are not relevant.

2. Procedures for the Collection of Information

The procedures for the information collection include (1) procedures for obtaining cooperation from the 21^st CCLC sites and the states and school districts in which they reside, (2) procedures for training study personnel, (3) scheduling and logistical arrangements for data collection, and (4) on-site and off-site data collection procedures for conducting interviews with site personnel and other affiliated individuals.

Procedures for Obtaining Cooperation from 21^st CCLC Sites, States, and School Districts

Several parties from whom cooperation must be obtained for this study will be previously involved in aspects of the site selection process. These parties would include the 21^st CCLC state coordinators, who will have input into the identification of potential sites, and 21^st CCLC project directors, who will be asked to return an online screening tool providing researchers with information about their sites. This prior involvement will help to dispose both state coordinators and potential 21^st CCLC project directors toward cooperation in this study.

Upon completion of the site selection, the formal process for obtaining cooperation will begin. The first step will involve sending notification letters from the Department to appropriate parties that fully explain the study’s procedure, purpose, and benefits; the letter will also inquire whether there are local Institutional Review Board (IRB) clearances that the research team will need to obtain in addition to federal IRB and OMB clearances. The letter will be personally addressed to the chief state school officer, the 21^st CCLC state coordinator, the superintendents of the school districts in which the selected sites reside, the principal of the school with which the 21^st CCLC program is affiliated, and the 21^st CCLC program project director. Under ESEA, Sec. 9306(a)(4), 21^st CCLC subgrantees are required to participate in research, and project directors will be made aware of this requirement as part of their notification. Furthermore, this study is intended to highlight innovative practices, so it is the research team’s anticipation that sites will want to participate. Based on these factors, a 100% response rate is anticipated. If there are instances of nonresponse, the research team will move on to alternate sites as identified in the selection process (see Section #1 above) until a 100% response rate is reached.

Subsequent to the notification letters, researchers will send a follow-up e-mail to the 21^st CCLC project directors along with a link to an online information survey requesting data and materials in advance of the site visit. In order to reduce data burden to the sites, the online information survey will be pre-populated wherever possible with information from the screening tool completed by the program during the site selection process; project directors will have the opportunity to approve or change information that has been pre-populated and to add information that was not covered in the screening tool. In addition to the follow-up email, researchers will follow up with each contact by phone in order to identify the list of key individuals who will participate in the study, to coordinate interview scheduling, and to check on the status of program materials requested in the information survey.

Training for Study Personnel

No more than four months prior to initiation of the data collection, the principal investigator will train field visitors to conduct on-site interviews and observations. The in-person training session will cover the use of the site visit protocols; overall site visit logistics, procedures, and policies; and general data collection, data coding, and quality assurance procedures. By the end of training, field visitors will be equipped with the professional skill set to represent Manhattan Strategy Group (MSG) in the field and will have all requisite knowledge about the study, its background, and its objectives. All site visitors will also obtain appropriate security clearances from the Department before beginning fieldwork.

Scheduling and Logistical Arrangements for Data Collection

Scheduling and logistical arrangements for field visits will be handled primarily by MSG’s topic- area subject matter experts. Working collaboratively with each site’s project director or designated contact, the subject matter experts will arrange dates for two-day site visits by two-member interview teams for each site. Field visit dates will be scheduled at the convenience of sites in a way to allow for the control of travel costs and to complete the data collection in a timely manner. Field visit scheduling will be arranged to accommodate pre-established commitments on the part of each site (e.g., field trips, special events) and will be sensitive to concerns about interruptions and loss of instructional time. Field visitors will receive their assignments two to three weeks in advance, along with data collected from the project director’s information survey. Sixty site visits will be completed in a seven-month period. The length of the period is due to the unique operating schedules of most 21^st CCLC programs, which typically do not operate at the very end of the school year (May/June) or the very beginning of the school year (August/September).

Each two-person site visit team will have a designated lead, who will contact the school prior to the site visit date in order to determine if there are any last-minute challenges that need to be addressed or logistical considerations that may have arisen. The lead is also responsible for preparing materials for the field visit and facilitating data collection on the date of administration.

On- and Off-Site Data Collection

Previsit interviews with state coordinators will be conducted by phone one to two weeks prior to the time of the first site visit in their state. The appropriate topic area subject matter expert or experts will have a previsit phone conversation with the appropriate state coordinator to gather relevant context and information about the site. Each phone interview is anticipated to be no more than 30 minutes in length. However, this time may increase in cases where a single state coordinator is being interviewed about a number of sites in the state, either in the same topic area or across topic areas.

Two trained field visitors will conduct the field interviews and observations of program activities on-site over a two-day period. The team will arrive at each site on the first day scheduled and will follow any specific instructions for checking in at the program. The site visit team will follow the schedule of interviews and observations prearranged by the project director and the site visit lead.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Subgrantees are required to participate in the study under ESEA, Sec. 9306(a)(4); however, the study is designed to minimize the burden on selected sites as a means of maximizing response rates. Site visit schedules will be determined through a coordinated effort between the researchers and the 21^st CCLC staff to accommodate scheduling preferences, and efforts will be made, through the use of online technology and deliberate protocol design, to limit the hours burden on sites before and during the visits. The study is not designed to support statistical estimation of population parameters, thus nonresponse is not a statistical concern. Nevertheless, in those cases where selected sites prove nonresponsive at the outset or become nonresponsive during the preparations for the site visit, despite their requirement to participate, researchers will have a ready pool of up to five alternate sites within each topic area chosen during the final phase of the site selection process. Individual personnel who decline to participate in the study at a given site will be replaced with other staff members as determined in consultation with the project director.

4. Tests of Procedures or Methods

In order to test procedures and methods for this study, MSG convened a 12-member TWG of individuals with expertise on 21^st CCLC programs, including researchers, practitioners, and administrators on December 12^th and 13^th, 2011. The purpose of these consultations was in part to verify the study design and methods, to assess the clarity of interview items, and to minimize respondent burden. The experts convened served either as part of a core TWG or as part of a topic area-specific TWG. Members of the core TWG provided comments on site visit protocols and site visit selection criteria, as well as on nonprocedural and nonmethodological items. Members of the topic area-specific TWG participated in reviewing procedural or methodological documents and plans within their content area, as well as nonprocedural and nonmethodological items. After the meetings, members of the research team reviewed the oral and written suggestions made by TWG members, provided a summary of suggested changes to the Department, and revised the protocols and other project documents in line with recommendations from the TWG.

MSG also conducted pilot tests of the interview protocols at three 21^st CCLC subgrantees, two in New York on January 10^th and 11^th, 2012 and one in Illinois on January 11^th, 2012. The subgrantees participated on a purely voluntary basis and were drawn from the network of programs with which MSG and its partner firm, American Institutes for Research (AIR), have existing relationships. Members of the research team visited the program for one day to conduct a trial run with the interview protocols. Based on these experiences, the research team determined where questions needed to be altered, eliminated, moved, or added to ensure that data is collected as efficiently and effectively as possible. Based on their participation in the pilot testing, the subgrantees that volunteered to be visited for the pilot will be deemed ineligible for participation in the study proper.

5. Individuals Collecting and Analyzing Data

Collection and analysis of data will be led by MSG in coordination with AIR. The key members of the research team are as follows:

Table 1

Individuals Collecting and Analyzing Data

Category	Team Member	Role
Key Staff	Project Director	David Kornhaber (MSG)
	Asst. Project Director	Jaime Stephanidis (AIR)
	Principal Investigator	Priscilla Little (MSG)
Topic Area Subject Matter Experts	STEM Subject Matter Expert	Lauren Amos (AIR)
	EL Subject Matter Expert	Jimena Quiroga (MSG)
	CTE Subject Matter Expert	Kelly Sparks (AIR)
	ILT Subject Matter Expert	Priscilla Little (MSG)
Additional Key Personnel	Senior Advisor	Robert Stonehill (AIR)
	Data Analysis Subject Matter Expert	Neil Naftzger (AIR)
	Technical Writer/Site Visitor	Sara Hill (MSG)