NPDP Study - Part B

Case Studies of Current and Former Grantees Under the Title III National Professional Development Program (NPDP)

Supporting Statement for Paperwork Reduction Act Submission

Description of Statistical Methods (Part B)

Contents

1. Sampling Design

2. Procedures for Data Collection

3. Methods to Maximize Response Rate

4. Expert Review and Piloting Procedures

5. Individuals and Organizations Involved in the Project

References

Exhibits

1. Overview of Sampling Criteria

2. Individuals Involved in Project

Supporting Statement for Paperwork Reduction Act Submission

Justification (Part B)

1. Sampling Design

The purpose of the case studies of NPDP grantees is to provide detailed information regarding the pre-service and in-service training approaches used by grantees to improve teacher preparation and readiness for teaching English Learners (ELs). The case studies will provide information about the approaches and strategies being employed, the successes and challenges encountered, the extent to which the grants have affected the overall teacher education program or its partner districts, the extent to which programs are following up with participants and what types of outcomes are being realized. The results can be used to inform planning for future competitions of the NPDP grants and planning for more rigorous testing of particular approaches or more comprehensive study of the grant program. They can also be used to inform the field about approaches to preparing new and current teachers to work with ELs.

Given this background, it is important that the sites selected for case studies represent a diversity of approaches and contexts and use practices that are identified by the literature review, ED, or the expert panel as worthy of further investigation. This will help guard against missing any important lessons that may be unique to particular features or settings, increase the usefulness of the findings for a wider variety of potential recipients, and maximize the chances that some potentially promising best practices will be captured.

Sampling the current grantees

Fifteen sites will be selected from the 139 grantees in the 2007 cohort based on diversity across several aspects of one of the key features identified in the research questions (stage and specialization of the target population) and evidence of an identified practice (See Exhibit 5). The study team will review grantee abstracts and any more detailed descriptions about the programs (e.g., those available through the National Clearinghouse for English Language Acquisition) to learn about the key featuresand will use the literature review and input from the expert panel to identify practices and program characteristics of interest for selection.

First, grantees will be sorted into mutually exclusive categories based on the career stage of the target population. The categories will include those focused primarily on:

1. pre-service teachers;

2. in-service teachers;

3. both pre- and in-service teachers; and

4. other populations such as IHE or district staff.

The third category indicating multiple populations is necessary because slightly more than one-third of the grantees do not focus specifically on either pre-service or in-service. Additionally, because more grantees focus exclusively on in-service teachers than on pre-service teachers, the study team will choose more sites that focus on in-service.

Exhibit 5. Overview of Sampling Criteria

Current Grantees
Primary criteria						Other criteria
Career stage of target population	Number of sites	Specialization of target population	Number of sites	Evidence of identified practices or characteristics
Pre-service teacher candidates	3	BE/ESL	At least 1	Yes		Size/background of the EL population EL population language characteristics
		General education	At least 1
In-service teachers	7	BE/ESL	At least 2
		General education	At least 1
Pre- and in-service teachers	4	BE/ESL	At least 1
		General education	At least 1
Other populations (e.g., IHE or district staff)	1	N/A	N/A
Former Grantees
Primary criteria						Other criteria
Career stage of target population	Number of sites	Specialization of target population	Number of sites	Emphasis on program tracking	Subsequent NPDP award
Pre-service teachers	3	BE/ESL	At least 1	Yes	1 if possible	Size/background of the EL population EL population language characteristics
		General education	At least 1
In-service teachers	3	BE/ESL	At least 1	Yes	1 if possible
		General education	At least 1
Multiple populations	3	BE/ESL	At least 1	Yes	1 if possible
		General education	At least 1

Ultimately, three sites will be selected from the first group, seven sites from the second group, four sites from the third group, and one from the final group. This provides a general representation of the prevalence of these different populations in the 2007 grantees.

Second, within those categories, the study team will give attention where possible to the distinctions between teachers who are specializing (in bilingual education [BE], or English-as-a-Second-Language [ESL]) and those who are training as general education teachers or receiving professional development not leading to a specialized endorsement. The abstracts show that of those grantees that focus exclusively on one of these groups, the large majority are focused on pre- or in-service teachers who are seeking a specialized qualification. Additionally, there are a large number of grantees that include both groups or that are not clearly specified as to this aspect of the target population. Thus, the sampling plan proposes to select a minimum number of grantees with each respective focus but allow the remaining grantees in the overall category to be selected on the basis of other criteria.

Ultimately, within each of the overall target populations identified, at least one site should be selected with a focus on specialists and at least one site should be selected with a focus on general education.

Third, sites will be selected based on evidence of an identified practice. As stated earlier, if an exploratory study is intended (in part) to inform future study, it is important to actively seek programs that have certain practices or characteristics that have been identified by experts, by the literature review, or by ED as potentially effective or important to investigate. The study team will review abstracts and other information about grantees to assess whether they are potentially implementing such practices or have such characteristics. Sites which are potentially implementing such practices or have such characteristics will be prioritized for selection. Based on expert input (first four bullets) and the literature review (last two bullets), identified practices will include those that:

• Assess incoming participants’ knowledge and encourage reflective activities on the part of participants;

• Address teachers’ attitudes towards teaching and working with diverse populations and their own self-identity;

• Utilize observations of master teachers, field work, or mentoring;

• Have IHE working closely with partner school staff (e.g., team teaching);

• Provide content in sheltered instruction, including specific models such as the Sheltered Instruction Observation Protocol (SIOP); and

• Use inquiry models, cultural immersion, or incorporate technology.

Selecting the sites along these primary criteria will narrow the lists significantly. However, if necessary to further prioritize among grantees and make final selections, the following criteria will be considered, in order of importance:

• Size and background of the EL population. The individual states in which grantees operate differ in the size of the EL student population and whether it is a historic or new population. The selected sites should include both those in which ELs historically represent a significant portion of the student population (e.g., California, Nevada, New York, Hawaii, Texas, Florida, Illinois, Arizona, New Jersey, Rhode Island, New Mexico, Illinois, and Massachusetts, where ELs are an above-U.S. average percentage of Pre-Kindergarten to Grade 5 students) (Capps et al., 2005) and those in which the EL population is rapidly growing (e.g., Nevada, Colorado, Iowa, Arkansas, Indiana, Kentucky, Tennessee, Alabama, Georgia, South Carolina, North Carolina, and Illinois, where the EL population has grown over 200% between 1994-95 and 2004-05) (Batalova, Fix, and Murray, 2007).

• EL population language characteristics. The partner districts in which programs operate may vary in terms of which language groups form the majority of the EL population, the degree of language diversity, degree of native language literacy, urbanicity of the population, etc.

Sampling the former grantees

Nine sites will be selected from the 130 2002 and 18 2004 grantees. The sampling approach for the nine former grantees will place a greater emphasis on issues related to follow-up and sustainability and less emphasis on the specifics of program implementation than the sampling of the current grantees.

As with the current grantees, former grantees will sorted into mutually exclusive categories based on the career stage of the target population—though there will just be three categories for the former grantees: those focusing on pre-service teachers, those focusing on in-service teachers, and those including multiple populations. There will be three sites selected from each category.

Within those categories, the study team will distinguish when possible between teachers who are specializing (in BE or ESL) and those who are general education teachers, with at least one site in each category focusing on specialists and at least one site in each category focusing on general education.

Additionally, within each of the overall categories:

• Sites in which follow-up with participants was emphasized will be priorities for inclusion;

• At least one grantee in each target population group will have received a subsequent NPDP award; and

• A range of size and background of the EL population, language characteristics of the EL population, and size of the program (to the extent possible after the other factors are taken into consideration).

1. Procedures for Data Collection

Once AIR has received final OMB clearance, the study team will immediately begin the process of selecting the final sample of 15 current grantees and 9 former grantees as case study sites according to the approved criteria. Once the sample is finalized, the study team will notify each of the selected sites using the introductory letters prepared (see Appendix A) and will follow up with individual sites to make the logistical arrangements. Every effort will be made to minimize the burden on participants by clearly communicating with them and remaining flexible to the changing demands of program schedules and logistics.

Qualitative data collection for this study will seek to be as consistent as possible across sites, while ensuring appropriate depth and detail associated with the unique features of each case.

Prior to all site visits, the study team will:

• Review program evaluations for the selected grantees, as available;

• Solicit quantitatively-oriented information in writing, such as program participation and completion statistics (to be collected ahead of time or at the site visits, as feasible, rather than in interviews); and

• Share an interview topic list with the grantees.

For the current grantees, members of the case study team will conduct a two-day visit in pairs to each of the selected current grantees. Site visits will include:

• Interviews with (a) the IHE-based project director, (b) up to 3 IHE staff involved in delivering the grant, and (c) up to 2 administrative staff (district or school) in partner districts; and

• Focus groups with (a) one group of up to 6 teachers from partner districts who are involved in delivery of the program (e.g., host teachers for practical experiences, mentors, coaches), and (b) two to four groups of up 6 teachers or others who are participating in the program (e.g., pre-service teachers, in-service teachers, and/or IHE staff). Not all categories may apply to all sites and, in some sites, there may be different numbers of participant focus groups, depending on the overall number of participants and respondent types.

All interviews and focus groups will be 60 minutes, with the exception of the project director interview, which is expected to be up to 90 minutes.

Both researchers on each site visit team will attend all interviews and focus groups. All interviews and focus groups will be audio-recorded, and each will have a detailed set of notes or a transcript.

The site visit schedules will be developed in concert with the appropriate staff in the selected program. The same pair of researchers will be responsible for scheduling and conducting the visits, thus allowing a relationship to develop. These interviews will begin in September 2012 since it is likely that the majority of activities will occur in concert with the academic calendar.

For the former grantees, the case study team will conduct telephone interviews with the former director of the selected project and another IHE staff member involved in the delivery of the project. An interviewer and a note-taker from the case study team will be assigned to each interview, which will be audio-recorded. These interviews will be conducted in the late summer, since as no-longer operating programs, they are not subject to the academic calendar.

For both current and former grantees, the case study team will collect and review extant data from the IHE and partner district websites (e.g., relevant background information), as well as request and review from the IHE pertinent documents such as course descriptions, syllabi, participant lists, quantitative information on program participation, and other artifacts.

In preparation for data collection, researchers will review the protocols and tailor them to the specific context of the program based on their review of extant data on the program and through preliminary communication with program staff. In addition, they will consider the reasons for which each site was selected for inclusion in the sample. For example, if a site was selected because of its location in a state with certain certification requirements for teachers of ELs, researchers may make a note to probe further on aspects of teacher preparation at that site.

Throughout the data collection process, quality control procedures will be employed including:

• Weekly meetings, to debrief on site visits as they are ongoing, so that the study team may identify themes as they emerge and make adjustments to data collection procedures as necessary;

• A formal follow up system to ensure that we are collecting the data intended from all the necessary respondent groups from each site; and

• Requirements for site visitors’ timely cleaning and posting of notes and observations to a secure, internal project website, so that task leaders can routinely check for consistency and completeness.

1. Statistical Methodology

This study involves collection of qualitative data. A discussion of statistical methodology is not applicable to this study.

2. Analysis Methods

All documents and interview data will be subjected to qualitative analysis procedures that rely on text recognition software and appropriate coding processes to identify cross-cutting themes and key details of (a) program content and structure including delivery methods, target populations, and content and grade focus, (b) changes in overall teacher education programs, (c) efforts to institutionalize programs, (d) efforts to follow up with participants, and (f) successes and challenges in implementation in each of these areas.

3. Degree of Accuracy Needed

The research team will do everything possible to maximize the accuracy of the data collected for each of the case studies. First, as described below, we will pilot test the interview protocols and revise them to increase the efficiency of the interview process and to ensure that we collect all of the data necessary for the case studies. Second, prior to data collection, we will provide a thorough orientation and training to the members of the site visit teams to familiarize them with the overall study design and their responsibilities for data collection, data handling, and reporting. This orientation will include special attention to the interview protocols, and responsibilities and strategies for tailoring and conducting individual interviews and focus group interviews. Third, all interviews will be recorded for later transcription and data analysis, and interviewers will also take detailed notes during each interview for use in checking the accuracy of the transcriptions.

4. Use of Periodic Data Collection

Data collection will occur only one time.

3. Methods to Maximize Response Rate

Data collection is a complicated process that requires careful planning. AIR has developed interview and data collection protocols that are streamlined and that are designed to place as little burden on respondents as possible. AIR also will pilot and subsequently refine all instruments to ensure they are user-friendly and easily understandable—all of which increases participants’ willingness to participate in the data collection activities and thus increases response rates. Consequently, the contractor anticipates 100 percent response rates from the selected sites.

4. Expert Review and Piloting Procedures

Interview protocols will be pilot tested to determine if they are an appropriate length, if the questions are understood appropriately by respondents, and if we are inadvertently omitting some important topics. AIR will work closely with the Department and the study’s advisors to finalize the protocols to be piloted and identify grantees where they can be tested. AIR will identify two grantees, each with varying characteristics, to conduct the initial pilot through phone interviews (or in person if local). Lessons learned during the piloting process will inform refinements to protocols, as well as procedures for scheduling and conducting interviews.

5. Individuals and Organizations Involved in the Project

AIR is the contractor for the study. Ms. Maria Stephens is the project director (PD); Dr. Meredith Ludwig is a Senior Advisor. There are no subcontractors. Contact information for these individuals is presented in Exhibit 6.

Exhibit 6. Individuals Involved in Project

Contact Name	Organization	Responsibility	Telephone Number
Ms. Maria Stephens	AIR	Project Director	202-386-0863
Dr. Meredith Ludwig	AIR	Senior Advisor	202-403-5246

References

Batalova, J., Fix M., and Murray, J. (2007). Measures of Change: The Demography and Literacy of Adolescent English Learners—A Report to Carnegie Corporation of New York. Washington, DC: Migration Policy Institute.

Capps, R., Fix, M., Murray, J., Ost, J., Passel, J.S., and Herwantoro, S. (2005). The New Demography of America’s Schools; Immigration and the No Child Left Behind Act. Washington, D.C.: The Urban Institute.