Description of Statistical Methods (Part B)
The research team will conduct interviews with officials in 50 states, Puerto Rico, and Washington, DC, as well as in 75 school districts nominated by the respective states as being on the “leading edge” of contending with and addressing the challenges of equitable distribution (3 districts nominated by each state, with 2 selected from each large state and 1 selected from each smaller/rural state for a total of 75). To assist in the nomination of districts, the contractor will share the district protocol with the state to ensure that they understand the types of questions that will be asked. The district sample consists of 75 leading-edge districts because those are ones that are most likely to be able to provide answers to the evaluation questions such as “What actions do LEAs report to analyze the distribution of teacher quality and to plan and monitor progress,” “What actions do LEAs report to develop new measures of teacher quality, and how do they use these measures to examine the distribution of teacher quality?” and “What strategies and actions do LEAs report to make the distribution of teachers more equitable across schools?
The contractor anticipates response rates of 100 percent for the states and 85 percent for the school districts. AIR has had success in previous studies in obtaining a 100 percent response rate for state officials (e.g., Study of State Implementation of No Child Left Behind). When seeking information from district officials, response rates have been lower. AIR’s district survey response rates were 93 percent and 96 percent, respectively, for the Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability and the National Longitudinal Study of No Child Left Behind. AIR’s district telephone interview response rate was 88 percent for the National Evaluation of the Eisenhower Professional Development Program.
Methods to Address Non-Cooperation
In the process of identifying state officials who would be the most appropriate respondents, the research team also will identify backup respondents in the event that those contacted initially are unwilling or unable to be interviewed. The compilation of backup respondent options will enable the contractor to ensure a strong response rate as well as complete and valid data from each state and school district.
The data collections for this study include telephone interviews with states and LEAs. Interviewers will prepare for each data collection by reviewing extant documents. Below are the extant documents to be reviewed and the procedures for each data collection.
Extant Documents
In preparation for the interviews, research team members will examine the following extant data sources for information related to ensuring teacher quality for disadvantaged students:
Consolidated state performance reports (CSPRs). In completing CSPRs annually, states are required to provide the percentage of classes taught by HQTs, disaggregated by school poverty level. The CSPR form does not contain prompts for information about additional indicators of teacher quality, or for disaggregation using variables other than school poverty level.
State and district report cards. The percentage of classes taught by HQTs, disaggregated by school poverty level, is also a requirement for state and LEA report cards. Although report cards may contain information about additional teacher quality indicators or further analyses of differences in teacher quality, providing such information was not a practice in the report cards from a small number of states and districts AIR selected for preliminary review.
Monitoring reports. ED’s Office of Elementary and Secondary Education (OESE) gathers information on compliance with the equitable distribution provisions of ESEA through monitoring visits for the ESEA Improving Teacher Quality State Grants Program (Title II, Part A). OESE completed its first round of visits to all states between 2004 and 2006. The current round of monitoring visits began in 2008 and will be complete by the end of 2010. AIR anticipates that all but a few monitoring reports from the second round of visits will be available for review online in advance of the state interviews, though most reports will be from 2008 or 2009. The information from monitoring reports that is relevant to this study addresses (a) compliance with the CSPR and report card requirements described above; (b) compliance with the requirement that the state have an equity plan; and (c) compliance with the requirement that the state ensures that LEA applications include the required assurance regarding teacher quality for disadvantaged students. Note that, in conjunction with each state visit, OESE calls or visits approximately three local education agencies (LEAs). Findings from visits to those LEAs are integrated into each state monitoring report, which name the LEAs but do not allow linkage of particular findings to individual LEAs.
State equity plans. To receive ESEA funds, each state was required to submit a highly qualified teacher plan, including a state equity plan, which, according to ED guidelines issued in March 2006, needed to:
Identify where inequities in teacher assignment exist
Delineate specific strategies for addressing inequities in teacher assignment
Provide evidence for the probable success of the strategies it includes
Indicate how the state will examine the issue of equitable teacher assignment when it monitors LEAs
The equity plans thus focus on several key evaluation questions for this study. AIR’s review of four equity plans found that all described the percentage of teachers who were inexperienced, by school poverty level, and defined what inexperienced meant. In addition, all disaggregated results by school poverty level and school minority level. AIR’s review also identified some limitations of equity plans as a data source; for example, in two of the four plans, the descriptions of strategies for addressing inequities did not explicitly indicate whether each program/strategy was already underway. Many states have updated their equity plans since initially submitting approved plans prior to 2009.
LEA equity plans. On the basis of preliminary research, AIR anticipates that some of the 75 largest LEAs will have local equity plans, though not with the detail found in state equity plans; and some LEAs will have submitted only the required assurance, without information on how the LEA will fulfill that assurance. In addition, in a few cases, LEAs may not have submitted the required assurances. In a sample of 12 OESE monitoring reports selected by AIR for preliminary review, 2 states did not require LEAs to provide the required assurances.
Applications to federal competitive grant programs. Some of the states and LEAs will have been awarded competitive grants from a program whose purpose is related to ensuring teacher quality for disadvantaged students (for a list of programs, see Introduction). Applications will contain descriptions of planned activities and in some cases include analyses of data on teacher quality. The scope of applications to be reviewed will include all RttT applications and the winning applications from all other programs.
Government Web sites. The extent of information available on government Web sites varies from state to state and district to district. AIR anticipates that Web sites will occasionally contain answers to some of the study’s evaluation questions. The research team will also review information made available on ED’s Improving Teacher Quality State Grants Web site.
Policy databases of the National Council on Teacher Quality. The National Council on Teacher Quality (NCTQ) maintains a state policy database that is updated annually and focuses on state policies related to teacher quality. Among the policy variables are some related to measures of teacher quality (EQ2) and some related to actions to improve teacher quality for disadvantaged students (EQ3), though the data are not sufficiently detailed to answer the protocol questions independently. NCTQ also maintains an LEA policy database focused on 100 school districts, including the 75 largest LEAs.
State and LEA Interviews
The procedures for conducting the state and LEA interviews are designed to gather complete, high-quality data efficiently, while minimizing the burden on respondents and maximizing response rates. These procedures will be completed by professional research staff, who have experience conducting interviews, and who will receive specialized training in the study context, goals, and procedures. The procedures and instruments will be reviewed through AIR’s IRB before use.
Research staff will commence these data collections immediately upon OMB approval, and they plan to complete interviews no later than March 31, 2012. The procedures will be completed for each state prior to interviews with any eligible school districts in the state, so that interviewers understand the state context when conducting the LEA interviews.
Initial contact. The research team will mail the notification letter and consent form to the state Title II director and follow up by phone (see Appendixes B and C). The purpose of the first phone call is to determine whether the Title II director should serve as the primary respondent and, if so, whether any supplemental respondents should be contacted. The team will obtain this information by reviewing the protocol topics with the Title II director and providing a copy of the interview protocol. They also will confirm with the Title II director that they have identified the most up-to-date equity plan available and confirm that they are examining the right portion of the state Web site for relevant information. The same procedures will be used for making initial contact with LEAs. The research team will ask the state contacts to indicate who the primary respondents in each LEA should be, if not the local Title II director.
Extant document review. The next step to be completed before interviewing respondents in each state and LEA will be to review the extant documents described above. We anticipate that these sources will provide partial answers to some questions on the protocols. Interviewers will use the information to annotate the protocols before each interview to reduce duplication and to identify areas for clarification and further elaboration.
Interviews. The interviews with the primary respondent and any additional respondents for each state or LEA will cover all of the evaluation questions and last approximately 105 minutes, divided among the respondents within each entity. The protocols are designed to communicate questions in a clear, conversational manner; generate systematic quantifiable data across all states; allow respondents to provide adequate contextual information on their state’s or district’s approaches to equitable distribution of high-quality teachers; and include appropriate questions to examine change over time. Drafts of the protocols are provided in Appendixes D and E. Each interview will be recorded digitally. Notes will be summarized after each interview. In the event that exact quotations or verifications are needed, the audio file will be available as a backup.
Data collection is a complicated process that requires careful planning. AIR has developed interview and data collection protocols that are streamlined and that are designed to place as little burden on respondents as possible. AIR also will pilot and subsequently refine all instruments to ensure they are user-friendly and easily understandable—all of which increases participants’ willingness to participate in the data collection activities and thus increases response rates. Consequently, the contractor anticipates 100 percent response rates from the states and an 85 percent or greater response rate from the districts. In a recent study involving interviews with 52 SEAs, AIR achieved a 100 percent response rate (see Birman et al., 2009).
State and LEA interview protocols will be pilot tested. AIR will work closely with ED and the study’s advisors to finalize the protocols to be piloted and identify states and districts where they can be tested. AIR will identify two states and two districts, each with varying characteristics, to conduct the initial pilot through phone interviews. Lessons learned during the piloting process will inform refinements to protocols, as well as procedures for scheduling and conducting interviews.
AIR is the contractor for the study. Dr. Beatrice Birman will serve as principal investigator (PI) and Dr. Andrew Wayne as project director (PD). There are no subcontractors. Contact information for these individuals and organizations is presented in Exhibit 8.
Exhibit 8. Individuals Involved in Project
Contact Name |
Organization |
Responsibility |
Telephone Number |
Dr. Beatrice Birman |
AIR |
Principal Investigator |
202-403-5318 |
Dr. Andrew Wayne |
AIR |
Project Director |
202-403-5483 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Andrew Abrams |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |