Responses to OMB questions of 11/4

M
EMORANDUM

600 Maryland Ave., SW, Suite 550

Washington, DC 20024-2512

Telephone (202) 484-9220

Fax (202) 863-1763

www.mathematica-mpr.com

TO: Elizabeth Warner and Yumiko Sekino

FROM: Steven Glazerman and Nancy Carey DATE: 11/12/2009

SUBJECT: Response to 11/4 OMB queries on Moving Teachers data collection

This memo addresses two questions that were raised in a November 4th phone call with OMB regarding our proposed data collection to support the evaluation of the Talent Transfer Initiative to move high performing teachers to low performing schools. The questions here are not verbatim, but they hopefully capture the issues raised by the OMB staff.

Question 1

Will the study be able to quantify the level of effort that was needed to recruit high performing teachers to low performing schools?

Answer 1

Yes. The study will be able to measure and describe the average level of effort. Many of the costs are fixed at the site level, although there are some variable costs that depend on the size of the applicant pool, the number of receiving schools, the number of teaching slots being filled, or the timing of those vacancies. (It is easier to fill many vacancies at once than it is to fill the same number of vacancies that are identified over a period of several months). One site manager from the project team was responsible for each participating district and the level of oversight by a senior manager responsible for all the site managers was fairly fixed as a percentage of that person’s time. Site managers may have spent different percentages of their time in depending on the size of a district, but we can recover the information on these differences using program management data. We can also recover information on the frequency and duration of visits to the districts, the cost of one-time events such as outreach/information sessions, and other direct costs. Much of the outreach and communication was conducted by telephone and email, the costs of which are primarily reflected in the labor hours charged by site managers.

Question 2

What is the analysis plan for quantifying the size of the bonus as a fraction of base pay and for using this information to determine whether the variation in bonus as a fraction of base pay influences teachers’ decisions to apply to the program and transfer?

Answer 2

We agree that measuring base salary is important because we can use it to compute the size of the bonus for each candidate as a fraction of base pay. For example, a fourth year teacher in Houston with a bachelor’s degree would earn $44,290 for a 10-month contract (using the salary schedule published on the Web), whereas the salary for the same duration contract for a teacher with 20 years of experience and a doctorate would be $59,943. The bonus of $10,000 per year amounts to 23 percent of the lower-paid teacher’s base pay and 17 percent of the higher-paid teacher’s base pay in this example.

Once we have information on this variable, we plan to use it, along with other factors that possibly contribute to teachers’ decisions, to predict their decision to attend an information session, apply to the TTI, interview, and transfer. The other factors include satisfaction with aspects of their current teaching position, their personal characteristics (race/ethnicity, gender, marital status, presence of children at home, home ownership status, residential location), prior teaching history in the district, distance from receiving schools, and attitudes toward receiving schools. For each discrete outcome we will estimate a logistic regression (or multinomial logistic regression as appropriate) to measure the strength of association between each explanatory factor and the mobility outcomes.

In our current draft of the Candidate Survey, we have some questions about additional compensation beyond base pay, but we had tried to avoid potentially sensitive questions like base salary. Our plan was to use the years of experience teaching in the district and degrees obtained (which are on the survey) to estimate base salary using each district’s uniform salary schedule. The uniform salary schedule, which is public information and often available on district websites, makes this calculation straightforward and allows us to avoid asking the question directly.

However, we recognize that estimating teacher salary can be inaccurate. For example, teachers sometimes receive extra years of service credits for teaching full or partial years in other districts or states. Also, some districts include separate salary lanes for teachers who have not completed an advanced degree (such as a master’s) but have course work that could count toward that degree. Finally, there are districts in which contractual pay raises are delayed or not awarded because of a budget crisis.

At the prompting of OMB, we contacted Stephen Cornman of NCES to assess the possibility of using another data source, state data on teacher salaries from administrative records. NCES is currently working with states to obtain salary information from administrative records, but we are concerned that the effort to obtain such information will not be trivial and such effort is not warranted in the context of this evaluation. Specifically, NCES currently collects data in only 2 of the 5 states represented in our study, although Stephen thought he might be able to help in at least two of the other states. For our analyses we need to be able to link the salary information to each teacher in our sample. Since NCES would not be able to do this directly, we would need to go back to the state to provide us with the salary information for just those teachers in our sample using the social security number, which we do not have for most respondents (TTI candidates).

Stephen did not have information about the relationship between administrative records, self-report salary information and inferring salary information from the salary schedule. However, he did convince us that salary schedule information would probably lead to errors in the salary information, and likely more error than self-reported salary information.

We carefully considered the costs and benefits of the three options discussed above: (a) ask about base salary on the survey; (b) use survey-reported data on teachers’ years of service and degree along with published salary schedules to compute teacher salary; and (c) use state administrative data on specific teachers. We concluded that option (a) is the most efficient method. Salary computed from a published salary schedule (option b) could produce inaccurate results and data from state administrative records (option c) would be costly to collect and ultimately incomplete if we do not have proper teacher identifiers. The survey approach involves almost no extra cost or effort and will be as accurate as we need for the purposes of the analysis. We have revised the questionnaire by adding one item on base salary. The proposed new section is attached. NOTE: we used definitions recommended by Stephen for the salary questions.

An Affirmative Action/Equal Opportunity Employer