TIF Stakeholders in TIF Sites Telephone Interview Protocol

Interview: Stakeholders in TIF sites

Telephone interviews in all 34 project sites – Fall 2009

Teacher Incentive Fund Evaluation

Background for Interview Protocol: TIF Stakeholders

Purpose and Notes:

Telephone interviews will be conducted in the fall of 2009 with key informants at all 34 TIF sites. The purpose of grantee telephone interviews is to verify grantees’ incentive project descriptions and address questions about the implementation of the TIF projects from a variety of perspectives.

This semi-structured interview protocol contains all of the questions that might be asked to stakeholders in TIF sites (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials, and local education reporters). Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Prior to the interviews, individuals will be contacted by their TIF project office regarding the evaluation.

This protocol identifies key topic areas for stakeholders in TIF sites. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form sent electronically. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: Be prepared for varying levels of knowledge about the initiative and tailor your interview accordingly. If an informant is unable to answer a question, it may be appropriate to inquire about who would know the answer.

Telephone Interview Protocol: TIF Stakeholders

I. Background

1. Tell me about your background.

1. How and when did you come to be involved in this project?

2. What is your role in this project?

2. What are the most important initiatives in your State/district/school at this time? (Note: this will be used as background for questions about how TIF fits into a broader plan and/or recent PFP history.)

1. Are there other initiatives focused on improving teacher quality and effectiveness? Please describe.

II. Context/Participation in TIF

3. Does the State/district/school have a history of pay for performance plans or an existing pay for performance plan? (Note to OMB: this is to learn about respondent’s awareness, not to gather facts.)

1. Are there successes of those projects that you hope will be replicated here? Are there difficulties/failures of those projects you hope will be avoided?

2. How does the TIF project relate to the pre-existing project?

4. Did you participate in planning your TIF project? If so, please describe your State/district/school’s process for planning the initiative. If not, describe your understanding of the process.

1. Why did your State/district/school apply?

2. Was there any particular champion for the issue?

3. Was their outside pressure to participate?

4. Who participated in planning the project?

5. (For Teacher Association Representatives) Please describe the process by which the district and the Teacher Association/union worked on the TIF initiative.

6. How long was the planning period?

7. What were the major issues?

III. Project Design

5. What are the key goals of the performance pay project? How do you feel about those goals?

6. Who is eligible to receive an award in the pay for performance project (school/administrator/teacher)? Do these seem to be the right individuals (e.g., the principal must be included in every TIF award: does that seem fair to you? Why or why not?

7. What outcomes and/or activities are rewarded in the project? Please describe. Do those seem like the right outcomes/activities? What others might you suggest?

8. What is the award range for each school/administrator/teacher?

1. How is the award amount for each activity determined? Does this seem fair to you?

2. What percent of the award is designated toward each activity? Does this seem about right?

3. Do you consider these awards to be substantial, relative to the existing pay of teachers/administrators and the amount of additional work/responsibility they take on? On what basis do you make this claim?

9. What are the criteria for receiving each award? Do you feel that they fairly take into account factors over which teachers have direct control, and objective and factors over which they have less control and which may be more subjective?

10. What data are currently available in the district/State about teacher and/or principal performance? How long has this data system been in place?

1. Are data available on teacher educational background, years of experience, date of hire in school or district? Do you believe that these are generally good indicators of quality for teacher recruitment, or would you prefer to have additional measures on which to base a hiring decision?

2. Are students’ achievement data linked to teachers: how is this process done (again, this level of detail is likely going to require the assistance of the data person)?

3. How are these data used to make performance pay decisions? How is that process going from your perspective? Have people you’ve encountered been generally satisfied with how it’s going?

III. Project Implementation and Communication

11. How were project components communicated to teachers, principals, and the community?

1. How do schools stay informed about [name of project]?

12. How has the project implementation gone so far?

1. Were there differences in rollout across schools and districts?

2. Which stakeholders were especially difficult to bring on board (if any)?

3. Which stakeholders were involved in rolling out the pay for performance project?

4. What worked or didn’t work?

5. Were there any components that your [grantee] did not have the capacity to implement as designed? (Note: this could include having budget to offer the PD that was required, provide any support designed into their project)

13. What changes to the project are expected? Why?

14. What is the State/district/school plan for continuing the performance pay project when TIF funding expires? (GPRA indicator): What percent of the teacher personnel budget is currently devoted to pay-for-performance? Is the amount expected to get larger or smaller in coming years? Why?

IV. Evaluation

15. How do you know if the project has achieved its goals? (How do you measure progress towards them?)

16. Does the project have an internal evaluation? External evaluation?

1. What do you expect to learn from an evaluation? Do you have a continuous improvement/formative evaluation process in place that allows you to modify the specifics of the project on the basis of whether more effective-teachers are being recruited/retained, their student’s achievement is going up, etc? How does this process work?

V. Perceptions of Outcomes: Recruitment and Retention

17. What are the characteristics of typical applicants to your school/district?

1. Level of education

2. Certifications

3. Years of experience

4. Do you believe these are good measures of whether a teacher would operate effectively in your high-need, hard-to-staff schools? Why or why not?

18. What is the average teacher turnover rate for your school/district in a given year? Why do people leave? Where do they go? What are their characteristics? Do you have a sense of whether they are more qualified? (Note: just knowing the teacher turnover rate would seem to be less informative than knowing at least something about the quality of the teachers that are leaving.)

2. How many vacant positions exist in your school/district at any given time?

3. What strategies do you use to recruit highly qualified teachers? Have those been successful? Why or why not?

4. Are there any particular subjects for which qualified candidates are difficult to find? What strategies do you use to recruit qualified candidates for these subjects?

VI. Perceptions of Outcomes: Principal Leadership

19. How easy is it for your district to attract and retain desired school leaders?

1. How do you define a “desired” school leader: do you have any standard objective measures?

2. What is the average qualified principal turnover rate for your school/district in a given year?

3. Are there any particular schools where attracting and retaining high quality principals is more challenging? What have you tried in the past to attract and retain them? How successful has that been?

4. What are the characteristics of typical applicants for principal positions? Do they come more from within your district or outside? Do they have previous experience as principals? In other leadership positions? Other than TIF, do you have any special principal leadership development academies that might explain any increase in the number of desired school leaders who are being recruited to and retained in your schools? Please describe, and how would it be possible, if at all, to tell whether the TIF project was more responsible for this shift in the principal stock than other projects you are attempting.

20. If not clear from preceding discussion of the awards principals are eligible for: what aspects of principal recruitment, retention or leadership did you hope would improve as a result of this project? How were the awards designed to target these particular issues?

VII. Perceptions of Outcomes: Teacher Practice

21. If not clear from preceding discussion of the awards teachers are eligible for: what aspects of teacher practice did you hope would improve as a result of TIF? Why? How were the awards designed to target these particular issues? Do you believe that, to date, teachers are teaching better (has teacher practice has improved in a way that meets your hopes)? (Probe: are they using research-based instructional techniques, improved curriculum, or improved strategies for working with diverse learners?) Why or why not? On what basis do you make that claim: for example, observation of instruction, numbers of teachers receiving awards, improvements (anecdotally) in evaluations of teachers, reports from parents/students, combination?

VIII. Perceptions of Outcomes: Student Learning

22. If not clear from preceding discussion of the awards teachers are eligible for: what aspects of student learning did you hope would improve as a result of this project? How were the awards designed to target these particular issues? D you perceive that, to date, TIF has had that desired effect? What other projects are going on simultaneously in your district that might explain any increases in student learning that might be confounded (confused) with an eventual outcome of TIF (e.g., student pay-for-grades, other teaching projects like Teach for America, major changes to curriculum and instruction, etc.?)

VIIII. Closing

23. From your perspective as [title], what do you think would improve the effectiveness of TIF [name of project]?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 9 of 9