TIF Educator in TIF Sites Telephone Interview Protocol

Interview: Educators in TIF sites

Telephone interviews in all 34 project sites – Fall 2009

Teacher Incentive Fund Evaluation

Background for Interview Protocol: Educators in TIF Sites

Purpose and Notes:

Telephone interviews will be conducted in the fall of 2009 with key informants at all 34 TIF sites. The purpose of grantee telephone interviews is to verify grantees’ incentive project descriptions and address questions about the implementation of the TIF projects from a variety of perspectives.

This semi-structured interview protocol contains all of the questions that might be asked to school employees—particularly principals and teachers. The protocols have been designed in such a way as to be broad enough to encompass the breadth and variation we expect to encounter both in who we talk to (and how much they know) as well as the variability in the projects themselves. Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions which are appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Prior to the interviews, individuals will be contacted by their TIF project office regarding the evaluation.

This protocol identifies key topic areas for educators. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form sent electronically. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: There will be variation in the groups of principals and teachers interviewed. Some will have been deeply involved in the development and implementation of the initiative and others not. Generally, we need to determine the interviewees’ level of involvement and expected knowledge of the initiative.

Telephone Interview Protocol: Educators in TIF Sites

I. Background

1. Tell me about your background.

1. What is your current job? If an administrator, are you a full principal? If a teacher, what grade(s)/subject(s) do you teach?

2. How and when did you begin this assignment?

3. What did you do prior to your current position?

4. Do you participate in the TIF project (if participation is optional)?

II. Context/Participation in TIF

1. Does your State/district/school have a history of pay for performance plans or an existing pay for performance plan (aside from TIF)?

1. How successful was the previous/other project(s)?

2. How does the TIF project relate to the pre-existing project? (Probe for whether the respondent can correctly identify what is and is not TIF if there is more than TIF currently being implemented).

2. Did you participate in planning your TIF project? If so, please describe your State/district/school’s process for planning the initiative. If not, please describe the process based on what you have heard.

1. Why did your State/district/school apply?

2. Who participated in planning the project?

3. How were individuals (you) selected to participate in the planning process? Do you consider yourself fairly typical of the attitudes and judgments of your colleagues, or would you describe yourself as more “reform-minded” or “willing to experiment” with education innovations, including compensation?

4. How long was the planning period?

5. What were the major issues?

III. Project Design and Implementation (Note:—the purpose of several of these questions is to check for an accurate understanding of the project and then perceived fairness. The “correct” answers about project design are gathered from other respondents.)

3. How did you first hear about [grantee’s name] TIF project? Is this how most teachers/principals heard about it? (If not, describe how others heard about it.)

4. What are the key goals of the TIF [name of project]?

5. What outcomes and/or activities are principals/teachers eligible to be rewarded for in the project?

1. Is the TIF project attaching additional pay to the “right” things? Why or why not?

6. How are teachers/principals evaluated for an award? Describe the process. (Note: principals should address these questions for the principal and teacher evaluation processes.)

1. What was the traditional evaluation system like? Was it perceived as fair by your colleagues? What did they like about it? What did they not like?

2. What do(es) the new evaluations measure?

3. Who conducts the evaluation(s)?

4. How are they trained (Note: this would apply mostly to principal evaluations of teachers, not value-added systems or evaluations of principals, which are presumably outside the knowledge of the interviewee)?

5. How and to what extent is performance feedback built-in to the system?

6. Are the evaluations seen as fair by teachers/principals? Which part(s)? Why or why not?

7. Did the evaluation system change significantly because of the TIF grant? If so, how?

7. How are award decisions made? What data are used?

8. What are your perceptions of the quality of the State/district data systems?

1. Are data perceived as complete and accurate?

2. Are measures perceived as useful and appropriate? (Probe for whether the evaluation system described above is used for determining awards.)

9. (If respondent participates in TIF) Have you received awards? What have they been for? Did you feel they were substantial given your base pay and the amount of work you had to do to get them? Why or why not?

10. Have you spoken with other teachers/principals about how they feel about [name of project] and/or incentive pay in general?

1. Do the teachers/principals that are participating agree with and support the goals and strategies of [name of project]?

2. Do most teachers/principals participate in TIF? (If participation is not universal) Why do some people choose not to participate in TIF?

3. (If participation is not universal) How do non-participating teachers feel about TIF?

11. How has project implementation gone so far?

1. Has anything gone particularly well?

2. If you have participated, is there anything that has been particularly confusing? Rewarding?

3. Has anything been particularly challenging? If any major errors were made, why do you think they occurred?

4. Overall would you say that the TIF project is fair? (Probe what it rewards and how it was implemented from planning, through rollout, to pay outs.)

12. Have there been any changes to your TIF project so far? What do you think led to those changes? (Probe for whether practitioner concerns were a cause of the changes). Do you think those changes have improved the project? Why or why not?

IV. Perceived Outcomes

13. (If the respondent participates in TIF) Do you think it is realistic that you could do what is necessary to receive the awards you want to earn? What supports are present or lacking to help you achieve the awards?

14. What, if anything, have you done differently because of the project?

1. Have you stayed in a particular position (e.g., principal at a high-need school, teacher in a hard-to-staff subject) because of the possibility of earning an award?

2. Have you participated in additional professional development activities specifically tied to TIF? Please describe. Have they been useful?

3. Have you tried new instructional/leadership practices or new teaching strategies as a result of TIF? Please describe.

15. What effects has the project had on school climate?

1. (If whole school) In those schools where the reward is first determined at the school level (before being divided among all or most teachers), does it seem that teachers are all pulling their weight to earn the award?

2. Based on your experience looking at a number of schools (not just your own), do teachers collaborate more or less because of the TIF project? Why?

3. Do teachers feel additional pressure to perform because of the award? If so, describe how the pressure plays out (Note:—try to understand the nuances in between “positive motivator” and “destructive pressure cooker”).

4. Have you heard of any incidents of cheating (e.g., cheating on tests to improve a value-added score) for people trying to get an award? If so, describe. Have you heard of any other ways that people are “gaming the system” in any way? (Note: this is designed to capture more subtle cases of working the system) Do you think this is a serious threat to the long-term practicality of TIF, or at least of its value-added component? Why or why not?

16. Is there yet a perception that student learning has improved as a result of the incentives? Is there any independent evidence you’re aware of besides increasing in-school test scores (e.g., SAT/ACT) to bear this out?

17. How do you think parents and the community view the TIF project? Do they know about it? How are the results of the system reported widely? Do you think the reporting is adequate?

VIIII. Closing

18. From your perspective as [title of respondent], what do you think would improve the effectiveness of the performance pay project?

19. Is there anything else you think we need to know about the TIF project in [grantee name]?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 8 of 8