TIF Stakeholders in TIF Sites Visit 2 Interview Protocol

Site Visit 2: Stakeholders in TIF sites

Face-to-face visits in 12 TIF sites – Spring 2012

Teacher Incentive Fund Evaluation

Background for Site Visit 2: TIF Stakeholders

Purpose and Notes:

The second set of site visits will be conducted in year two of the evaluation (spring 2010). Twelve sites will be selected and each visit will include three days of interviews by a two-person team of researchers to determine progress in implementation. (See below for more information on the selection of sites).

The purposes of the site visits are to (1) verify data gathered through the telephone interviews and project documents, and (2) complement the interviews with information about stakeholder satisfaction and potential for sustainability that can best be gathered through direct observation and in-person discussion. Researchers will observe the format and content of the funded activities, the types of activities teachers are participating in, the work teachers are expected to perform, and the objectives and goals teachers are expected to reach. In particular, site visits will be crucial in the event that subsequent outcome analyses reveal a significant influence of TIF on outcomes such as retention of effective teachers in high-need schools. By discussing stakeholder satisfaction and observing school climate in the TIF environment, the contractor will gain tools for potentially explaining the correlation of TIF projects and improvements with teacher outcomes.

This semi-structured interview protocol contains all of the questions that might be asked to stakeholders in TIF sites (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials, and local education reporters). Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Prior to the interviews, individuals will be contacted by their TIF project office regarding the evaluation.

This protocol identifies key topic areas for stakeholders in TIF sites. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Information collected in the telephone interviews conducted in the fall of 2009

• If applicable, information collected in site visit 1 in spring 2010

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: Be prepared for varying levels of knowledge about the initiative and tailor your interview accordingly. If an informant is unable to answer a question, it may be appropriate to inquire about who would know the answer.

Sample Information

An optional sampling approach for the second round of case study data collection is recommended. This option will require conducting the outcome analysis prior to the second round of case studies in order to sample based on grantee outcomes. This approach will provide in-depth data on the grantee plans, practices, and contextual factors that lead to a range of student and teacher outcomes. Sites will be selected using a stratified random sample of three high-performing projects and one low-performing project (with performance defined based on outcome measures in the impact study) from each of the three types of grantee projects:(1) projects in which 50 percent or more of the TIF grant activities award is allocated to merit pay (based on improved student achievement), (2) sites primarily implementing broader forms of differentiated compensation (e.g., increased pay for teaching in specific schools or subjects), and (3) sites implementing the most comprehensive systems (combining merit pay with various compensated teacher professional development activities, e.g., new teacher career ladder projects; or Teacher Advancement Program/TAP).

	Performance Pay	Differentiated	Comprehensive
High Performing	3 grantees	3 grantees	3 grantees
Low Performing	1 grantee	1 grantee	1 grantee

Site Visit 2 Interview Protocol: TIF Stakeholders

I. Background

1. Tell me about your background (If you have already interviewed this respondent, please confirm previous data).

1. How and when did you come to participate in or follow the development of this project?

2. What is your role in the project?

2. Has your role changed since the beginning of the project?

3. What are the most important initiatives in your State/district/school at this time? (Note: this will be used as background for questions about how TIF fits into a broader plan and/or recent PFP history.)

1. Have they changed since beginning participation in TIF?

2. Please describe any that are focused on improving teacher quality.

II. Context/Participation in TIF

4. Does your State/district/school have a history of pay for performance plans or an existing pay for performance plan?

1. Are there successes of those projects that have been replicated here? Are there difficulties/failures of those projects that have been avoided?

2. How does the TIF project relate to the pre-existing project?

5. What has been the process for planning or revising participation plans?

1. Who participates?

2. What have been the major issues?

3. (For Teacher Association Representatives) Please describe the process by which the district and the Teacher Association/union worked on the TIF initiative.

III. Project Design

6. What are the key goals of the pay for performance project?

1. Have they changed since the project began?

2. How do the goals relate to the project?

7. Who is eligible to receive an award in the pay for performance project (school/administrator/teacher)?

1. Has this changed since the beginning of the project?

8. What is the target goal for number/percentage of participating schools/administrators/teachers (if known)?

1. Has the goal changed since inception?

2. Do you feel these goals have been met?

3. What is the plan for increasing/sustaining participation?

4. What challenges have been faced regarding participation?

9. Please provide an estimate of the number/proportion of participating (schools/administrators/teachers).

10. From our previous conversation, we learned that participation was mandatory/voluntary.

1. Have there been any changes to this policy?

2. If so, what brought about the change?

11. How many students were affected by the project (based on scope of project and participation rates)? What proportion of all students in the school/district do these students represent?

12. In our previous conversation, you described X, Y, and Z outcomes and/or activities that are rewarded in the project. Have there been any changes to these activities? If so, please describe.

13. In our previous conversation, you said that professional development was/wasn’t included in the project. Has this changed? If so, what have the changes been?

14. (For grantees that include professional development in their project:)

1. Are all teachers/principals required to attend professional development to be eligible for an award under [name of the project]?

2. Is professional development part of the award criteria? How much weight does it have in making awards?

3. Do you know what percentage of your teachers have participated in professional development since the start of [name of project]?

15. What is the award range for each school/administrator/teacher?

1. How is the award amount for each activity determined?

2. What percent of the award is designated toward each activity?

3. What is the average amount (or typical) amount awarded to schools/administrators/teachers?

4. Do you consider these awards to be substantial, relative to the existing pay of teachers/administrators and the amount of additional work/responsibility they take on?

16. How many teachers have received awards each year (if known)?

1. If there are multiple categories of teacher awards, please provide the number of eligible teachers and award recipients for each teacher award category, if known.

17. What are the criteria for receiving each award?

18. What supports are in placed to help participants earn a reward?

19. What data are currently available in the State/district about teacher and/or principal performance? How long has this data system been in place?

1. Are data available on teacher educational background, years of experience, date of hire in school or district?

2. Are students’ data linked to teachers?

3. How have these data been used to make performance pay decisions?

III. Project Implementation and Communication

20. In our last conversation, you described the way in which teachers, principals, and community are informed about the project. Has the communication strategy changed since the beginning of the project?

21. How well was the initiative implemented? How did the rollout go?

22. What proportion of the State/district/school budget was allocated to pay for performance and TIF in particular, during the first year of TIF?

1. Has this percentage increased or decreased?

2. What is the current proportion of the budget allocated to TIF projects?

IV. Evaluation

23. How do you know if the project has achieved its goals? (How do you measure progress towards them?)

24. What have you learned, thus far, from evaluation?

1. How, if at all, have you seen results from internal evaluation(s) affect change in the performance pay project?

2. How, if at all, have you seen results from external evaluation(s) affect change in the performance pay project?

V. Perceived Outcomes: Recruitment and Retention

25. What strategies do you use to recruit teachers?

1. Are there any particular subjects for which qualified candidates are difficult to find?

2. Have recruitment strategies changed since implementation of the project?

26. Have you noticed a change in the qualifications of applicants since the initiation of your pay for performance project?

27. Have you seen a change in teacher turnover rates since initiation of your pay for performance project?

1. Describe the change – discuss magnitude

2. Would you attribute this change to the project? Completely? Partially?

VI. Perceived Outcomes: Principal Leadership

28. How easy is it for your district to attract and retain desired school leaders?

1. What is the average principal turnover rate for your school/district in a given year?

2. Are there any particular schools where attracting and retaining high quality principals is more challenging?

3. What are the characteristics of typical applicants for principal positions? Do they come more from within your district or outside? Do they have previous experience as principals? In other leadership positions?

29. What evidence is there that principal recruitment, retention or leadership has changed as a result of the incentives?

30. Have there been any challenges or successes in implementation that have affected the outcomes attained?

VII. Perceived Outcomes: Teacher Practice

31. What outcomes did you hope to see in teacher practice at the outset of this project? Do you feel there has been progress toward those goals?

32. What evidence is there that teaching has improved as a result of the incentives?

33. Have there been any challenges or successes in implementation that have affected the outcomes attained?

VIII. Perceived Outcomes: Student Learning

34. What were the goals associated with student learning at the beginning of the project? Do you feel progress has been made toward these goals?

1. What evidence is there that student learning has improved as a result of the incentives?

35. Have there been any challenges or successes in implementation that have affected the outcomes attained?

VIIII. Closing

36. What changes to the project have occurred or are expected in the future?

1. What issues arose that caused these changes to be recommended?

2. How and when will changes be implemented

37. What changes to the project have occurred (if not already discussed)?

1. What issues arose that caused these changes to be recommended?

2. How and when were changes implemented?

3. How were stakeholders involved in the changes?

4. Do you foresee additional changes?

38. From your perspective as [title], what do you think would improve the effectiveness of TIF [name of project]?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 7 of 7