TIF Stakeholders in TIF Sites Visit 1 Interview Protocol

Site Visit 1: Stakeholders in TIF sites

Face-to-face visits in 12 TIF sites – Spring 2010

Teacher Incentive Fund Evaluation

Background for Site Visit 1: TIF Stakeholders

Purpose and Notes:

The first set of site visits will be conducted in year two of the evaluation (spring 2010). Twelve sites will be selected and each visit will include three days of interviews by a two-person team of researchers to determine progress in implementation. (See below for more information on the selection of sites).

The purposes of the site visits are to (1) verify data gathered through the telephone interviews and project documents, and (2) complement the interviews with information about stakeholder satisfaction and potential for sustainability that can best be gathered through direct observation and in-person discussion. Researchers will observe the format and content of the funded activities, the types of activities teachers are participating in, the work teachers are expected to perform, and the objectives and goals teachers are expected to reach. In particular, site visits will be crucial in the event that subsequent outcome analyses reveal a significant influence of TIF on outcomes such as retention of effective teachers in high-need schools. By discussing stakeholder satisfaction and observing school climate in the TIF environment, researchers will gain tools for potentially explaining the correlation of TIF projects and improvements with teacher outcomes.

This semi-structured interview protocol contains all of the questions that might be asked to stakeholders in TIF sites (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials, and local education reporters). Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Prior to the interviews, individuals will be contacted by their TIF project office regarding the evaluation.

This protocol identifies key topic areas for stakeholders in TIF sites. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Information collected in the telephone interviews conducted in the fall of 2009

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: Be prepared for varying levels of knowledge about the initiative and tailor your interview accordingly. If an informant is unable to answer a question, it may be appropriate to inquire about who would know the answer.

Sample Information

Based on the information gathered in the telephone interviews and a thorough review of extant grantee documents, a stratified random sample of active grantees will be selected for site visit 1. The sites will represent the diversity of TIF grantees’ approaches to developing and implementing performance-based compensation systems roughly divided into three strata: (1) projects in which 50 percent or more of the TIF grant activities award is allocated to merit pay (based on improved student achievement), (2) sites primarily implementing broader forms of differentiated compensation (e.g., increased pay for teaching in specific schools or subjects), and (3) sites implementing the most comprehensive systems (combining merit pay with various compensated teacher professional development activities, e.g., new teacher career ladder projects; or Teacher Advancement Program/TAP).

Performance Pay	Differentiated	Comprehensive
4 grantees	4 grantees	4 grantees

Site Visit 1 Interview Protocol: TIF Stakeholders

1. Background (Begin with an overview of the TIF project and the purpose of the interview).

1. Tell me about your background (If you have already interviewed this respondent, please confirm previous data).

1. How and when did you come to participate in or follow the development of this project?

2. What is your role in the project?

2. What are your responsibilities in this role? Has this changed?

If needed, probe for:

1. Liaison with other partners, such as ED, technical assistance provider, evaluators, etc?

2. Serve as the district representative? In what ways?

3. Participate in internal evaluations?

4. Reporting?

5. Communicate with the public/parents/teachers?

6. Support implementation?

3. What are the most important initiatives in your State/district/school at this time? Have these changed? (Note: this will be used as background for questions about how TIF fits into a broader plan and/or recent PFP history.)

1. Have they changed since beginning participation in TIF?

2. Please describe any other initiatives focused on improving teacher quality. Do you see any difference between these and TIF as you understand it?

II. Context/Participation in TIF

4. Does your State/district/school have a history of pay for performance plans or an existing pay for performance plan? Has anything new been added recently?

1. Are there successes of those projects that have been replicated here? Are there difficulties/failures of those projects that have been avoided?

2. How does the TIF project relate to the pre-existing project?

5. What are the major policy initiatives in the State/district and how do they impact teacher compensation reform? Do you see this as a positive or negative trend, or some of both?

6. What has been the process for revising participation plans?

1. Who participates?

2. What have been the major issues?

3. (For Teacher Association Representatives) Please describe the process by which the district and the Teacher Association/union worked on the TIF initiative.

III. Project Design

7. What are the key goals of the performance pay project? How have these changed?

1. Have they changed since the project began?

2. How do the goals relate to the project?

8. Who is eligible to receive an award in the pay for performance project (school/administrator/teacher)?

1. Has this changed since the beginning of the project?

9. What is the target goal for number/percentage of participating schools/administrators/teachers (if known)?

1. Has the goal changed since inception?

2. Do you feel these goals have been met?

3. What is the plan for increasing/sustaining participation?

4. What challenges have been faced regarding participation?

10. Please provide an estimate of the number/proportion of participating (schools/administrators/teachers).

11. From our previous conversation, we learned that participation was mandatory/voluntary.

1. Have there been any changes to this policy?

2. If so, what brought about the change?

12. How many students were affected by the project (based on scope of project and participation rates)? What proportion of all students in the school/district do these students represent?

13. In our previous conversation, you described X, Y, and Z outcomes and/or activities that are rewarded in the project. Have there been any changes to these activities? If so, please describe.

14. In our previous conversation, you said that professional development was/wasn’t included in the project. Has this changed? If so, what have the changes been?

15. (For grantees that include professional development in their project:)

1. Are all teachers/principals required to attend professional development to be eligible for an award under [name of the project]?

2. Is professional development part of the award criteria? How much weight does it have in making awards?

3. Do you know what percentage of your teachers have participated in professional development since the start of [name of project]?

4. Have there been any issues in terms of the districts capacity for offering the professional development (e.g. capacity/funding)?

16. What is the award range for each school/administrator/teacher? And how has this changed and why?

1. How is the award amount for each activity determined?

2. What percent of the award is designated toward each activity?

3. What is the average amount (or typical) amount awarded to schools/administrators/teachers?

4. Do you consider these awards to be substantial, relative to the existing pay of teachers/administrators and the amount of additional work/responsibility they take on?

17. How many teachers have received awards each year (if known)?

1. If there are multiple categories of teacher awards, please provide the number of eligible teachers and award recipients for each teacher award category, if known.

18. What are the criteria for receiving each award?

19. What supports are in placed to help participants earn a reward?

20. What data are currently available in the State/district about teacher and/or principal performance? How long has this data system been in place?

1. Are data available on teacher educational background, years of experience, date of hire in school or district?

2. Are students’ data linked to teachers?

3. How have these data been used to make performance pay decisions?

III. Project Implementation and Communication

21. In our last conversation, you described the way in which teachers, principals, and community are informed about the project.

1. Has the communication strategy changed since the beginning of the project?

2. How have changes in the project been shared with teachers, principals, and the community?

3. How do schools stay informed about [name of project]?

22. How well was the initiative implemented? How did the rollout go?

1. What were the barriers to successful implementation? What worked well?

2. Were there differences in rollout across schools (and districts)?

3. What accounts for those differences? (Probe: buy-in, history of student achievement, turnover in leadership/teachers, capacity to do it)

4. Which stakeholders were especially difficult to bring on board (if any)?

5. Which stakeholders were involved in rolling out the pay for performance project?

6. Were there any components that your [grantee] did not have the capacity to implement as designed? (Note: this could include having budget to offer the PD that was required, provide any support designed into their project)

23. Is there a group that is designated for oversight and/or continued project adjustments?

1. If yes, who participates in this group?

2. What authority or responsibility does the group have?

24. What additional costs were incurred as a result of the transition to pay for performance?

1. How were those costs absorbed/covered?

2. Was outside support provided?

25. What is the State/district/school plan for continuing the performance pay project when TIF funding expires?

1. Increased assumption of costs?

2. Offsetting with reduced teacher turnover costs?

3. Does the district/school plan to receive outside support from partners or other funders for continuation of the project?

26. Has the funding plan changed since we last spoke? If so, how and why?

27. What proportion of the State/district/school budget was allocated to pay for performance and TIF in particular, during the first year of TIF?

1. Has this percentage increased or decreased?

2. What is the current proportion of the budget allocated to TIF projects?

IV. Evaluation

28. How do you know if the project has achieved its goals? (How do you measure progress towards them?)

29. Does the project have an internal evaluation? External evaluation?

1. How was the external evaluator selected? Who is the external evaluator?

2. Who is conducting the internal evaluation?

30. What have you learned, thus far, from evaluation?

1. How, if at all, have you seen results from internal evaluation(s) affect change in the performance pay project?

2. How, if at all, have you seen results from external evaluation(s) affect change in the performance pay project?

31. Has the evaluation design changed since we spoke on the phone? Please describe the changes in the evaluation design and the analysis plan?

31. What data is available and what is the quality of the data? Has this changed?

31. What are you finding about student learning and other student outcomes? Teacher learning, skills, and other teacher outcomes (e.g., recruitment, retention, attendance?)

31. What have been the main challenges associated with the evaluation? (e.g., State and local data systems? Access?)

31. How are the evaluation results being used? By whom?

V. Perceived Outcomes

31. What were the goals associated with teacher recruitment and retention at the beginning of the project? Do you feel progress has been made toward these goals? Why?

31. What were the goals associated with principal recruitment, retention or leadership at the beginning of the project? Do you feel progress has been made toward these goals?

31. What evidence is there that teaching has improved as a result of the incentives?

31. What were the goals associated with student learning at the beginning of the project? Do you feel progress has been made toward these goals?

VIIII. Closing

31. What changes to the project have occurred or are expected in the future?

1. What issues arose that caused these changes to be recommended?

2. How and when will changes be implemented

31. What changes to the project have occurred (if not already discussed)?

1. What issues arose that caused these changes to be recommended?

2. How and when were changes implemented?

3. How were stakeholders involved in the changes?

4. Do you foresee additional changes?

31. From your perspective as [title], what do you think would improve the effectiveness of TIF [name of project]?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 7 of 7