TIF program Staff Site Visit 1 Interview Protocol

Site Visit 1: TIF Project Staff

Face-to-face visits in 12 TIF sites – Spring 2010

Teacher Incentive Fund Evaluation

Background for Site Visit 1: TIF Project Staff

The first set of site visits will be conducted in year two of the evaluation (spring 2010). Twelve sites will be selected and each visit will include three days of interviews by a two-person team of researchers to determine progress in implementation. (See below for more information on the selection of sites).

The purposes of the site visits are to (1) verify data gathered through the telephone interviews and project documents, and (2) complement the interviews with information about stakeholder satisfaction and potential for sustainability that can best be gathered through direct observation and in-person discussion. Researchers will observe the format and content of the funded activities, the types of activities teachers are participating in, the work teachers are expected to perform, and the objectives and goals teachers are expected to reach. In particular, site visits will be crucial in the event that subsequent outcome analyses reveal a significant influence of TIF on outcomes such as retention of effective teachers in high-need schools. By discussing stakeholder satisfaction and observing school climate in the TIF environment, researchers will gain tools for potentially explaining the correlation of TIF projects and improvements with teacher outcomes.

This semi-structured interview protocol contains all of the questions that might be asked to TIF project staff. Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions which are appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Both the number and the nature of key informants/respondents will vary somewhat based on the specifics of each grantee. For example, in a TIF site in which the grantee is a State entity and the project includes multiple districts and many schools, there will likely be roles filled by a number of individuals in the project office. In other sites, such as in the case of a small network of charter schools, a single person may represent the project office. The breadth of involvement will also vary from site-to-site making the flexible protocols essential to the smooth running of the interviews. For example, the technical assistance providers may be integrally involved in the project for some grantees while playing a smaller role in others just as the scope and stage of evaluations may vary.

This protocol identifies key topic areas for project directors, other project staff, TA providers, and evaluators. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Information collected in the telephone interviews conducted in the fall of 2009

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: This protocol covers a fairly broad set of interviewees. Be prepared for varying levels of knowledge about the initiative and tailor your interview accordingly. If a respondent is unable to answer a question, it may be appropriate to inquire about who would know the answer.

Sample Information

Based on the information gathered in the telephone interviews and a thorough review of extant grantee documents, a stratified random sample of active grantees will be selected for site visit 1. The sites will represent the diversity of TIF grantees’ approaches to developing and implementing performance-based compensation systems roughly divided into three strata: (1) projects in which 50 percent or more of the TIF grant activities award is allocated to merit pay (based on improved student achievement), (2) sites primarily implementing broader forms of differentiated compensation (e.g., increased pay for teaching in specific schools or subjects), and (3) sites implementing the most comprehensive systems (combining merit pay with various compensated teacher professional development activities, e.g., new teacher career ladder projects; or Teacher Advancement Program/TAP).

Performance Pay	Differentiated	Comprehensive
4 grantees	4 grantees	4 grantees

Site Visit 1 Interview Protocol: TIF Project Staff

I. Background

1. Tell me about your background. (Confirm that nothing has changed since the telephone interview).

1. How and when did you come to serve as [role] for this project?

2. What role did you play prior to becoming [role] for this project?

3. Do you currently hold other roles in addition to being the [role]?

2. What are your responsibilities as [role]? (Confirm that nothing has changed since the telephone interview).

If needed, probe for:

1. Liaison with other partners, such as ED, evaluators, etc?

2. Serve as the grantee representative? In what ways?

3. Participate in internal evaluations?

4. Reporting

3. What are the most important initiatives in your state/district/school at this time? (Has this changed since we interviewed you on the phone?) (Note: this will be used as background for questions about how TIF fits into a broader plan and/or recent PFP history.)

1. Please describe any other initiatives focused on improving teacher quality.

II. Context/Participation in TIF

4. Have there been any changes in your State/district/school pay for performance initiatives or plans since we last interviewed you in [date]?

1. Are there successes of those projects that you hope to replicate here? Are there difficulties/failures of those projects you hope to avoid?

2. How does the TIF project relate to the pre-existing project?

5. What are the major reform initiatives in your State, district, and school? What has changed since we spoke on the phone?

6. Please describe your State/district/school’s process for refining your TIF Project.

1. What were the major issues?

III. Project Design

7. How have the key goals of the pay for performance project changed since the start of the initiative?

8. Who is eligible to receive an award in the pay for performance project (school/administrator/teacher)? Has this changed since the start of the initiative?

1. For each group/category, please provide the number of eligible schools/administrators/teachers if known.

9. What is the target goal for number/percentage of participating schools/administrators/teachers (if known)?

1. What is the plan for increasing/sustaining participation?

2. Do you foresee any challenges in implementing the plan?

10. Please provide the number of participating (schools/administrators/teachers).

1. Could individuals opt-in or was participation mandatory? How many opted out?

2. If mandatory for some but not others, how was that determined?

11. How many students were affected by the project (based on scope of project and participation rates)? What proportion of all students in the school/district do these students represent?

12. What outcomes and/or activities are rewarded in the project? Please describe any changes since our telephone interview.

13. Is professional development part of the project? What, if anything, has changed over time?

1. Are all teachers/principals required to attend professional development to be eligible for an award under [name of the project]? What type of follow-up is required?

2. Is professional development part of the award criteria? How much weight does it have in making awards?

3. Do you know what percentage of your teachers have participated in professional development since the start of [name of project]?

14. In addition to PD, what supports are in placed to help participants earn a reward? How have these changed?

15. What is the award range for each (school/administrator/teacher)? Has this changed?

1. How is the award amount for each activity determined?

2. What percent of the award is designated toward each activity?

3. What is the average amount (or typical) amount awarded to schools/administrators/teachers?

4. Do you consider these awards to be substantial, relative to the existing pay of teachers/administrators and the amount of additional work/responsibility they take on?

16. How many teachers have received awards each year (if known)?

1. If there are multiple categories of teacher awards, please provide the number of eligible teachers and award recipients for each teacher award category, if known.

17. What are the criteria for receiving each award? Has this changed?

18. Please describe the annual award payment schedule for school/administrator/teacher-level awards including if the awards are paid in full at one time or provided in parts.

1. When were the first awards distributed?

19. What data are currently available in the district/State about teacher and/or principal performance? How long has this data system been in place?

1. Are data available on teacher educational background, years of experience, date of hire in school or district?

2. Are students’ data linked to teachers?

3. How are these data used to make performance pay decisions? How is that process going?

III. Project Implementation and Communication

20. How were project components communicated to teachers, principals, and the community?

1. How do schools stay informed about [name of project]?

21. How well was the initiative implemented? How did the rollout go?

1. What were the barriers to successful implementation? What worked well?

2. Were there differences in rollout across schools and districts?

3. Which stakeholders were especially difficult to bring on board (if any)?

4. Which stakeholders were involved in rolling out the pay for performance project?

22. What changes to the project have been made or are expected? Changes may include incentivized activities, eligibility, award levels, timing of data collection/awards, communication strategies, etc.

1. What issues arose that caused these changes to be recommended?

2. How and when will changes be implemented

IV. Evaluation

23. How do you know if the project has achieved its goals? How do you measure progress?

24. Has the internal evaluation produced and reports and had any findings? Has the internal evaluator changed?

IV. Sustainability

25. What is the district/State/school plan for continuing the pay for performance project when TIF funding expires?

1. Increased assumption of costs?

2. Offsetting with reduced teacher turnover costs?

3. Does the district/school plan to receive outside support from partners or other funders for continuation of the project?

V. Perceived Outcomes

26. Have you noticed a change in the qualifications of applicants since the initiation of your pay for performance project?

27. Have you seen a change in turnover rates since initiation of your pay for performance project?

1. Would you attribute this change to the project? Completely? Partially?

28. What evidence is there that principal leadership has improved as a result of the incentives?

29. What evidence is there that teacher practice has improved as a result of the incentives?

30. What evidence is there that student learning has improved as a result of the incentives?

VIIII. Closing

31. From your perspective as [role], what do you think would improve the effectiveness of the performance pay project?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 7 of 7