TIF Educator in TIF Sites Visit 2 Interview Protocol

Site Visit 2: Educators in TIF sites

Face-to-face visits in 12 TIF sites – Spring 2012

Teacher Incentive Fund Evaluation

Background for Site Visit 2: Educators in TIF Sites

Purpose and Notes:

The second set of site visits will be conducted in year four of the evaluation (spring 2012). Twelve sites will be selected and each visit will include three days of interviews by a two-person team of researchers to determine progress in implementation. (See below for more information on the selection of sites).

The purposes of the site visits are to (1) verify data gathered through the telephone interviews and project documents, and (2) complement the interviews with information about stakeholder satisfaction and potential for sustainability that can best be gathered through direct observation and in-person discussion. Researchers will observe the format and content of the funded activities, the types of activities teachers are participating in, the work teachers are expected to perform, and the objectives and goals teachers are expected to reach. In particular, site visits will be crucial in the event that subsequent outcome analyses reveal a significant influence of TIF on outcomes such as retention of effective teachers in high-need schools. By discussing stakeholder satisfaction and observing school climate in the TIF environment, the contractor will gain tools for potentially explaining the correlation of TIF projects and improvements with teacher outcomes.

This semi-structured interview protocol contains all of the questions that might be asked to school employees—particularly principals and teachers. The protocols have been designed in such a way as to be broad enough to encompass the breadth and variation we expect to encounter both in who we talk to (and how much they know) as well as the variability in the projects themselves. Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions which are appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Prior to the interviews, individuals will be contacted by their TIF project office regarding the evaluation.

This protocol identifies key topic areas for educators. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Information collected in the telephone interviews conducted in the fall of 2009

• If applicable, information collected in site visit 1 in spring 2010

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: There will be variation in the groups of principals and teachers interviewed. Some will have been deeply involved in the development and implementation of the initiative and others not. Generally, we need to determine the interviewees’ level of involvement and expected knowledge of the initiative.

Sample Information

An optional sampling approach for the second round of case study data collection is recommended. This option will require conducting the outcome analysis prior to the second round of case studies in order to sample based on grantee outcomes. This approach will provide in-depth data on the grantee plans, practices, and contextual factors that lead to a range of student and teacher outcomes. Sites will be selected using a stratified random sample of three high-performing projects and one low-performing project (with performance defined based on outcome measures in the impact study) from each of the three types of grantee projects:(1) projects in which 50 percent or more of the TIF grant activities award is allocated to merit pay (based on improved student achievement), (2) sites primarily implementing broader forms of differentiated compensation (e.g., increased pay for teaching in specific schools or subjects), and (3) sites implementing the most comprehensive systems (combining merit pay with various compensated teacher professional development activities, e.g., new teacher career ladder projects; or Teacher Advancement Program/TAP).

	Performance Pay	Differentiated	Comprehensive
High Performing	3 grantees	3 grantees	3 grantees
Low Performing	1 grantee	1 grantee	1 grantee

Site Visit 2 Interview Protocol: Educators in TIF Sites

I. Background

1. Tell me about your background. (If you have already interviewed this respondent, please confirm previous data).

   1. What is your current job?

   2. How and when did you begin this assignment?

   3. What role did you play prior to your current position?

   4. Do you participate in the TIF project?

2. What are the most important initiatives in your State/district/school at this time?

   1. If there are any other initiatives focused on improving teacher quality please describe. (Note: probe about specific initiatives including where they come from and why they matter. Some respondents may not have detailed information)

II. Project Design and Implementation (Note:—probe for any changes in the project design since previous data collection).

3. What are the key goals of the pay for performance project? How does the project fit into the broader goals for your school?

4. What outcomes and/or activities are rewarded in the project? Please describe. (Note:—probe for any changes in the project since previous data collection).

5. What are the criteria for receiving each award?

   1. Have there been any changes to the award criteria since the initial year of the awards? Probe for the reason for these changes.

   2. Are the criteria considered fair and understandable?

6. Do you participate in the project?

   1. What motivated you to participate?

   2. If so, have you received any awards? What have you received awards for?

   3. How do you think implementation has gone? Are there particularly notable successes or challenges that have affected your perception of the project?

   4. Overall, how fair do you think the TIF project is? Why?

III. Outcomes: Principal Recruitment, Retention, and Leadership (for principals only)

7. What aspects of principal recruitment, retention or leadership are TIF designed to address?

8. How has TIF affected your practice and your decisions about where to work? Probe for:

   1. Decisions about moving to or staying in a hard-to-staff school

   2. Decisions about leading a low-performing school

   3. Amount of time spent on teacher evaluation

   4. Data-driven decision-making

   5. Monitoring of instruction

   6. Teacher development

   7. Communication with parents and community

9. Have there been any challenges or success in implementation that have affected the outcomes attained?

IV. Perceived Outcomes: Teacher Recruitment and Retention (questions would be slightly modified based on whether respondent was a principal or teacher)

10. What strategies do you use to recruit teachers? (For new teachers ask about their own recruitment process.)

    1. Are there any particular subjects for which qualified candidates are difficult to find?

11. What are the characteristics of typical applicants to your school/district? (Principal only)

1. Level of education

2. Certification(s)

3. Experience

12. Have you noticed a change in the qualifications of applicants since the initiation of your pay for performance project? (Principal only)

13. How many vacant positions exist in your school in a typical year? (Principal only)

    1. Have you noticed a change in the number of vacant positions or the difficulty of filling them since the initiation of your pay for performance project?

    2. Would you attribute this change to the project? Completely? Partially? Why do you attribute the change to the project?

14. What is the average teacher turnover rate for your school in a given year? (Principal only)

15. Have you seen a change in turnover rates or applications since initiation of your pay for performance project? (Principal only)

    1. Would you attribute this change to the project? Completely? Partially? Why do you attribute the change to the project?

16. Can teachers receive an award specifically for teaching in your school?

17. How has TIF contributed to the overall working conditions in your school? (Probe: could be related to ability to assign teachers based on their expertise, better induction and PD projects, accessibility of high-quality curriculum)

V. Perceived Outcomes: Teacher Practice (questions would be slightly modified based on whether respondent was a principal or teacher)

18. Has teacher practice or the way teachers teach changed because of TIF? How? (Describe. Probe for

    1. content taught,

    2. strategies used,

    3. instructional planning processes,

    4. alignment to standards or tests,

    5. use of formative and/or summative assessment data,

    6. articulation and coherence across grades/subjects).

19. How do you know teacher practice or the way teachers teach has changed?

20. Have there been any challenges or success in implementation that have affected the outcomes attained?

VI. Perceived Outcomes: Student Learning

21. What evidence is there that student learning has improved as a result of the incentives?

22. Have there been any challenges or success in implementation that have affected the outcomes attained?

VII. Sustainability

23. What is the State/district/school plan for continuing the pay for performance project when TIF funding expires?

    1. Do you anticipate any changes to the project?

    2. How long do you think your state/district/school will have a pay-for-performance project?

    3. To your knowledge, is the State/district/school going to fund the project beyond the length of the TIF grant?

VIIII. Closing

24. From your perspective as [title of respondent], what do you think would improve the effectiveness of the performance pay project?

25. Is there anything else you think we need to know about the TIF project in [grantee name]?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 6 of 6