TIF Program Staff Telephone Interview Protocol

Interview: TIF Project Staff

Telephone interviews in all 34 project sites – Fall 2009

Teacher Incentive Fund Evaluation

Background for Interview Protocol: TIF Project Staff

Telephone interviews will be conducted in the fall of 2009 with key informants at all 34 TIF sites. The purpose of grantee telephone interviews is to verify grantees’ incentive project descriptions and address questions about the implementation of the TIF projects from a variety of perspectives.

This semi-structured interview protocol contains all of the questions that might be asked to TIF project staff. Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions which are appropriate to each respondent and for which researchers do not already have verified data. The section below on grantee variation explains the rationale for this protocol structure and the nature of the training necessary to consistently implement these protocols.

Based on the uniqueness of each TIF project as well as the dynamics of the districts or States in which they operate, the protocols have been designed to maximize the information collected from each individual while minimizing the burden on their time. Key informants at each site and for each role will be identified with assistance from the TIF program office in the U.S. Department of Education along with the grantee’s project leadership (most often the project director). Both the number and the nature of key informants/respondents will vary somewhat based on the specifics of each grantee. For example, in a TIF site in which the grantee is a State entity and the project includes multiple districts and many schools, there will likely be roles filled by a number of individuals in the project office. In other sites, such as in the case of a small network of charter schools, a single person may represent the project office. The breadth of involvement will also vary from site-to-site making the flexible protocols essential to the smooth running of the interviews. For example, the technical assistance providers may be integrally involved in the project for some grantees while playing a smaller role in others just as the scope and stage of evaluations may vary.

This protocol identifies key topic areas for project directors, other project staff, TA providers, and evaluators. In preparation for the interviews, researchers will review the following:

• TIF grantee profile prepared (and updated over the course of the study) based on documents provided by the TIF program office in the Department and the Center for Educator Compensation Reform (CECR)

• Annual performance reports

• Evaluation reports

• Demographics of schools and students within the project

Each interview will start with review and signature of the consent form sent electronically. Researchers will also provide their contact information in the event that a respondent needs to follow-up with additional information.

Grantee Variation

The 34 TIF grantees vary widely across a number of attributes that are reflected in the protocols. Because of the variation in the grantees, it is imperative to have flexible protocols so that each interview is tailored and appropriate for a given grantee’s experiences and project structure. Many of the grantees are local school districts, but grantees also include State agencies, individual schools and non-profits (such as charter schools or charter school networks). They also vary geographically (a grantee may be a State or a single school for example) as well as in the number of eligible educators (from fewer than 100 to more than 10,000).

Interviewer training and preparation will focus on customizing each interview appropriately to respond to the variation in grantee characteristics. The training will help team members develop common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which interview study data will be used.

Before beginning data collection, interviewers will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, available background documents for the grantees). Interviewers will review extant documents that have been submitted to the Department, including grantee applications, annual reports, research reports, and background materials on the grantees and schools to be visited and pre-populate the protocols with the information in these documents. Interviewers will highlight the specific sections or questions of the protocols for each informant and will and tailor the language to reflect the grantee (e.g. State, district, school, or non-profit).

How to use the protocols: EACH INTERVIEWER SHOULD REVIEW EXTANT DOCUMENTS AND DRAFT ANSWERS TO EACH OF THE QUESTIONS IN THE PROTOCOL PRIOR TO THE INTERVIEW. To reduce the potential for redundancy, make sure the interviewee is aware that you have reviewed available documents so they can reference information contained in the documents.

Background: Establish which sections of the protocol are appropriate to this interviewee based on their roles, responsibilities and experience (e.g., how long they have held their current position and/or previous positions).

Context/Participation in TIF: It is important to know whether the grantee (e.g. State, district, individual school or non-profit) has a history of performance pay or whether they are starting a new project with no prior experience. In training, you will be provided with a preliminary list of grantees with known pre- or co-existing incentive pay projects. If you confirm (through review of extant data or responses to these questions) that the grantee has another performance pay project co-existing with TIF, please ask respondents to (1) differentiate between TIF and other sources of support, and (2) describe the relationship between projects.

Planning and Project Design: This section is the heart of the interview. It is important that the interviewer capture the details of the project design. These interviewees may be eligible for awards under the project, but may have misconceptions about the project design. Additionally, in grantees with other pre- or co-existing performance pay projects, the interviewer can attempt to clarify whether the interviewees’ responses relate to TIF and/or the other project. However, these respondents may be unaware of the distinction between projects.

Implementation: This section is designed to determine the respondent’s perception of how well the initiative was implemented. We need to understand barriers, challenges, and successes. Also listen for changes to the original plan during implementation because of capacity (e.g. the plan was to offer certain opportunities but then there was neither staff nor funds).

Outcomes: This section tries to draw out evidence of whether or not teacher practice, principal leadership, or student learning has been affected by the initiative. This is limited to respondent perceptions of outcomes, which may affect how well the project motivates educators. Other aspects of the study will measure actual outcomes.

Closing: This is an opportunity for interviewees to reiterate what may be most important to them (successes and challenges) and to add anything that we did not ask about, but that they feel is important.

Note: This protocol covers a fairly broad set of interviewees. Be prepared for varying levels of knowledge about the initiative and tailor your interview accordingly. If a respondent is unable to answer a question, it may be appropriate to inquire about who would know the answer.

Telephone Interview Protocol: TIF Project Staff

I. Background

1. Tell me about your background.

1. How and when did you come to serve as [role] for this project?

2. What role did you play prior to becoming [role ] for this project?

3. Do you currently hold other roles in addition to [role]?

What are your job responsibilities?

1. Liaison with other partners, such as ED, evaluators, etc?

2. Serve as the grantee representative? In what ways?

3. Participate in internal evaluations?

4. Reporting

5. [For the Project Director] Are there other individuals you work with as a team on this project (e.g., administrative data, personnel, professional development)? What are their names/contact info? Would you be willing to contact them to let them know that we will likely need to contact them for successive interviews? What are their areas of qualifications, experience, education, and training?

2. What are the most important initiatives in your State/district/school at this time? (Note: this will be used as background for questions about how TIF fits into a broader plan and/or recent PFP history.)

a. Are there other initiatives focused on improving teacher quality and effectiveness? Please describe.

II. Context/Participation in TIF

3. Does your State/district/school have a history of pay for performance plans or an existing pay for performance plan? Does the TIF project build on an existing project?

1. Are there successes of those projects that you hope to replicate here? Are there difficulties/failures of those projects you hope to avoid?

2. How does the TIF project relate to the pre-existing project?

4. Did you participate in planning your TIF project? If so, please describe your State/district/school’s process for planning the initiative. If not, please describe your understanding of the process.

1. Why did your State/district/school apply?

2. Who participated in planning the project?

3. How long was the planning period?

4. What were the major issues?

III. Project Design

5. What are the key goals of the pay for performance project? Included discussion of “official” and unofficial goals (stated/unstated).

6. Who is eligible to receive an award in the pay for performance project (school/administrator; administrators and subset of teachers, e.g., academic only; all teachers regardless of subject etc.)?

1. For each group/category, please provide the number of “eligible” (to be defined) schools/administrators/teachers

7. What is the target goal for number/percentage of participating schools/administrators/teachers (if known)?

1. What is the plan for increasing/sustaining participation?

2. Do you foresee any challenges in implementing the plan?

8. Please provide the number of participating (schools/administrators/teachers).

1. Could individuals opt-in or was participation mandatory? Can you describe any differences between those who opted-in compared to those who did not?

2. If mandatory for some but not others, how was that determined?

9. How many students were potentially affected by the project (based on scope of project and participation rates)? What proportion of all students in the school/district/State do these students represent?

10. What outcomes and/or activities are rewarded in the project? Please describe.

11. Is professional development part of the project? If YES,

1. What type of professional development has been offered? How were decisions about topics, focus, who would develop and deliver?

2. Are all teachers/principals required to attend professional development to be eligible for an award under [name of the project]? What type of follow-up is required?

3. Please describe the kind of professional development paid for under TIF: are project funds used to pay for stipend for teachers interested in /eligible for participation?

4. Is professional development part of the award criteria? How much weight does it have in making awards?

5. Do you know what percentage of your teachers have participated in professional development since the start of [name of project]?

12. In addition to PD, what supports are in placed to help participants earn a reward?

13. What is the award range for each (school/administrator/teacher)?

1. How is the award amount for each activity determined?

2. What percent of the award is designated toward each activity?

3. What is the average amount (or typical) amount awarded to schools/administrators/teachers?

4. Do you consider these awards to be substantial, relative to the existing pay of teachers/administrators and the amount of additional work/responsibility they take on? How do you know this?

5. Have there been any changes to the award structure over time?

14. How many principals and teachers have received awards each year (if known)?

    1. If there are multiple categories of teacher awards, please provide the number of eligible teachers and award recipients for each teacher award category, if known.

15. What are the criteria for receiving each award?

16. What data are currently available in the district/State about teacher and/or principal performance? How long has this data system been in place?

1. Are data available on teacher educational background, years of experience, date of hire in school or district? (Note: for new teachers, these would likely serve as proxy measures for effectiveness rather than effectiveness per se.)

2. Are students’ data linked to teachers? How: describe the process used for linking: its efficiency, validity, fairness, etc.)

3. How are these data used to make performance pay decisions? How is that process going? (Take us through the stages of linking data to teachers, to making performance pay decisions, notifying teachers and others, appeals process if any, etc.: what does the process look like from beginning to end?)

III. Project Implementation and Communication

17. How were project components communicated to teachers, principals, and the community, including the media, if at all (narrative question b2)?

1. How do schools stay informed about [name of project]?

18. How has the project implementation gone so far?

1. Were there differences in rollout across schools and districts?

2. Which stakeholders were especially difficult to bring on board (if any)?

3. Which stakeholders were involved in rolling out the pay for performance project?

4. What worked or didn’t work?

5. What aspects of the project have been hardest or harder to put in place?

6. Were there any components that your [grantee] did not have the capacity to implement as designed? (Note: this could include having budget to offer the PD that was required, provide any support designed into their project)

19. What changes to the project are expected? Changes may include incentivized activities, eligibility, award levels, timing of data collection/awards, communication strategies, etc.

1. What issues arose that caused these changes to be recommended?

2. How and when will changes be implemented?

IV. Evaluation

20. Does the project have an internal evaluation? External evaluation?

1. Who is the external evaluator/contact info? How was the external evaluator selected?

2. Who is conducting the internal evaluation? To what extent do you include adequate formative evaluation procedures for ensuring feedback and continuous improvement in the operation of the project?

FOR EVALUATORS ONLY

21. Please describe the evaluation design. What is the analysis plan?

22. What data are available and what is the quality of the data?

23. What are you finding about student learning and other student outcomes? Teacher learning, skills, and other teacher outcomes (e.g., recruitment, retention, attendance?)

24. What have been the main challenges associated with the evaluation? (e.g., state and local data systems? Access?)

25. How are the evaluation results being used? By whom?

V. Perceived Outcomes

26. Have you noticed a change in the qualifications of applicants you would attribute to initiation of your pay for performance project? What gives you that impression?

27. Have you seen a change in turnover rates of effective (to be defined) teachers since initiation of TIF? Discuss magnitude of change.

    1. Would you attribute this change to the project? Completely? Partially? (Bear in mind that other factors—including the local economy and other changes in instructional projects—can often affect turnover rates in high-need schools: why do you attribute this change at least partly to TIF?) What other things would you say could be attributed to these changes?

28. Do you think that principal leadership will eventually improve as a result of the incentives? If so why? If not, why not?

29. Do you think that teacher practice will eventually improve as a result of the incentives? If so, why? If not, why not?

30. Do you think that student learning has improved as a result of the incentives? (Note: this is not the same as whether student test scores have gone up, but what gives you confidence—or does not give you confidence—that student learning will eventually improve as a result of TIF specifically as opposed to other pay-for-performance or teacher quality initiatives you may be undertaking?

31. What is the district/State/school plan for continuing the pay for performance project when TIF funding expires?

1. Increased assumption of costs?

2. Offsetting with reduced teacher turnover costs?

3. Does the district/school plan to receive outside support from partners or other funders for continuation of the project?

4. What do you think are the challenges to continuing the performance pay project?

VIIII. Closing

32. From your perspective as [role], what do you think would improve the effectiveness of the performance pay project?

Protocols: Evaluation of the Teacher Incentive Fund (TIF) Program #1875-NEW (3999)

Page 9 of 9