District Interview Protocol

Teacher and Leader Evaluation Systems Study:

District Interview Protocol

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB number. The valid OMB control number of this information collection is 1850-0890. The time required to complete this information collection is estimated to average 90 minutes per interview. If you have any comments concerning the accuracy of the time estimates(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Melanie Ali, U.S. Department of Education, Institute of Education Sciences, Room 502B, 555 New Jersey Avenue, NW, Washington, DC 20208-5500.

[Date]

[Recipient Title]

[Recipient Street Address]

[Recipient City, ST Zip]

Dear [Recipient],

Thank you for your district’s continued participation in the Teacher and Leader Evaluation Systems Study. I am writing regarding the upcoming district interview. I have attached a copy of the interview questions for your review. I encourage you to review the questions beforehand in case you need to consult a colleague regarding a question prior to the interview.

In preparation for the interview, I’d like to acquire some documentation regarding your systems for evaluating teachers and principals that are in effect in 2012-13. In particular, I am requesting that you send me blank copies of the forms you use for rating teacher and principal performance. If you have a separate form for classroom observations, please send that as well. In addition, I am requesting any written guidance that you provide to those who complete the ratings. Guidance may include, but is not limited to, evaluator handbooks, manuals, or memos regarding how to rate instruction. Gathering these documents is an important part of our data collection process and will help us ensure that we are fully prepared for our interview with you.

I appreciate your assistance in helping us collect this information. Please send us this information at your earliest convenience. Electronic versions of these forms can be mailed to awayne@air.org, or sent via standard mail to my address, below.

Sincerely,

Andrew Wayne, Ph.D.

Project Director, TLES Study

American Institutes for Research

1000 Thomas Jefferson Street, NW

Washington, D.C. 20007

DISTRICT INTERVIEW PROTOCOL

I. Respondent Background Information

The questions in this section will be asked of all respondents to verify that each respondent has the information necessary to answer the interview questions. In setting up the interview, the interviewer will work with each district to identify respondent(s) with the necessary knowledge to respond to our questions and allocate questions to respondent(s) accordingly.

First, we would like to discuss your specific responsibilities related to the topics we are going to ask about.

1. Information related to district role in teacher and principal evaluation

1. Please describe your position in the district and your role in the following areas, if any:

1. Teacher evaluation

2. Teacher transfer and staffing

3. Teacher professional development

4. Principal evaluation and professional development

5. Implementation of activities associated with TLES

II. Questions Related to Teachers and Teacher Evaluation

The questions in this section will be asked of respondents who are knowledgeable about policies and practices related to teachers and teacher evaluation.

Now, I’d like to focus on current policies and practices related to teachers. I will be asking specifically about policies that are in effect for the 2012–13 school year.

2. Teacher evaluation policies and practices

1. Ratings of teacher performance

   1. According to your district’s human resources policies, are all teachers supposed to receive an annual performance rating? Does the frequency differ depending on the type of teacher (e.g., novice teachers, teachers who received a satisfactory rating in the past)? If such ratings are not given annually, how often do teachers receive an overall performance rating?

   2. Is the overall rating derived by completing one of the forms that you provided? [Determine which form collected ahead of the interview is the correct form]

   3. What sources of information are used to determine the overall performance rating? Are the ratings based in part on:

      1. Observations of classroom practice? [If yes, ask Question A.2 below]

      2. Analyses of the achievement test scores of the teacher’s students? [If yes, ask Question A.3 below]

      3. Administrator’s judgments about other aspects of performance (e.g., communication with parents). [If yes, obtain a brief description]

      4. Other sources (e.g., portfolios). [If yes, obtain a brief description]

   4. Are there rules for how information from these sources should be combined to create the overall performance rating?

2. Details on classroom observations as a source of information for teacher performance ratings

   1. To help determine performance ratings for teachers, how often are teachers required to be observed in their classroom? Does the answer differ depending on the type of teacher (e.g., novice teachers, teachers who received a satisfactory rating in the past)?

      1. Are there rules about how long observations typically need to last?

      2. Are there rules about whether observations should be announced in advance?

   2. Who conducts classroom observations of teachers (e.g., principals, assistant principals)?

   3. Are the criteria for classroom observations those that appear on the overall evaluation form we discussed earlier, or is there a separate form used for classroom observations? [If separate, confirm understanding of which of the forms collected ahead of the interview is the form for classroom observations]

   4. Are those who observe given flexibility to use a different form or different set of criteria? If so, please explain.

   5. Does the district require that observers be trained in how to conduct classroom observations and/or how to provide feedback based on an observation?

      1. Who is responsible for providing the training to classroom observers?

      2. Approximately how many hours of training are spent on how to rate teacher practice through observation?

      3. Approximately how many hours of training are spent on how to provide feedback?

      4. What, if any, assessment do observers need to pass before they conduct classroom observations for purposes of rating teachers?

3. [If applicable] Details on student achievement as a source of information for teacher ratings

   1. In taking student achievement into account to determine the annual performance ratings for teachers, are there requirements about which types of student assessment data may be used? [If yes] Which of the following may be used [Note yes or no for each]:

      1. State mandated tests

      2. District mandated tests

      3. Final examination grades

      4. Other (please specify)

   2. [If not already addressed] For teachers in untested grades or subjects, is there a different policy regarding how student achievement is to be taken into account? If so, please briefly describe the policy.

   3. In analyzing the student achievement information, is it your policy to focus on the level of student achievement at the end of the year or the growth in achievement over the course of a year? If growth is considered, how is growth determined?

4. Performance feedback for teachers

1. How often are teachers supposed to receive feedback on their performance? For example, are teachers given feedback after each observation of their teaching? Are there other occasions when teachers are supposed to receive feedback?

2. For these feedback occasions, I’d like to know whether teachers are supposed to receive (i) oral feedback, (ii) ratings, and/or (iii) a written narrative of some kind.

3. After an observation, is there a time frame within which feedback must be given?

5. Use of teacher performance information

1. Can teachers earn additional pay based on their performance?

2. Are teachers required to set professional growth goals? In what way, if any, is teacher performance taken into account in goal setting?

3. Are certain teachers required to participate in professional development based on their performance?

4. In what way, if any, is teacher performance taken into account in determining whether teachers earn tenure in your district?

5. Are some teachers on contracts that have to be renewed? If so, in what way, if any, is teacher performance taken into account in decisions regarding contract renewal.

6. Were any teachers identified for layoffs in the last 12 months? If so, in what way, if any, was teacher performance taken into account in decisions regarding layoffs.

7. Were any teachers terminated for performance reasons in the last 12 months?

8. In addition to the uses of individual teacher performance information, which were just mentioned, does the district use teacher performance information that has been aggregated (e.g., school-level summary, district-level summary)? For what purposes is such information used (e.g., for planning district-wide professional development)?

3. Teacher mobility policies and practices

Next, I would like to ask you about policies related to teacher mobility in your district. We would like to understand the process by which teachers change schools.

1. First, I have some questions about teachers who seek to transfer from one school to another within the district—that is, voluntary transfers.

1. Which teachers are eligible to voluntarily transfer? How many years of experience (in the district) do teachers need before they can voluntarily transfer?

2. Do certain types of voluntary transfers receive priority over others in filling vacancies (e.g., because of seniority)? If so, which teachers receive priority?

3. Can voluntary transfers fill positions that are currently held by novice teachers or other types of teachers?

4. Do principals have any discretion regarding voluntary transfers?

2. I also have some questions about teachers who are involuntarily transferred between schools. Under what circumstances can a teacher be involuntarily transferred?

1. When school enrollments change, in what ways do principals have discretion to select the teachers who will be involuntarily transferred?

2. Are there other circumstances under which a teacher can be transferred involuntarily? What role does the principal play in identifying who will be transferred under those circumstances?

3. Were any of the policies you just described new in 2012–13? [If so:] Could you say a little about the timeline for the implementation of that policy? I’m trying to get a sense of whether it was in full effect throughout the 2012–13 school year.

4. Implementation of TLES components for teacher evaluation

Now, I’d like to turn to the evaluation system components related to teacher evaluation that have been implemented in the pilot schools only—the schools that were randomly assigned to receive services using the study’s resources. My first questions are about the classroom observation system, and then I have some questions about the student growth system.

1. Approach to implementing TLES components

1. After the initial summer training for those who conduct classroom observations, we had the principals and DSOs convene for additional support meetings approximately once per month. Have you done anything outside these meetings to help principals and DSOs successfully conduct classroom observations and feedback sessions?^

2. Have you encountered any challenges with the performance of principals in observing teachers and providing feedback? What about DSOs? How did you address those challenges?

3. We provided some parameters for scheduling the observations. Have you encountered any challenges in ensuring that the observations were conducted in the specified time windows. What strategies were used to overcome any challenges? [Ask with respect to (i) principals and (ii) DSOs]

4. Have you personally used the [CLASS/FFT] reporting system, or directed others to use it on your behalf?

1. Please indicate if you used the reporting system for management of the implementation of the classroom observations. Did you use it for tracking purposes, to ensure that observations were talking place? Did you use it to monitor the quality with which observation reports were being prepared by observers?

2. Please indicate if you used the reporting system to access performance information. Did you use it to access individuals’ performance information? Did you use it to access performance information for particular groups of teachers (e.g., all teachers in a particular school).

3. Were there any challenges that limited your use of the [CLASS/FFT] reporting system? Were there challenges in getting access to the reporting system, or were there challenges in understanding the reports that the system provides? How did you overcome these challenges?

4. Can you identify any suggested improvements to the [CLASS/FFT] reporting system?

1. Have you personally used the student growth reporting system, or directed another person to use it on your behalf?

1. Please indicate how you used the system. Did you use it to access individuals’ performance information? Did you use it to access performance information for particular groups of teachers (e.g., all reading teachers)? Did you use it to access student growth information for individual schools?

2. Were there any challenges that limited your use of the student growth reporting system? Were there challenges in getting access to the reporting system, or were there challenges in understanding the reports that the system provides? How did you overcome these challenges?

3. Can you identify any suggested improvements to the student growth reporting system?

1. Was the webinar provided shortly after the release of the student growth results sufficient, or did you perceive a need for additional support?

2. Approach to integration of TLES and current evaluation system practices

1. In your implementation plan, you indicated that the district [WOULD/WOULD NOT] use TLES observations in place of the observations required under the district’s current policy. Has this decision given rise to any challenges for the district, and how were those challenges overcome?

2. In your implementation plan, you indicated that TLES performance information [SHOULD/COULD/SHOULD NOT] be taken into account in giving performance ratings to teachers. Has this decision given rise to any challenges for the district, and how were those challenges overcome?

3. To clarify, could performance information from TLES be used in (i) tenure review, (ii) contract renewal, or (iii) termination proceedings?

3. Implementation of additional, local uses of TLES system

In reviewing your implementation plan, I see that you have planned several activities to make use of the TLES system for teacher evaluation. For study purposes, we are very interested in knowing which of these activities were implemented and when. We’d also like to know about the challenges you have encountered and the solutions that you have developed to overcome those challenges.

[The interviewer will refer to the implementation plan as updated after the spring 2013 implementation planning visit. Because there is no spring 2014 implementation planning visit, the interviewer for the spring 2014 district interview will need to begin by asking about the status of each activity (i.e., implemented vs. not implemented) before proceeding with the questions.]

1. [For each activity that has reportedly already been implemented] I see you have already implemented [ACTIVITY].

1. Could you briefly describe the timeline for implementing this activity? I’m trying to get a sense of whether it was in full effect throughout the 2012–13 school year. [Determine when the activity was initiated and the extent of implementation achieved since initiation. For example, indicate how many pilot schools implemented the activity and what proportion of teachers in those schools were likely affected.]

2. In implementing this activity, did you encounter any challenges, and how did you overcome those challenges?

2. [For each activity that has reportedly NOT yet been implemented] I see you have not yet implemented [ACTIVITY].

1. Could you say briefly why this activity has not yet been initiated? Have you encountered any specific challenges?

III. Questions Related to Principals and Principal Evaluation

These questions will be asked of respondents who are knowledgeable about policies and practices related to principals and principal evaluation.

Now, I’d like to focus on current policies and practices related to principals. I will be asking specifically about policies that are in effect for the 2012–13 school year.

5. Principal evaluation policies and practices

1. Ratings of principal performance

   1. According to your district’s human resources policies, are all principals supposed to receive an annual performance rating? Does the frequency differ depending on the type of principal (e.g., novices, those who received a satisfactory rating in the past)? If such ratings are not given annually, how often do principals receive an overall performance rating?

   2. Is the overall rating derived by completing one of the forms that you provided? [Determine which form collected ahead of the interview is the correct form]

   3. What sources of information are used to determine an overall performance rating? For each source, please indicate how often it is required to be used.

1. [If not already discussed]Are principals required to complete a self-assessment as part of the principal evaluation process?

2. [If not already discussed]In what way, if any, is input from teachers supposed to be taken into account to determine performance ratings for principals?

2. [If applicable] Details on student achievement as a source of information for principal performance ratings

   1. In taking student achievement into account to determine performance ratings for principals, are there requirements about which types of student assessment data may be used? [If yes] Which of the following may be used [Note yes or no for each]:

      1. State mandated tests

      2. District mandated tests

      3. Final examination grades

      4. Other (please specify)

   2. In analyzing the student achievement information, is it your policy to focus on the level of student achievement at the end of the year or the growth in achievement over the course of a year? If growth is considered, how is growth determined?

3. Performance feedback for principals

1. How often are principals supposed to receive feedback on their performance from their supervisor? For each occasion on which feedback is expected, I’d like to know whether the feedback is supposed to be based on the form identified earlier, or any other form. I’d also like to know whether the feedback is supposed to include (i) oral feedback, (ii) ratings, and/or (iii) a written narrative of some kind.

2. Were any principals terminated for performance reasons in the last 12 months?

3. In addition to uses of individual principal performance information, does the district use principal performance information that has been aggregated (e.g., district-level summary)? For what purposes is such information used (e.g., for planning district-wide coaching of principals)?

6. Implementation of TLES components for principal evaluation

Now, I’d like to turn to the evaluation system components related to principal evaluation that have been implemented in the pilot schools only, using the study’s resources.

1. Approach to implementing TLES components

   1. First I have some questions about your experiences with administering the VAL-ED survey

1. In administering the survey, did you encounter any concerns about the survey from the intended respondents? Please describe any concerns you encountered, and the type of respondent who expressed the concern (e.g., elementary school teachers). If you tried to address the concern, how did you do so, and were you successful?

2. Did you have any challenges in getting the intended respondents to respond and to submit their responses within the intended timeline? How did you address those challenges?

3. Did you think the respondent training was adequate? Did you do anything to supplement the respondent training?

1. My next questions are about feedback given to principals after each VAL-ED administration.

1. Did you encounter any challenges in the feedback sessions with principals? If so, what were the challenges, and how did you overcome those challenges?

2. Did you think the supervisor training was adequate? Did you do anything to supplement the supervisor training?

1. I also have some questions about the VAL-ED reporting system. Did you personally use the reporting system, or direct others to use it on your behalf?

1. Please indicate if the reporting system was used to access performance information. Was it used to access individuals’ performance information? Was it used to access performance information for all principals collectively?

2. Were there any challenges that limited your use of the VAL-ED reporting system? Were there challenges in getting access to the reporting system, or were there challenges in understanding the reports that the system provides? How did you overcome these challenges?

3. Can you identify any suggested improvements to the VAL-ED reporting system?

2. Approach to integration of TLES and current evaluation system practices

1. In your implementation plan, you indicated that TLES performance information [SHOULD/COULD/SHOULD NOT] be taken into account in giving performance ratings to principals. Has this decision given rise to any challenges for the district? If so, how were those challenges overcome?

2. To clarify, could performance information from TLES be used in ways that had consequences for principals (e.g., promotion, termination, etc.)? Please explain.

3. Implementation of additional, local uses of TLES system

In reviewing your implementation plan, I see that you have planned several activities to make use of the TLES system for principal evaluation. For purposes of the study, we are very interested in knowing which of these activities were implemented and when. We’d also like to know about the challenges you have encountered, and solutions that you have developed to overcome those challenges.

[The interviewer will refer to the implementation plan as updated after the spring 2013 implementation planning visit. Because there is no spring 2014 implementation planning visit, the interviewer for the spring 2014 district interview will need to begin by asking about the status of each activity (i.e., implemented vs. not implemented) before proceeding with the questions.]

1. [For each activity that has reportedly already been implemented.] I see you have already implemented [ACTIVITY].

1. Could you briefly describe the timeline for implementing this activity? I’m trying to get a sense of whether it was in full effect throughout the 2012–13 school year. [Determine when the activity was initiated and the extent of implementation achieved since initiation. For example, indicate how many pilot schools implemented the activity and what proportion of teachers in those schools are likely affected]

2. In implementing this activity, have you encountered any challenges, and how have you overcome those challenges?

2. [For each activity that has reportedly NOT yet been implemented] I see you have not yet implemented [ACTIVITY].

1. Could you say briefly why this activity has not yet been initiated? Have you encountered any specific challenges?

IV. Use of Time

These questions will be asked of respondents who are knowledgeable about time spent by district staff in support of the student growth component of the study.

7. Use of time to support student growth component of study

One purpose of our interviews is to determine the total amount of time spent by district staff in working with the American Institutes for Research (AIR) to implement the student growth component of TLES. Please answer the following questions with respect to the most recently completed round of student growth reports.

1. After the district agreed on the covariates to be included in the value-added model, how many central office staff members were involved in working with AIR to implement the student growth component of TLES? If applicable, please divide the staff into those considered “regular” and “senior.” How many regular staff were involved, and how many senior staff were involved?

2. I’m interested in determining the amount of time that regular and senior staff spent after the district agreed on the covariates to be included in the value-added model. For each of the following, please provide an estimate of the total amount of staff time spent by regular staff and the total time spent by senior staff.

1. Discussing data requests with AIR

2. Preparing and submitting data to AIR

3. Troubleshooting and resubmitting information as needed

This item and some additional items on this protocol are shaded to indicate that the burden for the items was not counted in the burden estimate. The respondent for these items will be the individual in the district who is paid by the study to assist with study coordination activities.