Alternative student outcomes for growth measures case studies

OMB Supporting Statement Part B: Alternative Student Growth Measures for Teacher Evaluation: Case Studies of Early Adopters

Appendix F: Teacher Interview Protocol

Instruction to interviewer: when the respondent is unable to respond completely to a question, please always follow up by asking for the names and contact information of other staff in the district who can provide more information.

Introduction and Consent for Audio Recording

The purpose of this section is to introduce the respondent to the study and request consent for audio recording.

1. Thank you for taking the time for the interview today. My name is [name] and I work for Mathematica Policy Research, which is conducting this study for the Mid-Atlantic Regional Educational Laboratory (REL). The Mid-Atlantic REL’s mission is to “build research alliances with education practitioners and policymakers to help states and districts use data to answer important issues of policy and practice.” The results of this study will inform publically available reports aimed at states and districts considerting implementing measure of teacher performance that include student growth. As I mentioned [in my email/when we spoke on the phone], we are conducting case studies of districts implementing measures of teacher performance that include student growth, but that measure growth on outcomes other than state assessments. These alternative outcome measures might include nationally normed assessments, end-of-course curriculum-based assessments, or “student learning objectives” (SLOs) developed by individual teachers for their own students. Student growth might be measured through a simple subtraction of end-of-year achievement from beginning-of-year achievement, through a student growth percentile method, through a value-added model, or implicitly in SLOs that are set based on an understanding of baseline achievement levels in a class at the beginning of the school year. We plan to examine what alternative outcome measures are used, how the growth measures are implemented, challenges and obstacles in implementation, how the measures are being used, and the effectiveness of these measures in differentiating teacher performance. We expect our findings to benefit states and districts considering the use of such alternative growth measures. The purpose of this interview is to gather information on the implementation of [insert specific growth measures used in the district] in the district from your perspective as a district administrator. We will be interviewing teachers, principals, and union representatives in the district to gather information on their perspectives as well. We will not identify you by name in the study reports.We also will not identify the districts included in the study in subsequent reports.

Before beginning, I would like to get your permission to record the interview. This will ensure that we accurately capture information about district policies and will prevent me from having to ask questions and take notes at the same time. The audio recordings from all individuals interviewed within your district will be compiled into a single district summary that is for internal analysis purposes only and will not be published or shared with anyone outside the study team. You will not be identified in the district summary. This summary will then be aggregated with other district summaries and analyzed for trends and variations across multiple districts. Do I have your permission to record the interview?

Type of Student Outcome and Growth Measure Used

1. What alternate measures of student growth are used in assessing your performance?

a. Prompt: Does the district use any end-of-course curriculum-based assessments?

b. Prompt: Does the district use student learning objectives (SLOs) for purposes of measuring student growth associated with individual teachers?

c. Prompt: Does the district use any end-of-year, nationally normed or summative assessments purchased from testing companies (not including interim or benchmark assessments)?

2. (If the district uses SLOs) Please briefly describe any formal guidelines the district provides to you regarding the development of SLOs. Is there a district-wide template used by teachers? What is the process for having your SLOs approved? Are you required or permitted to submit more than one SLO for a class? Do your SLOs involve classwide targets or separate targets for each individual student? From your perspective, are SLOs of comparable difficulty across all teachers in the district?

3. (If the district uses alternate assessments to measure growth) Does the district use a growth or value-added model, applied to a measure of student achievement other than the state assessment, to measure your performance? If yes, what student assessment is used?

4. What role did teachers play in the alternative measure(s) design process? Can you describe the process for rolling out the alternative measure(s) of student growth (including SLOs) in the district?

5. Did you receive any training or professional development related to the implementation of the measure?

Implementation of Data Collection and Analysis

1. I would like to learn more about the administration of these alternative student assessments in the district. (For each assessment) How is the assessment administered (e.g., timing, frequency, who administers)? What is the process for collecting assessment results?

2. What kinds of obstacles, if any, have you encountered in preparing for and administering student assessments or submitting student assessment results? How many extra hours, if any, have you had to devote to these tasks?

3. Have you made any changes to your approach to teaching or to the content of your lessons since the implementation of this alternative assessment or SLO?

4. Have there been any changes to the district’s approach to using alternative student growth measures? Have these changes resulted in notable improvements?

Use of Alternative Growth or VA Measure

1. For what purposes is the district using each alternative student outcome measure used in assessing teacher performance? Is it used for evaluating you? Is it used as part of a performance-based compensation system? Do teachers receive targeted professional development based on their performance? Does the measure affect teacher assignments? Are there any other ways in which the district is using these measures?

2. Do you receive any feedback regarding your performance on each measure? When and how is this feedback provided? Who provides this feedback to teachers? Is feedback provided in person? in writing? Is there a way for you to respond to the feedback?

3. Does your performance rating on the alternative measure affect whether you are awarded tenure?

Benefits and Drawbacks

1. Could you summarize some of the main benefits, from your perspective, of implementing this alternative VAM or growth measure of teacher performance in the district? For example, have you found that receiving this kind of information about your performance has had any benefits for your effectiveness in the classroom? How has receiving this kind of performance information affected your teaching, including how you feel about teaching and the way you set goals and relate to students?

2. What do you perceive as the primary drawbacks or costs of implementing the measure? Do you think the measure allows for a sufficient level of comparability across teachers on performance? Do you think the measure accurately reflects your contribution to student learning? If SLOs are used in the district, do you think all teachers are setting sufficiently high standards for student growth?

3. How do you perceive the district’s communication of the use of the alternative measure in evaluations to teachers? Do you think teachers in the district understand the measure and how they can change its level? Do you think teachers in the district have a positive or negative view of the use of alternative student outcomes to measure their performance?

District Context

1. Have there been any changes in the district administration or policies that may have affected the implementation of these alternative measures of teacher performance (for example, change in superintendent or implementation of a performance-based pay system)?

2. Did the district have the support of the teachers’ union/association for the implementation of the VAM or growth measure of teacher performance using an alternative student outcome? Why or why not?

Is there anything else that you think might be important to know about the development and implementation of this alternative growth or value-added measure in the district? Are any other key stakeholders in the district with whom you think it would be important for us to speak?

OMB Package: Alternative Student Outcomes for Growth Measures Case Studies 1