State Interview Protocol
7.28.11
State: |
Interviewer: |
Interviewee(s): |
Date/Time: |
Note to Interviewer:
|
Introduction
Thank you for taking the time to speak with me today. Just as a reminder, this interview is for the Equitable Distribution of Effective Teachers study, which is funded by the U.S. Department of Education. There’s considerable evidence that disadvantaged students are less likely to be taught by high-quality teachers. Under the No Child Left Behind Act, federal legislation has used teacher qualifications, such as having a college degree or state certification, to measure teacher quality. Federal policy is now moving towards using measures of teacher effectiveness, and encouraging the use of student achievement in those measures.
This study examines state and local efforts to develop measures of teacher effectiveness and to ensure that schools that serve high proportions of disadvantaged students have high-quality teachers. The study is intended to inform future federal policy making in these areas.
We’ll use the terms teacher quality, teacher qualifications, and teacher effectiveness frequently during this interview. Just to clarify, when we talk about teacher quality during this interview, we are referring to the general concept only. We also refer to two types of “measures of teacher quality.” We call these:
“Measures of teacher qualifications,” which includes characteristics of teachers that are thought to be related to teacher quality, such as highly qualified teacher status or years of experience; and
“Measures of teacher effectiveness,” which includes measures that use data on student achievement or instructional practice.
Our questions are about your state’s policies or practices. You signed a consent form which stated that our reports will associate your responses with your state but not with you as an individual. In addition, to capture the large amount of data your responses will provide, your interview will be recorded. The audio record may be shared with others within our evaluation team for purposes of analysis but will not be shared outside the evaluation team. All data that we collect from you will be stored on a password-protected computer network and destroyed at the conclusion of the study.
We know that state officials are very busy, so we’ve made an effort to collect as much information as possible through available documents. We’ll sometimes reference that information during the course of the interview to confirm that our information is correct. This interview will take about 2 hours, including time for follow-up.
Do you have any questions before we begin?
Overview
To begin with, can you tell me your title and a little about your responsibilities in the state related to teacher quality. Probe for relevant responsibilities regarding:
Federal programs
ESEA Title II
Race to the Top
Data systems
Teacher education, accreditation, or licensure
Other responsibilities
If interviewer is unable to find definitions of high quality or effective teachers online, ask: Does your state have a definition of what it means to be a “high-quality” teacher? Is that definition the same as HQT or does it go beyond it in any way? An “effective teacher”? If yes to either, ask a and b:
How long have you used these definitions?
Are your definitions currently undergoing any revisions?
If interviewer is able to find definitions of high quality or effective teachers online, ask a-d:
I found the following definition of a “high quality” teacher on your state’s website: ________________________. Is this the definition that the state currently uses?
I found the following definition of an “effective” teacher on your state’s website: __________________________. Is this the definition that the state currently uses?
How long have you used these definitions?
Are your definitions currently undergoing any revisions?
I’m interested in how your state approaches teacher quality in general. What are your top two or three approaches for improving the quality or effectiveness of teachers in your state? Why do you think these approaches will improve the quality or effectiveness of your teaching force?
Measures of Teacher Qualifications and Teacher Effectiveness
Which of the following measures of teacher qualifications did you collect in 2010-2011: [see list below]. Which did you look at by high and low-poverty schools, by high and low-minority schools, and by urbanicity (urban, suburban versus rural schools)? I’m also interesting in knowing what you are collecting this year, in 2011-2012. Are there any other measures of teacher qualifications or other related data that you are collecting in 2011-2012?
Interviewer will confirm, for 2010-2011 and 2011-2012, which of the following measures the state is using:
Percentage of classes taught by highly qualified teachers
Percentage of classes taught by certified general education teachers who have not demonstrated subject competency
Percentage of inexperienced teachers
Percentage of teachers with degree higher than bachelor’s
Percentage of teachers who graduated from “top-rated” teacher preparation programs
Rate of teacher absenteeism
Rate of teacher turnover
Other measures (e.g., credentialing exam scores, time-to-passage for licensure)
As I mentioned earlier, federal policy is moving towards measuring teacher quality using measures of effectiveness. One of the objectives of this study is to determine what states and districts are doing now with respect to such measures, using student achievement data, teacher observations, or some combination of measures.
During the 2011-2012 school year, is the state promoting the use of student achievement data, data based on demonstrations of instructional practice (such as observations or portfolios), or a combination, to measure teacher quality? If yes, ask a and b.
Does the state require all districts to use the same measures, or does the state provide guidelines for the districts to create such measures?
Just briefly, for what purposes are these measures used?
Teacher evaluation
Teacher compensation
Other
Measures of Effectiveness Using Data on Student Achievement
Questions for states that have measures that use student achievement data If state does not have a measure that uses student achievement data, skip to the subsection “Measures of Effectiveness Using Data on Instructional Practice.” |
I’d like to learn about how student achievement is used to measure teacher quality. My questions are mostly about the use of scores from standardized tests, although I’ll ask a question at the end about nonstandardized tests. When I say “standardized,” I mean that the same test is used, under the same testing conditions (e.g., the SAT-9 or your state test, [NAME]). Also, when I say “tested grades and subjects,” I mean any grades or subjects that are given a standardized student achievement test that you use to measure teacher quality.
For this part of the interview, I’d like to ask you a series of basic questions, and I’d like you to just answer yes or no, okay? For items 7-9, remind respondent to answer Yes/No if needed.
Let’s start by talking about how you use student achievement data from standardized tests to measure teacher quality in tested grades and subjects.
Ask a-c: For teachers in tested grades and subjects, does your state base your [TEACHER QUALITY MEASURE / GUIDELINES FOR A TEACHER QUALITY MEASURE] at least in part on…
analyses of standardized test data using scores from all students in that teacher’s school? (Yes/No)
analyses of standardized test data using scores from all students in that teacher’s grade? (Yes/No)
analyses of standardized test data using scores from that teacher’s assigned students? (Yes/No)
Now I’d like to talk about how you use student achievement data from standardized tests to measure teacher quality in subjects that are not tested.
To measure the quality of teachers in untested subjects, does your state [USE / PROVIDE GUIDELINES FOR DISTRICTS TO USE] student achievement data from standardized tests? If yes, ask a-c: Does your state base your [TEACHER QUALITY MEASURE / GUIDELINES FOR A TEACHER QUALITY MEASURE] for these teachers at least in part on…
analyses of data from an achievement test in another subject, using scores from all students in that teacher’s school? (Yes/No)
analyses of data from an achievement test in another subject, using scores from all students in that teacher’s grade? (Yes/No)
analyses of data from an achievement test in another subject, using scores from that teacher’s assigned students? (Yes/No)
Now I’d like to talk about how you use student achievement data from standardized tests to measure teacher quality in grades that are not tested.
To measure the quality of teachers in untested grades, does your state [USE / PROVIDE GUIDELINES FOR DISTRICTS TO USE] student achievement data from standardized tests? If yes, ask a-b: Does your state base your [TEACHER QUALITY MEASURE / GUIDELINES FOR A TEACHER QUALITY MEASURE] for these teachers at least in part on…
analyses of data using scores from all students in that teacher’s school? (Yes/No)
analyses of another configuration of students (please explain)? (Yes/No)
Thank you. I also have a couple of questions that apply more generally.
In general, do any of the measures we’ve discussed focus on changes in achievement, rather than achievement status at a point in time? If yes, identify exactly how and when data on achievement gains are used.
In general, do you have any [MEASURES / GUIDELINES FOR MEASURES] that use student achievement data from assessments that are not standardized assessments? If yes, identify exactly how and when data from nonstandardized assessments are used (e.g., use in grades/subjects without standardized achievement tests?) and describe any guidelines about the required features of nonstandardized assessments.
Is there anything else you’d like to tell me about how your state measures teacher quality using student achievement data?
Measures of Effectiveness Using Data on Instructional Practice
Questions for states that have measures that use instructional practice data If state does not have a measure that uses instructional practice data (observations, portfolios, or other practice data), skip to the subsection called “Developing Measures of Effectiveness.” |
Now I’d like to talk about your polices regarding use of data on instructional practice to evaluate teachers.
First, is instructional practice data given more weight for teachers in untested grades, or in untested subjects, than it is for teachers of tested grades or subjects?
Could you just briefly describe the nature of any [MEASURES / GUIDELINES FOR MEASURES] that use data on instructional practice? And, does the description differ across the applicable grade levels and subject areas? Confirm whether state uses a-c.
Classroom observations. (Yes/No)
Portfolio assessment. (Yes/No)
Other assessment that uses data on instructional practice. (Yes/No)
If state uses or has guidelines on classroom observations, ask a-e: Who conducts the classroom observations?
School principal. (Yes/No)
School-based content-area specialist (e.g., department chair). (Yes/No)
District-based content-area specialist. (Yes/No)
Other district-based staff. (Yes/No)
Other. Obtain explanation. (Yes/No)
Does your state have policies regarding the training of those who gather these data and determine the instructional practice ratings?
The state provides training directly to those who observe/rate
The state sets parameters regarding the training of those who observe/rate
Does your state set parameters regarding the qualifications of those who observe/rate teachers? If yes, please explain.
While we have examined information available online, do you have any additional documentation about this that I can access on the internet, or could you send me a copy? I’d like to see any forms that show how instructional practice is documented. For example, is there a standard observation form or a description of what should go into a portfolio? If found in extant documents, confirm that the documents are up-to-date.
Developing Measures of Effectiveness
Questions for states that have any measure of teacher effectiveness Ask Question 19 and Question 20 if the state has either a measure that uses student achievement, a measure that uses demonstration of instructional practice, or both. |
I’d like to learn about why you developed these [EFFECTIVENESS MEASURES / GUIDELINES FOR EFFECTIVENESS MEASURES], and what your experience with them has been.
Why did you develop these [MEASURES / GUIDELINES] and what year did you begin using them? Probe for separate responses, as applicable, regarding (1) measures that use student achievement and (2) measures that use instructional practice.
Because of dissatisfaction with other measures
Other measures did not distinguish adequately among teachers
Because of grant requirements/to obtain a grant
Because of federal requirements/policies
Other reasons
Are there any state initiatives underway to refine the [MEASURES / GUIDELINES], or implement them in a better way? Probe for separate responses, as applicable, regarding (1) measures that use student achievement and (2) measures that use instructional practice.
Questions for states that DO NOT have measures that use student achievement data and/or measures that use demonstration of instructional practice |
Are there any state initiatives underway to develop a measure of teacher effectiveness that uses student achievement or a measure of effectiveness that uses demonstration of instructional practice? Are there any initiatives underway to develop guidelines for districts to develop such measures?
If initiatives are underway, ask a-b and probe for separate responses, as applicable, regarding (1) measures that use student achievement and (2) measures that use instructional practice.
Please describe the initiatives.
Why is the state undertaking initiatives in this area?
Because of dissatisfaction with existing measure
Existing measures do not distinguish adequately among teachers
Because of grant requirements/to obtain a grant
Because of federal requirements/policies
Other reasons
If initiatives are not underway, ask a and probe for separate responses, as applicable, regarding (1) measures that use student achievement and (2) measures that use instructional practice.
Why is the state not undertaking actions in this area?
Not considering because satisfied with existing indicators of teacher quality
Not considering because of state law
Not considering because of cost
Not considering because of insufficient staff expertise
Not considering because of insufficient staff time
Not considering because of likely resistance
Not considering because of likely changes to federal policy
Other challenges to developing such measures
Using Teacher Quality Data to Determine Differences Across Schools
We know that most states use their highly qualified teacher (HQT) measure(s) to look at the distribution of teachers across schools, and we know what measures of teacher quality you collect. Do you use any other teacher quality measures to look for differences in the teachers at different types of schools, such as high-poverty schools or schools with high minority populations? Ask a-b.
Do you use any teacher qualification measures to look for differences in the teachers at different types of schools?
Do you use any teacher effectiveness measures to look for differences in the teachers at different types of schools?
We talked about looking at teachers from different types of schools across your state. Do you use measures of teacher qualifications or teacher effectiveness to look for differences in teachers between different types of districts in your state?
On the basis of the measures you collect: Ask a-e.
How would you quantify the differences in teacher quality between high and low poverty schools? Probe to discover differences in each measure identified in question 24.
How would you quantify the differences in teacher quality between high and low minority schools? Probe to discover differences in each measure identified in question 24.
How would you describe differences in teacher quality between urban, suburban, and rural schools?
How much of a difference must there be for the state or districts to consider there to be a problem in the distribution of high-quality teachers across schools?
For states that use teacher effectiveness measures to look across schools: Compared to using teacher qualifications, such as highly qualified teacher status, does using teacher effectiveness measures change your understanding of whether disadvantaged students have high-quality teachers? Whether such teachers are distributed equitably across schools?
What factors have facilitated or hindered using the measures you mentioned (i.e., teacher qualification or effectiveness measures in addition to HQT) to examine the distribution of teachers across different types of schools or districts?
What have been the challenges? Multiple challenges may apply. Probe until interviewee cannot recall other challenges.
Challenge collecting the data
Challenge establishing the validity/quality of the measure
Challenge incorporating the measures into a data system that allows disaggregation by school type (e.g., low-poverty and high-poverty schools).
Challenge determining criteria for what is “equitable”
Challenge reporting the data
Resistance to using the data
Challenges specific to rural districts
Other challenges
What factors have facilitated your use of your measures to examine teacher distribution? What have been your successes?
Strategies to Ensure High-quality Teachers for Disadvantaged Students
Now I’d like to talk about your state’s strategies and the actions that the state is taking that are intended to ensure high-quality teachers for schools with high proportions of disadvantaged students (i.e., high-minority and/or high-poverty schools). I’d like to talk about actions that are targeted at these schools. For example, if you have a state program that provides additional salary to teachers in high-poverty schools, I’d like to talk about that kind of program now, because it is targeted at these schools.
OK, let’s get started,
Please list the strategies or actions your state is undertaking to ensure that schools with high proportions of disadvantaged students have high-quality teachers. Use the sub-questions in the following box, Questions About Actions, for each strategy mentioned.
If strategies in the following categories are not mentioned, interviewer will use the list below to probe for strategies in these areas of interest. Also probe on policies identified in extant sources as current or intended.
Recruitment and retention
Monetary incentives to attract or retain teachers to/in schools with high proportions of disadvantaged students
Non-monetary incentives to attract or retain teachers to/in schools with high proportions of disadvantaged students
Other recruitment efforts to attract teachers to schools with high proportions of disadvantaged students.
School conditions
State policies to promote improved working conditions (e.g., school safety, school facilities, professional learning community)
State policies to promote high-quality school leadership
Preparation and professional development
State policies to promote specialized teacher preparation programs to prepare teachers for schools with high proportions of disadvantaged students (e.g., residency program, specialized student teaching arrangements, specialized alternative certification programs)
State policies to promote in-service professional development that focuses on schools with high proportions of disadvantaged students
State policies to promote induction and mentoring in schools with high proportions of disadvantaged students
Questions About Actions For each targeted action identified in 28, ask the lettered questions below. If not targeted, allow respondent to describe the action and ask only sub-questions a-d.
|
What factors have facilitated or hindered implementation of any the strategies you’ve described?
What have been the challenges with implementing the strategies you’ve described? Interviewer will probe for overarching challenges as well as challenges related to specific actions discussed above. If rural challenges are not mentioned, interviewer will ask i.
Lack of existing policies and practices
Lack of managerial capacity
Lack of fiscal capacity and public support
Lack of stakeholder support
Lack of external support
Other
Have urban, suburban, and rural districts faced different challenges in implementing these strategies? Have there been any challenges in implementing these strategies that are unique to your rural districts?
What factors have facilitated implementation of any of the strategies you’ve described?
Existing policies and practices
Managerial capacity
Fiscal capacity and public support
Stakeholder support
External support
Other
Role of Federal Programs and Policies
I’d like to understand the role of federal policies and programs in your activities with respect to ensuring that schools with high proportions of disadvantaged students have high-quality teachers. How would you characterize the role of federal programs and policies in your decisions to pursue the activities we’ve talked about today, including your efforts to develop measures of teacher effectiveness, determine the distribution of teachers, and ensure that schools with large numbers of disadvantaged students have high-quality teachers?
Refer to the following programs (Interviewer will use online sources to discover whether the state has applied for Race to the Top or received funding for any of the programs in c-g):
Requirements for equity plans & assurances (Elementary and Secondary Education Act (ESEA), State Fiscal Stabilization Fund (SFSF))
If applied, Race to the Top (RttT)
If received, Teacher Incentive Fund (TIF)
If participating, Investing in Innovation Fund (i3)
If received, Statewide Longitudinal Data Systems grant program (SLDS)
If received, Teacher Quality Enhancement grant program (TQE)
If received, Title I School Improvement Grant (SIG)
Record an answer for each program that applies to the state,
Reinforced existing actions or decisions
Spurred new actions
Stimulated new thinking but no new actions
Provided financial support but SEA would have undertaken effort without support.
Are there other funding sources that are playing a role in your efforts to develop measures of teacher effectiveness, determine the distribution of teachers, or ensure that schools with large numbers of low income and/or minority students have high-quality teachers?
Grants from national foundations
Grants from state or local foundations
Now I’d like to focus on Title I equity plans for a few minutes.
Does your equity plan enter into the activities we’ve talked about today? If so how? If not, why not?
How often have you updated your equity plan, and what is the reason for the updates? Probe to determine how many times the state has updated its plan, approximate dates, and reasons.
How does the state make sure that its plan is implemented?
Do you require your LEAs to provide assurances about equitable distribution of high-quality teachers? If yes, How do you make sure that LEAs are carrying out those assurances? Do you examine state data on the distribution of teachers within each district?
Do you require your LEAs to write similar plans—that is, “local equity plans”? If so, ask subitems a-f.
Do all LEAs have to write equity plans, or just some districts? If some, how do you determine which districts must write a plan?
What does the state tell LEAs about the required content of local plans?
What does the state tell them about how often to update the plans?
Are there differences in how you approach the planning process for your rural LEAs, compared to urban LEAs?
How do you track LEAs’ progress in enacting their plans?
How are the LEAs accountable for enacting their plans?
Wrap Up
That’s all the questions I have. Thank you very much for participating in the study.
Is there anything else you’d like to tell me?
Thank you so much for your time!
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Andrew Abrams |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |