Download:
pdf |
pdfImpact Study of Feedback for
Teachers Based on Classroom
Videos
Part A: Supporting Statement for
Paperwork Reduction Act Submission
May 2, 2017
Submitted to:
U.S. Department of Education
National Center for Education Evaluation
Institute of Education Sciences
550 12th Street, S.W.
Washington, DC 20202
Project Officer: Elizabeth Warner
Contract Number: ED-IES-16-C-0021
Submitted by:
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005
Project Director: Susanne James-Burdumy
Reference Number: 50330
This page has been left blank for double-sided copying.
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
CONTENTS
PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION ................. 1
Justification ................................................................................................................................ 2
A1.
Circumstances necessitating the collection of information ......................................... 2
A2.
Purpose and use of data ............................................................................................. 9
A3.
Use of technology to reduce burden ......................................................................... 10
A4.
Efforts to avoid duplication of effort ........................................................................... 11
A5.
Methods of minimizing burden on small entities ....................................................... 11
A6.
Consequences of not collecting data ........................................................................ 11
A7.
Special circumstances .............................................................................................. 12
A8.
Federal register announcement and consultation ..................................................... 12
A9.
Payments or gifts....................................................................................................... 13
A10. Assurances of confidentiality ..................................................................................... 14
A11. Justification for sensitive questions ........................................................................... 15
A12. Estimates of hours burden ........................................................................................ 15
A13. Estimate of cost burden to respondents ................................................................... 17
A14. Annualized cost to the federal government ............................................................... 17
A15. Reasons for program changes or adjustments ......................................................... 17
A16. Plans for tabulation and publication of results .......................................................... 17
A17. Approval not to display the expiration date for OMB approval .................................. 19
A18. Exception to the certification statement .................................................................... 19
REFERENCES ............................................................................................................................................ 20
APPENDIX A:
TEACHER PARTICIPATION FORMS
APPENDIX B:
TEACHER SURVEY WITH INVITATION LETTER AND NONRESPONSE
MATERIALS
APPENDIX C:
ADMINISTRATIVE RECORDS DATA REQUEST
APPENDIX D:
STUDENT ENUMERATION AND DATA REQUEST FORM
APPENDIX E:
ACTIVE AND PASSIVE PARENT PERMISSION FORMS
APPENDIX F:
DISTRICT RECRUITMENT LETTER
APPENDIX G:
TEACHER RECRUITMENT LETTERS
APPENDIX H:
CONFIDENTIALITY PLEDGE
iii
This page has been left blank for double-sided copying.
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
TABLES
A.1
Data collection .................................................................................................................................. 5
A.2
Research questions and data sources............................................................................................. 9
A.3
Schedule of major study activities .................................................................................................. 10
A.4
Source, mode, length and timing ................................................................................................... 11
A.5
Estimated response time for data collection .................................................................................. 16
v
This page has been left blank for double-sided copying.
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT
SUBMISSION
This package requests clearance for data collection activities to support a rigorous
evaluation of video-based classroom observations and feedback for novice and early career
teachers. This evaluation is being conducted by the Institute of Education Sciences (IES),
National Center for Education Evaluation, U.S. Department of Education (ED). It is being
implemented by Mathematica Policy Research, Inc. (Mathematica) and its partners: Clowder
Consulting, LLC; Decision Information Resources, Inc. (DIR); Educopia; IRIS Connect;
Pemberton Research; WestEd; and Teachstone.
The goal of this evaluation is to examine the impact of video-based observations and
feedback on the classroom practices and student achievement of novice teachers (in their first
year of teaching) and early career teachers (in their second through fourth years of teaching).
This study provides an important test of whether intensive, individualized support for teachers
improves their instructional practices and ultimately student achievement. By focusing on novice
teachers, the study can inform both teacher induction policies and potentially teacher preparation
programs. Examining the impact of this intervention on novice and early career teachers can also
inform the effectiveness of providing individualized feedback as a model for teacher professional
development programs.
The evaluation will include implementation and impact analyses. The implementation
analysis will use information on teachers’ participation, the amount and type of feedback
received, and teaching practices covered to document program implementation.1 We will also
use responses to the teacher survey to describe teachers’ professional support and development
experiences. The impact analysis will draw on data from teacher surveys, assessments of
teachers’ pedagogical knowledge and their attitudes towards teaching, video observations of their
classroom practices,2 and district administrative records.
This package provides a detailed discussion of the procedures for these data collection
activities and copies of the forms and instruments developed by the study team.
1
We are not requesting OMB approval for the collection of this information because they will be collected by the
study team and will not impose any burden on teachers or district staff.
2
Ibid
1
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Justification
A1. Circumstances necessitating the collection of information
a.
Policy context and statement of need
The specific legislation authorizing this data collection includes Title II sections 2001-2002
and Title VIII section 8042 of the Elementary and Secondary Education Act of 1965 (ESEA),
which permits ESEA program funds to be used to evaluate activities that are authorized under
this act. The Every Student Succeeds Act (ESSA), which reauthorized ESEA, emphasizes the
importance of teacher quality in improving student achievement. In particular, Title II, Part A of
ESEA—the Improving Teacher Quality State Grants program—provides funds to states to
prepare, train, and recruit high-quality teachers. One allowable use of Title II, Part A funds is
carrying out activities that “provide support to teachers or principals, including support for
teachers and principals new to their profession.”
There is little evidence to guide districts and policymakers on the most promising
approaches to supporting new teachers. Research has consistently found that novice teachers are
less effective than experienced teachers at raising student achievement (Boyd et al. 2006;
Hanushek et al. 2005; Kane et al. 2008). However, rigorous, large-scale evaluations suggest that
content-focused professional development for teachers at various experience levels does not
meaningfully influence teacher effectiveness (Garet et al. 2008, 2011, 2016). In addition, there
are inconsistent findings regarding the efficacy of programs that provide comprehensive
induction to new teachers (Cohen and Fuller 2006; Glazerman et al. 2010; Rockoff 2008).
Providing individualized feedback to teachers may be an effective strategy for supporting
novice and early career teachers. Recent small-scale studies found that providing individualized
feedback to teachers based on classroom observations improved student achievement (Taylor
and Tyler 2012; Steinberg and Sartain 2015; Allen et al. 2011, 2015; Campbell and Malkus
2011). In addition, surveys show that educators prefer this type of support over more traditional
forms of professional development (Bill & Melinda Gates Foundation 2014; Parise et al. 2015).
Providing individualized feedback based on video recordings rather than live classroom
observations may be particularly promising, given the labor- and resource-intensive nature of the
feedback process. Using video recordings allows observers to conduct observations and provide
feedback at times and locations that are convenient to them, and allows teachers to reflect on
their practices in new ways (Greenberg et al. 2015; Roth et al. 2011; Sherin and van Es 2009a;
Sherin et al. 2009b).
Despite this promise, policymakers and educators lack evidence on the effectiveness of
video-based instructional feedback. This study will address that gap by providing rigorous
evidence on the effectiveness of feedback and coaching based on videos of classroom instruction
for novice and early career teachers.
b. Treatment
This study will measure the impact of an intervention that provides intensive feedback to
teachers based on video recorded observations of their instruction. The study team will video
record the classroom instruction of teachers assigned to receive the intervention multiple times
during the year, and send the videos to the intervention provider. Coaches from the intervention
2
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
provider will review the videos using an observation instrument to measure these teachers’
performance on a targeted set of teaching practices. Coaches will use a consistent, systematic
approach to develop written feedback for teachers that includes individualized actionable steps
that focus on improving the teachers’ performance on one or more of the targeted teaching
practices. Based on videos of their teaching in their classroom, teachers will participate in oneon-one sessions with their coach to review the teacher’s performance and provide feedback.
Teachers will also have access to the videos of their classroom instruction.
We will evaluate two versions of the intervention – the full intervention and a less intensive
version of the intervention. The full and less intensive versions of the interventions will differ in
the number of feedback and coaching sessions that the teachers will receive from the coach.
Teachers assigned to the less intensive version of the intervention will participate in 5 feedback
cycles that include one-on-one sessions with a coach to review the teacher’s performance and
provide feedback. Teachers assigned to the full version of the intervention will participate in an
additional 5 feedback cycles and one-on-one sessions with a coach to review the teacher’s
performance and provide feedback – for a total of 10 sessions over the course of the school year.
This study is evaluating two versions of the intervention in order to provide districts and
preparation programs with information on how to efficiently allocate their resources, if they
choose to implement this type of intervention. For example, if the study finds positive impacts
for the full intervention, but not the less intensive intervention, it suggests that the extra cost of
hiring or using coaches to provide more feedback and coaching sessions with teachers is needed
to achieve an impact. On the other hand, if the less intensive intervention is as effective as the
full intervention in improving teacher practices and student achievement, districts and
preparation programs could implement the less intensive intervention, which requires fewer
resources, and achieve the same impact as the full intervention.
c.
Study design and research questions
This study will use a random assignment design to estimate the impact of video-based
observations and individualized feedback on teachers’ classroom practices and student
achievement. The study will recruit 200 novice teachers from 12 districts for the first year of the
study.3 These teachers will be randomly assigned to one of two groups – the full intervention and
a control group which receives none of the intervention supports. In the following year, we will
randomly assign approximately 300 early career teachers into one of three groups – to the full
intervention, the less intensive version of the intervention (see descriptions in the previous
section) or the control group. The early career teacher sample will exclude novice teachers who
participated in the study in the prior year. This study design will allow us to (1) estimate the
impact of the full intervention (compared to the control group) for novice teachers, (2) estimate
the impact of two versions of the intervention (compared to the control group), and (3) examine
the relative effectiveness of the two versions of the intervention for early career teachers. The
study will also support and monitor program implementation to ensure high quality
implementation. It will include implementation analyses which will provide context for
3
Appendices F and G contain copies of letters that will be used to inform and recruit districts and teachers.
3
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
interpreting impact results and shedding light on the mechanisms through which the
interventions may affect teacher and student outcomes.
The primary research questions for this study are:
1.
What is the impact on teaching practices and student achievement of providing novice
teachers with 10 feedback and coaching sessions based on video recordings of their
classrooms (the full intervention)?
2.
What is the impact on teaching practices and student achievement of providing teachers
early in their career (years 2–4) with 10 feedback and coaching sessions based on video
recordings of their classrooms (the full intervention)?
3.
What is the impact on teaching practices and student achievement of providing early career
teachers with 5 feedback and coaching sessions based on video recordings of their
classrooms (the less intensive intervention)?
The evaluation will also address the following secondary research questions:
4.
Were the feedback and coaching interventions implemented with fidelity? How did the
experiences of teachers in the treatment and control groups compare?
5. What challenges did teachers encounter?
6. What is the impact of the interventions on key intermediate outcomes, including teachers’
knowledge of teaching practices and their skill at implementing the practices?
7. What is the impact of ten versus five feedback and coaching intervention sessions for early
career teachers?
8. How do impacts vary for teachers with different preparation experiences and background
characteristics?
9. What baseline teacher practices appear most important for effective teaching? Is teachers’
knowledge of teaching practices predictive of their future effectiveness? What teacher
beliefs and attitudes are most important for effective teaching?
10. Could screening teachers for certification or hiring based on instructional practices,
knowledge, attitudes, or beliefs help improve teacher effectiveness?
11. Do intervention effects on teachers’ knowledge of practices or their skill in implementing
them appear to explain impacts on student achievement?
12. How cost effective are the interventions?
d.
Data collection
This study includes multiple data collection efforts. Data for the impact analyses will be
collected from districts, schools, and teachers. The study team will also collect data that will be
used to describe implementation fidelity. Since we are video recording classrooms, we will also
collect permission from classroom students’ parents to include their child in video recordings.
All of these data are described below and summarized in Table A.1.
4
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Table A.1. Data collection
Instrument
Data need
Respondent
Mode
Schedule
(1) Two hour computerbased Praxis Principles
of Learning and
Teaching (PLT): Grades
K–6
(1) Baseline
(summer 2017) and
follow-up (spring
2018)
1) Teachers’ pedagogical
knowledge and ability to apply
that knowledge (Praxis)
(2) Teachers’ attitudes and
beliefs related to teaching in
high-poverty schools (Haberman)
Novice teachers
only
Teacher
participation
forms
Teachers’ preparation and
teaching experience, edTPA
score, beliefs of their
preparedness to manage
classroom behavior and teach
math and English/language arts
Teachers
15-minute paper
enrollment and eligibility
form
Summer 2017 and
summer 2018
Teacher survey
Receipt of professional
development and feedback
(amount, quality, usefulness),
teacher supports, experience
with program (characteristics,
usefulness, challenges) and,
background and demographic
information
Teachers
30-minute web-based
survey
Spring 2018 and
spring 2019
District
administrative
records
Student test scores and
characteristics (such as gender,
age, and English language
learner status), teacher-student
links
District
Electronic records of
scores on standardized
math and reading tests,
data on student
characteristics
Fall 2018 and fall
2019
Student
enumeration form
List of students in study teachers’
classes used to prepare parent
permission packets
School
Paper or electronic list of
students enrolled in
study teachers’
classrooms
Fall 2017 and fall
2018 (updated with
new arrivals during
2017-2018 and
2018-2019 school
years by field staff)
Parent permission
forms
Active and passive permission
forms (depending on district
requirements) for parent or
guardian to document consent
for student to be included in
classroom instruction video
recordings
Parent or
guardian
Paper permission form
indicating consent or
non-consent for students
to be included in video
recordings
Fall 2017 and fall
2018 (distributed
to new arrivals
during 2017-2018
and 2018-2019
school years by
teachers and field
staff)
Videos of
teachers’
classroom
instruction
Classroom Environment and
Instruction domains; scores on
items in those domains most
relevant for teaching practices
targeted by the intervention
Study team
records treatment
and control group
classrooms and
codes recordings
using two rubrics
Computer-based
observation rubrics used
to score classroom
videos
Score three
baseline videos
from fall and three
follow-up videos
from spring (for
2017–2018 and
2018–2019 school
years)
Coaching logs
Teaching practices covered,
coaching activities, feedback
session length, goals, and action
plans
Coaches
Electronic form
completed by coaches
after each feedback
session
Throughout the
2017-2018 and
2018-2019 school
years
Feedback session
observations
Teaching practices covered;
extent to which coach used
videos, provided actionable
feedback, set goals, and
developed action plans
Study team
Closed-ended electronic
protocol completed for a
randomly selected set of
feedback sessions
Throughout the
2017-2018 and
2018-2019 school
years
Teacher
knowledge and
attitudes/beliefs
assessments
(2) 30-minute computerbased Haberman Star
Teacher Pre-Screener
5
(2) Baseline only
(summer 2017)
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Assessment of teachers’ knowledge of teacher practices. All novice teachers in the study
(treatment and control groups) will take the Praxis Principles of Learning and Teaching (PLT):
Grades K-6 assessment at baseline during the training session in summer 2017 and at follow-up
in spring 2018. We will ask teachers who have already taken the Praxis PLT as part of their
teacher licensure, to give us permission to obtain their scores from the Educational Testing
System (ETS). Teachers whose scores may be obtained from ETS will not be asked to retake the
assessment in the summer of 2017. This two-hour computer-based exam measures teachers’
knowledge of teaching practices and their ability to apply that knowledge. The test is a valid,
reliable, policy-relevant measure used for teacher licensure in 17 states, with a reasonable
administration burden. (Teacher assessments are not included in the appendices because they are
copyrighted.) We will use the Praxis PLT to describe teachers’ knowledge of teaching practices
at baseline and to measure the impact of the feedback and coaching interventions on their
knowledge of practices. The Praxis PLT scores will also be used in an analysis that measures
whether changes in teachers’ knowledge of practices explains any impacts on observed teacher
practices and student achievement. Data obtained from the Praxis PLT will be used to address
research question 6, 9-10, and 11.
Assessment of teachers’ attitudes and beliefs. All novice teachers in the study will also
take the Haberman Star Teacher Pre-Screener during the training session in summer 2017. This
is a 30-minute computer-based questionnaire that assesses the attitudes and beliefs needed to
effectively teach low-income students in urban areas. The Haberman screener is used to screen
teachers for hiring in more than 200 districts. We will use the Haberman Star Teacher PreScreener to describe teacher attitudes and beliefs and to measure whether these attitudes and
beliefs are related to variation in the impact of the interventions. Data obtained from the
Haberman Star Teacher Pre-Screener will be used to address research questions 9-10.
Teacher participation forms. All teachers participating in the study will complete and sign
a teacher participation form prior to random assignment. By signing this form, the teacher agrees
to participate in the study, cooperate with data collection activities, and to participate in
intervention activities if assigned to the feedback and coaching group (Appendix A). There is
one form for novice teachers and one form for early career teachers. Both forms gather key
information on teaching assignment for the upcoming year, teacher preparation (route to
teaching), years of teaching, feedback received on their teaching, and level of confidence to
teach math and English/language arts and to manage student behavior. This form also asks
teachers who have taken the edTPA performance assessment to provide their score, and (for
novice teachers who have already taken the Praxis PLT assessment) to give the study team
permission to obtain their Praxis PLT score from ETS. Teacher baseline characteristics can be
used to determine eligibility and to create similar pairs or groups of teachers to implement the
random assignment plan.
6
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Teacher survey. The teacher survey will provide information about the amount, quality and
usefulness of the professional development received by teachers (Appendix B). The survey will
also provide insights on other teacher supports, teacher preparation, and background
characteristics. The survey will be conducted in spring 2018 and spring 2019. The survey will
take approximately 30 minutes to complete. The teacher survey will be used to describe teachers’
background characteristics, teacher preparation experiences, and their experiences with feedback
and coaching during the school year. These data will allow us to examine whether the impact of
the interventions differed depending on teachers’ preparation experiences (for example, the
amount of time they received feedback on their teaching as a teacher candidate) and their
background characteristics. We can also assess how much feedback and coaching teachers
received and any implementation challenges they faced. In addition, we will use the survey to
assess the interventions’ impacts on intermediate outcomes such as: teachers’ feelings of
preparedness for teaching and the extent to which teachers receive support for teaching. Data
obtained from the teacher survey will be used to answer research questions 4-6 and 8-10.
District administrative records. We will collect student test scores, attendance,
demographic information such as gender, age, and English Language Learner status (Appendix
C). We will also collect data to confirm student-teacher links as well as what subject(s) and
grade(s) the teachers are teaching. This data will be collected in fall 2018 and fall 2019. We will
use state test score data in reading and math to estimate the impact of the feedback and coaching
interventions on student achievement, the key outcome of interest. We will estimate impacts on
the achievement of the students taught by study teachers at the end of each intervention year
(2017-2018 and 2018-2019) as well as one year after the intervention is complete for the novice
teacher sample. Information on students’ demographic and socioeconomic characteristics (e.g.,
gender, age, free and reduced-price lunch status) will be used to describe the students in the
study and to develop more precise impact estimates. We will use data linking teachers to students
to identify which students were assigned to each treatment and control teacher. These data will
also be used to measure impacts for subgroups to measure the relationship between teacher
effectiveness and teachers’ knowledge of practices and their skill in implementing practices.
Data obtained from district administrative records will be used to address research questions 1-3
and 7-11.
Student enumeration form. A list of students in study teachers’ classrooms will be
obtained at the start of each academic year in fall 2017 and fall 2018 to develop parent
permission packets (Appendix D). The list will be updated during the course of the school year
and permission packets will be distributed by field staff to new arrivers during the 2017-2018
and 2018-2019 school years.
Parent permission forms. We will distribute paper permission forms for parents or
guardians. In districts that require active consent, we will collect permission forms from parents
or guardians to document permission for students to be included in classroom instruction video
recordings (Appendix E). In districts that permit passive consent, we will collect forms from
parents who indicate that they do not give permission for their child to be included in classroom
instruction video recordings. The permission forms will be collected in fall 2017 and fall 2018
and permission forms will be distributed to new arrivals by teachers and field staff during the
2017-2018 and 2018-2019 school years.
7
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Classroom observations scores. We will video record treatment and control teachers’
classrooms three times at the beginning of the school year (to obtain a baseline measure of
teacher classroom practices) and three times at the end of the school year (for a year-end
outcome measure of teacher classroom practices), for both years of the study (the 2017-2018 and
2018-2019 school years). To assess the quality of teachers’ instructional practices, we will score
each video using two formal observation rubrics: (1) the observation rubric used by the coaching
provider, and (2) a valid classroom observation measure. The video recording and scoring of the
videos will be done by study staff. There is no respondent burden associated with this data
collection activity.
Baseline and year-end classroom observation scores will be used to measure the impact of
the interventions. Baseline measures will provide a measure of teachers’ practices before
participating in the program, and controlling for baseline practices in our analytic models will
increase the precision of the impact estimates. In addition to controlling for teachers’ baseline
observation scores in the impact models, we will use these scores for two additional analyses.
First, we will examine whether the quality of teachers’ baseline teaching practices is correlated
with variation in program impacts. Results from these analyses will be useful for understanding
whether the impact of the interventions may depend on the teacher’s initial level of effectiveness.
Second, we will explore whether changes in teacher practices mediate impacts on student test
scores. These results will provide important evidence about the potential mechanisms through
which the interventions may influence student achievement. Data obtained from classroom
observations will be used to address research questions 6, 8-11.
Coaching logs. We will use information from web-based coaching logs to document key
aspects of the feedback sessions that coaches hold with teachers. For example, the logs will
include information discussed during the feedback session on (1) practices that the teacher needs
to improve, (2) video clips from the teacher's classroom (provided to the teacher in advance of
the feedback session), and (3) steps outlined in the action plan. We will assess if the sessions
were implemented as intended based on the teaching practices covered, the types of coaching
activities implemented, and the length of the feedback session. In addition, these data will be
used to examine whether implementation features are associated with impacts on teacher and
student outcomes. Data from coaching logs will be used to address research question 4. The
study coaches will complete the study team developed logs. There is no respondent burden
associated with this data collection activity.
Feedback session observations. We will collect data from a randomly selected set of
feedback sessions between coaches and teachers. We will collect information on teaching
practices covered, the extent to which coaches used videos, the type of feedback that was
provided, the goals that were set, and the action plan developed. The data will be collected by the
study team using a closed-ended electronic protocol. We will use the observations of feedback
sessions to describe implementation of the feedback sessions and assess implementation fidelity.
For example, we will describe whether coaches provided feedback focused on the targeted
practices, how long the feedback session lasted, and the format of the feedback session. Data
from feedback session observations will be used to address research question 4. The study team
will observe the feedback sessions and collect the data. There is no respondent burden associated
with this data collection activity.
8
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
A2. Purpose and use of data
Data for this evaluation will be collected and analyzed by Mathematica and its partners. This
work will be completed under contract number ED-IES-16-C-0021. The data will be used to
address the study’s research questions, as shown in Table A.2.
Table A.2. Research questions and data sources
Research questions (RQs)
Data sources
Experimental impact analyses
Primary outcomes
What is the impact on teaching practices and student achievement of
providing novice teachers with 10 feedback and coaching sessions
based on video recordings of their classrooms (the full intervention)?
(RQ 1)
What is the impact on teaching practices and student achievement of
providing teachers early in their career (years 2–4) with 10
feedback and coaching sessions based on video recordings of their
classrooms (the full intervention) ?(RQ 2)
What is the impact on teaching practices and student achievement of
providing early career teachers with 5 feedback and coaching
sessions based on video recordings of their classrooms (the less
intensive intervention)? (RQ 3)
Intermediate outcomes
What is the impact on intermediate outcomes, including teachers’
knowledge of teaching practices, their skill at implementing the
practices, and their feelings of preparedness for teaching? (RQ 6)
District administrative records
District administrative records
District administrative records
Classroom observations
Praxis PLT
Teacher survey
Implementation analyses
Were the interventions implemented with fidelity? (RQ 4)
Extant coaching provider records
What challenges did teachers encounter? (RQ 5)
Teacher survey
Other analyses
Subgroup analysis
What is the impact on student achievement of 10 versus 5 sessions of
the intervention (RQ 7)?
How do impacts vary for teachers with different preparation
experiences and background characteristics? (RQ 8)
Other quantitative analysis
What baseline teacher practices are most important for effective
teaching and are teachers’ knowledge of teaching practices and
attitudes and beliefs predictive of their future effectiveness (RQ 9)?
Could screening teachers based on these characteristics increase
average teacher effectiveness?(RQ 10)
Do intervention effects on specific teacher practices or competencies
appear to explain impacts on student achievement? (RQ 11)
How cost effective are the interventions? (RQ 12)
9
District administrative records
District administrative records
Teacher survey
Classroom observations
Teacher participation form
District administrative records
Classroom observations
Teacher survey
Praxis PLT and Haberman
Teacher participation form
District administrative records
Classroom observations
Praxis PLT
Study team records
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
The evaluation will be completed in six years. Table A.3 shows the schedule of data
collection activities and the overall evaluation timeline.
Collect teacher participation
forms
X
X
Collect teacher summer
orientation attendance
X
X
Conduct teacher assessments
X
Spring 2022
Summer 2021
Summer 2020
Summer 2019
Spring 2019
Fall 2018
Summer 2018
Spring 2018
Fall 2017
Activity
Summer 2017
Table A.3. Schedule of major study activities
X
Obtain student lists
X
Obtain parent permission
X
Collect classroom video
recordings
X
Conduct teacher survey
X
X
X
X
X
Collect administrative records
data from districts
X
X
Prepare first report
X
X
X
Prepare second report
X
Prepare third report
X
A3. Use of technology to reduce burden
The data collection plan is designed to obtain information in an efficient way that minimizes
respondent burden. When feasible, we will gather information from existing data sources, using
the most efficient methods available. Table A.4 provides information on the source, mode,
length, and timing for each data collection activity.
We will ask teachers to complete a web-based survey. The web-based surveys will enable
respondents to complete the data collection instrument at a location and time of their choice, and
its built-in editing checks and programmed skips will reduce the level of response errors. We will
also offer teachers the choice to respond to the survey by phone or in-person if they prefer.
We will ask districts to provide electronic copies of student records. While we will specify
the required data elements, we will accept any format in which the data are provided, to reduce
burden for the district. To help ensure study participants’ confidentiality, we will store data at the
Institute of Education Sciences (IES) Data Center, a secure data storage facility.
10
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Table A.4. Source, mode, length and timing
Data source
Mode, length, and timing
Respondent group
Teacher assessments
Administered in summer 2017: (1) Two hour
computer-based Praxis PLT teacher
assessment, and (2) 30-minute computerbased Haberman Star teacher pre-screener.
Treatment and control Novice
Teachers
Teacher Survey
30-minute web-based survey, with telephone
and hard-copy options and in-person followup, administered in spring 2018 and spring
2019
Treatment and control Novice
Teachers
Student records
Electronic student records data for prior
school years, requested from districts in fall
2018 and fall 2019.
District staff
A4. Efforts to avoid duplication of effort
No similar evaluations are being conducted and there is no equivalent source for the
information to be collected. Moreover, the data collection plan reflects careful attention to the
potential sources of information for this study, particularly to the reliability of the information
and the efficiency in gathering it. The data collection plan avoids unnecessary collection of
information from multiple sources. For example, student achievement will be measured using
scores from state- or district-administered student assessments, instead of administering an
assessment.
Information obtained from the classroom observation videos, teacher knowledge assessment,
teacher participation forms, teacher survey, and parent permission forms is not available
elsewhere. Teachers who have already taken the Praxis PLT and for whom we have obtained
scores from ETS will not need to retake the assessment.
A5. Methods of minimizing burden on small entities
No small businesses or entities will be involved as respondents.
A6. Consequences of not collecting data
The data collection plan described in this submission is necessary for ED to conduct a
rigorous evaluation of individualized feedback and coaching for teachers. Collecting these data
will allow us to estimate the impact of the interventions on teachers’ practices and student
achievement. Additionally, the data will be used to understand how the provider implemented
the coaching and feedback and how impacts vary based on the implementation of the
interventions and teacher characteristics.
The consequences of not collecting specific data are outlined below:
Without the district administrative records, we would have to administer student
assessments instead of using their state math and reading test scores. Without information on
student characteristics, we would not be able to fully describe the study sample or verify the
effectiveness of teacher random assignment. Teacher-student links are necessary to estimate
the impact of the intervention on student achievement.
11
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Without the videos of teachers’ classroom instruction that will be used to create classroom
observation rubric scores, we would not be able to measure the impact of feedback and
coaching on teachers’ practices. In addition, without these observation scores, we will not
know if impacts on teachers’ practices explain any impacts on student achievement.
Without the assessment of teachers’ knowledge of teaching practices and the assessment
of teachers’ attitudes and beliefs, we will not be able to determine how teachers’
knowledge, attitudes, and beliefs are related to the effectiveness of the intervention. These
data will also inform whether school districts could use information about teachers’
knowledge, attitudes, and beliefs more effectively in hiring decisions. In addition, the
follow-up test of teachers’ knowledge will provide information about the intervention’s
impact on a key intermediate outcome—teachers’ knowledge of teaching practices.
Without the teacher participation forms and the teacher survey, we would not have the
data needed for the sample design and to describe teachers’ preparation experiences and
background characteristics, implementation of the feedback and coaching during the school
year, other teacher supports, and their feelings of preparedness and efficacy. Without these
data we would not be able to compare the amount of feedback and coaching received by
treatment and control teachers. The survey data will also allow us to assess whether the
intervention is more effective for teachers with certain background characteristics.
A7. Special circumstances
There are no special circumstances associated with this data collection.
A8. Federal register announcement and consultation
a.
Federal register announcement
A 60-day notice to solicit public comments was published in the Federal Register, Volume
82, No. 41, page 12450 on Friday, March 3, 2017.
To date, no public comments have been received.
The 30-day notice will be published to solicit additional public comments.
b.
Consultations outside the agency
In formulating the intervention and evaluation design for the proposal for this evaluation, the
study team sought input from several individuals with expertise in teacher preparation and
support, including Suzanne Wilson, Pam Grossman, and Jim Knight.
Additionally, in collaboration with ED we will form a technical working group (TWG) to
provide input on the study design, data collection instruments, analyses, and reports. This input
will help ensure the study is of the highest quality and that findings are relevant to policymakers,
school districts, and teacher preparation programs.
We will work with ED to select TWG members with expertise in teacher certification and
preparation (including teacher preparation providers), teacher quality, teacher policy, and
evaluation methods.
12
CONTRACT NUMBER: ED-IES-16-C-0021
c.
MATHEMATICA POLICY RESEARCH
Unresolved issues
There are no unresolved issues
A9. Payments or gifts
Incentives have been proposed for teachers participating in the study. The proposed amounts
are within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for
Incentives for NCEE Evaluation Studies,” prepared for OMB. To maximize the success of our
data collection effort we will provide incentives to teachers to offset their time and effort with
completing the data collection activities. Incentives are also proposed because high response
rates are needed to make the study findings reliable. Based on feedback from a study on the
feasibility of and design for this evaluation (conducted under another contract), we learned that
many new teachers are likely to be eager to participate in the evaluation, and districts will be
willing to encourage their new hires to participate. However, we are aware that teachers are the
targets of numerous requests for data on a wide variety of topics from state and district offices,
independent researchers, and the Department of Education. Although some districts will have
solicited buy-in from teachers to participate in the evaluation, our recent experience with
numerous teacher data collection efforts supports our view that obtaining teacher buy-in does not
guarantee teachers will devote the time it takes to complete data collection activities, and
monetary incentives increase the likelihood of their cooperation.
Teacher incentive for collecting parent permission forms. We propose providing teachers
with an incentive for collecting parent permission forms allowing us to record students during
the video observations. Teachers will receive $25 for distributing and collecting parent consent.
Because it will be critical for the study to obtain parental permission for as many students in
study classrooms as possible, we will offer teachers an additional $25 for collecting parent
permission forms for at least 85 percent of their students. This represents a maximum of $50 for
any one teacher (roughly $2 per student form and less than the NCEE $3 per low burden student
report). We expect teachers will have to remind students and call or email parents to obtain 85
percent returns. Our goal is to ensure that we have as many students in the classroom as possible
during the video recordings to accurately evaluate the teacher’s performance during a typical day
of instruction. Field staff will be responsible for collecting the permission forms from the
teachers. We believe that a differential incentive to collect parent permission forms from students
will motivate teachers to collect them.
Teacher respondent payment. We propose offering a $30 incentive to teachers who
complete a web-based survey to acknowledge the 30 minutes required to complete the survey.
The survey will be administered in spring 2018 to novice teachers and spring 2019 to early
career teachers. The survey will collect data on novice and early career study sample teachers’
professional development experiences, feelings of preparedness, pre-service training,
certification, other background characteristics, and receipt of professional development and
feedback (amount, quality, and usefulness).
Teacher assessment payments. Novice teachers will receive $100 for completing the twohour Praxis PLT. We will administer the Praxis PLT two times—during the summer 2017
training session and within four weeks of the end of the 2017-2018 school year. In our evaluation
of teachers from highly selective alternate route programs for IES, we achieved an 84 percent
13
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
response rate on the Praxis PLT using a similar incentive. Teachers will also receive $30 for
completing the 30 minute Haberman Star assessment during the summer 2017 training session.
A10. Assurances of confidentiality
Mathematica and its research partners will conduct all data collection activities for this study
in accordance with relevant regulations and requirements, which are:
The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a)
The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR
Part 99)
The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98)
The Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183
The research team will protect the confidentiality of all data collected for the study and will
use it for research purposes only. The Mathematica project director will ensure that all
individually identifiable information about respondents remains confidential. All data will be
kept in secured locations and identifiers will be destroyed as soon as they are no longer required.
All members of the study team having access to the data will be trained and certified on the
importance of confidentiality and data security. When reporting the results, data will be
presented only in aggregate form, such that individuals, schools, and districts are not identified.
Included in all voluntary requests for data will be the following statement:
“Responses to this data collection will be used only for statistical purposes. The reports
prepared for this study will summarize findings across the sample and will not associate
responses with a specific district, school, or individual. We will not provide information that
identifies you, your school, or your district to anyone outside the study team, except as
required by law. Additionally, no one at your school or in your district will see your
responses. While your participation in this study is voluntary, it is very important that you
complete the questionnaire.”
The following safeguards are routinely employed by Mathematica to carry out
confidentiality assurances, and they will be consistently applied to this study:
All Mathematica employees sign a confidentiality pledge (Appendix H) that emphasizes the
importance of confidentiality and describes employees’ obligations to maintain it.
Personally identifiable information (PII) is maintained on separate forms and files, which are
linked only by sample identification numbers.
Access to hard copy documents is strictly limited. Documents are stored in locked files and
cabinets. Discarded materials are shredded.
Access to computer data files is protected by secure usernames and passwords, which are
only available to specific users.
Sensitive data is encrypted and stored on removable storage devices that are kept physically
secure when not in use.
14
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Mathematica’s standard for maintaining confidentiality includes training staff regarding the
meaning of confidentiality, particularly as it relates to handling requests for information, and
providing assurance to respondents about the protection of their responses. It also includes builtin safeguards concerning status monitoring and receipt control systems. In addition, all data
systems will be collected using the IES Data Center as the host in order to be compliant with all
IT security requirements of IES.
The program is currently preparing a system of records notice (SORN) and a privacy
threshold assessment (PTA) to assess the need for a privacy impact assessment (PIA). A PIA
will be prepared if applicable. The data is to be stored both electronically and in paper copy. It is
to be retrievable by ID and the data will be maintained and disposed of in accordance with the
Department’s Records Disposition requirements. The electronic file will kept in a password
protected server. The paper copy will be kept in a locked file cabinet and all access to data in
both electronic and paper form will be restricted to study staff on a need to know basis. The
security protections for the content will be identified in the SORN.
A11. Justification for sensitive questions
No questions of a sensitive nature will be included in this study.
A12. Estimates of hours burden
Table A.5 provides an estimate of time burden for the data collections, broken down by
instrument and respondent. These estimates are based on our experience collecting
administrative data from districts and obtaining parent permission. The estimates for
administering surveys to teachers and items in the teacher participation forms are based on
pretest findings.
15
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Total (rounded)
Total burden time
(hours)
Total response time
(hours/year) for 3
years
Unit response time
(hours)
Number of
respondents
Expected response
rate (%)
Respondent/ Data request
Districtsa
Student records data (1 time per
round, rounds 1 and 2)
Teacher records data (1 time per
round, rounds 1 and 2)
Schools
Student lists (round 1 – 200 novice
teachers)
Student lists (round 2 – 300 early
career and 200 novice teachers)
Teachersb
Summer 2017 teacher participation
form collected from novice
teachers (round 1)
Summer 2018 teacher participation
form collected (round 2)
Spring 2018 teacher survey
administered to novice teachers
Spring 2019 teacher survey
administered to early career
teachers
Parents
Parent permission form (5,000 for
round 1)
Parent permission form (7,500 for
round 2)
Number of targeted
respondents
Table A.5. Estimated response time for data collection
12
100
12
8.0
64
192
12
100
12
8.0
64
192
200
100
200
1.0
66.7
200
500
100
500
1.0
166.7
500
200
100
200
0.25
16.7
50
300
100
300
0.25
25
75
200
85
170
0.5
28.3
85
300
85
255
0.5
42.5
127.5
5,000
85
4,250
0.25
354.2
1,062.50
7,500
85
6,375
0.25
531.3
1,593.75
14,224c
12,274d
1,359
4,078
a Depending
on the district, administrative records data may be provided by another source, for instance the state.
burden estimates for teachers do not include the time for novice teacher assessments (PLT and Haberman).
c The total number of targeted respondents (14,224) is the sum of targeted responses across data requests from a
total of 13,512 unique respondents including 12 districts, 500 schools, 500 teachers (participation form and survey
requests), and 12,500 parents across the three years of implementation
d The total number of respondents (12,274) is the sum or responses received from 12 districts (student and teacher
data requests), 500 schools, 500 teachers, and 10,625 parents across the three years of implementation. Although
expected response to surveys for teachers is 85 percent (425 teachers), all 500 teachers enrolled in the study will
complete a teacher participation form.
b The
The number of targeted respondents and responses are 14,224 and 12,274, respectively and
the annual number of responses is 4,091. The total burden is estimated at 4,078 or an estimated
average annual burden of 1,359 burden hours calculated across 3 years. Teacher assessments are
not calculated in the burden estimates.
16
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
The total of 4,078 hours includes the following efforts: up to 16 hours, annually for two
years, for each of the 12 districts to assemble administrative records for students and teachers
participating in the evaluation; up to 167 hours annually for class lists provided by schools; 15
minutes for an average of 250 teachers to complete teacher participation forms (with 200 in
round 1 and 300 in round 2); 30 minutes annually for two rounds of the teacher surveys (85
percent of the anticipated samples of 200 teachers in round one and 300 teachers in round 2); and
15 minutes for up to 25 parents per teacher to review parent permission materials.
A13. Estimate of cost burden to respondents
There are no direct or start-up costs to respondents associated with this data collection.
A14. Annualized cost to the federal government
The total cost to the federal government for this study is $18,367,939. The estimated average
annual cost—including recruiting districts, designing and administering all collection
instruments, processing and analyzing the data, and preparing reports—is $3,061,323 (the total
cost divided by the six years of the study).
A15. Reasons for program changes or adjustments
This is a new collection.
A16. Plans for tabulation and publication of results
a.
Analysis plan
The evaluation will estimate the impact of the interventions on student and teacher outcomes
and document program implementation. Additionally, the study includes several supplementary
analyses to understand the relative effectiveness of the interventions based on key intervention
and teacher characteristics. Below, we describe the main impact and implementation analyses.
Impact analyses. We will use regression models to estimate the impact of the interventions
on student outcomes (standardized math and reading test scores) and teacher outcomes (teaching
practices, pedagogical knowledge, and survey responses). Because the study has a randomized
controlled trial design, comparing the outcomes of teachers and their students randomly assigned
to intervention and control groups should yield unbiased estimates of the interventions’ impacts.
To increase the precision of our estimates, however, we will also control for student
characteristics, teacher characteristics, and school characteristics. We will estimate the same
models on several different subgroups of teachers to determine how program impacts vary by
teacher experience, years participating in the program, and baseline characteristics. Results from
the impact analyses will provide evidence about the effectiveness of the full and less intensive
interventions at improving teacher practices and achievement.
17
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Implementation analyses. We will conduct several implementation analyses. First, we will
describe the interventions in terms of their intensity, teaching practices covered, use of videos
and rubric ratings, and structure of the feedback sessions. Second, we will assess implementation
fidelity, focusing on teacher participation, the amount and type of feedback received, and
teaching practices covered. Third, we will measure the contrast in professional development and
support experienced by teachers in the treatment and control groups. Finally, we will analyze
teachers’ perspectives on implementation challenges and on the quality of supports provided.
Understanding the implementation experiences and challenges of districts, schools, and
teachers participating in the intervention will provide important information for districts and
teacher preparation programs considering similar interventions. The implementation analyses
will support replication of the interventions in other districts and teacher preparation programs
and provide necessary context for impact results.
b.
Publication plan
We will prepare three reports presenting the results of these analyses. The first report, with a
projected release date of June 2020, will describe the implementation of the intervention for
novice teachers and report the intervention’s impacts on teachers’ perceptions of their
preparedness for teaching, teaching practices, and student achievement at the end of the school
year in which the intervention is delivered.
The second report, with a projected release date of June 2021, will contain similar analyses
as those described for report one, but it will focus on implementation and impact findings for the
early career teacher sample at the end of the intervention year and on impacts on novice teachers
one year after intervention delivery. It will also include a cost-effectiveness analysis.
Specifically, it will include the following analyses:
impacts of the full intervention (10 feedback and coaching sessions) for a subset of early
career teachers at the end of the intervention year;
impacts of the less intensive intervention (5 feedback and coaching sessions) for a subset of
the early career teachers at the end of the intervention year;
impacts of the full intervention for one novice teachers a year after the intervention was
delivered; and
cost-effectiveness analysis.
The third report, with a projected release date of May 2022, will summarize findings from
the first two reports and conduct additional analyses to further inform teacher preparation and
professional development. These additional analyses include:
impacts one year after early career teachers received the full or less intensive version of the
intervention;
impacts two years after novice teachers received the full intervention;
correlational analyses to learn about the teaching practices and competencies most important
for effective teaching;
18
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
simulations of the effects of teacher screening criteria;
correlational analyses to examine how key features of implementation relate to impacts; and
mediation analyses to learn about the mechanisms through which the interventions influence
student achievement.
Reports will be written in a style and format accessible to policymakers and practitioners
and will comply fully with the standards set by the National Center for Education Statistics.
A17. Approval not to display the expiration date for OMB approval
The Institute of Education Sciences is not requesting a waiver for the display of the OMB
approval number and expiration date. The study will display the OMB expiration date.
A18. Exception to the certification statement
No exceptions to the certification statement are requested or required.
19
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
REFERENCES
Allen, Joseph P., Christopher A. Hafen, Anne C. Gregory, Amori Y. Mikami, and Robert Pianta.
“Enhancing Secondary School Instruction and Student Achievement: Replication and
Extension of the My Teaching Partner-Secondary Intervention.” Journal of Research on
Educational Effectiveness, vol. 8, no. 4, 2015, pp. 475–489.
Allen, Joseph P., Robert C. Pianta, Anne Gregory, Amori Yee Mikami, and Janetta Lun. “An
Interaction-Based Approach to Enhancing Secondary School Instruction and Student
Achievement.” Science, vol. 333, no. 6045, 2011, pp. 1034–1037.
Bill and Melinda Gates Foundation. “Teachers Know Best: Teachers’ Views on Professional
Development.” Seattle: Bill and Melinda Gates Foundation, 2014.
Boyd, Donald, Pamela Grossman, Hamilton Lankford, Susanna Loeb, and James Wyckoff.
“How Changes in Entry Requirements Alter the Teacher Workforce and Affect Student
Achievement.” Education Finance and Policy, vol. 1, no. 2, 2006, pp. 176-216.
Campbell, Patricia F., and Nathaniel N. Malkus. “The impact of elementary mathematics
coaches on student achievement.” The Elementary School Journal, vol. 111, no. 3, 2011,
pp. 430-454.
Cohen, Benjamin A., and Edward J. Fuller. “Effects of mentoring and induction on beginning
teacher retention.” In Annual Meeting of the American Educational Research Association,
San Francisco, 2006.
Garet, Michael S., Stephanie Cronen, Marian Eaton, Anja Kurki, Meredith Ludwig, Wehmah
Jones, Kazuaki Uekawa, Audrey Falk, Howard S. Bloom, Fred Doolittle, Pei Zhu, and Laura
Sztejnberg. “The Impact of Two Professional Development Interventions on Early Reading
Instruction and Achievement.” NCEE 2008-4030. Washington, DC: National Center for
Education Evaluation and Regional Assistance, Institute of Education Sciences,
U.S. Department of Education, 2008.
Garet, Michael S., Jessica B. Heppen, Kirk Walters, Julia Parkinson, Toni M. Smith, Mengli
Song, Rachel Garrett, Rul Yang, and Geoffrey D. Borman. “Focusing on mathematical
knowledge: The impact of content-intensive teacher professional development.” NCEE
2016-4010. Washington, DC: U.S. Department of Education, Institute of Education
Sciences, National Center for Education Evaluation and Regional Assistance, 2016.
Garet, Michael S., Andrew J. Wayne, Fran Stancavage, James Taylor, Marian Eaton, Kirk
Walters, Mengli Song, Seth Brown, Steven Hurlburt, Pei Zhu, Susan Sepanik, and Fred
Doolittle. “Middle School Mathematics Professional Development Impact Study: Findings
After the Second Year of Implementation.” NCEE 2011-4024. Washington, DC: National
Center for Education Evaluation and Regional Assistance, Institute of Education Sciences,
U.S. Department of Education, 2011.
20
CONTRACT NUMBER: ED-IES-16-C-0021
MATHEMATICA POLICY RESEARCH
Glazerman, Steven, Eric Isenberg, Sarah Dolfin, Martha Bleeker, Amy Johnson, Mary Grider,
and Matthew Jacobus. “Impacts of comprehensive teacher induction: Final results from a
randomized controlled study.” NCEE 2010-4027. Washington, DC: National Center for
Education Evaluation and regional Assistance, Institute of Education Sciences,
U.S. Department of Education, 2010.
Greenberg, Miriam, Mark Nelson, and Sylvia Zelaya. “Best Foot Forward: A Toolkit for FastForwarding Classroom Observations Using Video.” Cambridge, MA: Center for Education
Policy Research, Harvard University, 2015.
Hanushek, Eric A., John F. Kain, Daniel M. O’Brien, and Steven G. Rivkin. “The Market for
Teacher Quality.” National Bureau of Economic Research Working Paper No. 11154.
Cambridge, MA, 2005.
Kane, Thomas J., Jonah E. Rockoff, and Douglas O. Staiger. “What does certification tell us
about teacher effectiveness? Evidence from New York City.” Economics of Education
Review, vol. 27, no. 6, 2008, pp. 615-631.
Kane, Thomas J., and Douglas O. Staiger. “Gathering Feedback for Teaching: Combining HighQuality Observations with Student Surveys and Achievement Gains. Research Paper. MET
Project.” Seattle: Bill & Melinda Gates Foundation, 2012.
Parise, Leigh, Carla Finkelstein, and Emma Alterman. “We Always Want to Get Better:
Teachers’ Voices on Professional Development.” New York City: MDRC, 2015.
Rockoff, Jonah E. “Does mentoring reduce turnover and improve skills of new employees?
Evidence from teachers in New York City.” National Bureau of Economic Research
Working Paper No. 13868. Cambridge, MA, 2008.
Roth, Kathleen J., Helen E. Garnier, Catherine Chen, Meike Lemmens, Kathleen Schwille, and
Nicole I. Z. Wickler. “Videobased Lesson Analysis: Effective Science PD for Teacher and
Student Learning.” Journal of Research in Science Teaching, vol. 48, no. 2, 2011,
pp. 117-148.
Sherin, Miriam G., and Elizabeth A. van Es. “Effects of Video Club Participation on Teachers’
Professional Vision.” Journal of Teacher Education, vol. 60, no. 1, 2009, pp. 20-37.
Sherin, Miriam G., Katherine A. Linsenmeier, and Elizabeth A. van Es. “Selecting Video clips to
Promote Mathematics Teachers’ Discussion of Student Thinking.” Journal of Teacher
Education, vol. 60, no. 3, 2009, pp. 213–230.
Steinberg, Matthew P., and Lauren Sartain. “Does Teacher Evaluation Improve School
Performance? Experimental Evidence from Chicago’s Excellence in Teaching Project.”
Education Finance and Policy, vol. 10, no. 4, 2015, pp. 535–572.
Taylor, Eric S., and John H. Tyler. “The Effect of Evaluation on Teacher Performance.” The
American Economic Review, vol. 102, no. 7, 2012, pp. 3628–3651.
21
This page has been left blank for double-sided copying.
This page has been left blank for double-sided copying.
www.mathematica-mpr.com
Improving public well-being by conducting high quality,
objective research and data collection
PRINCETON, NJ ■ ANN ARBOR, MI ■ CAMBRIDGE, MA ■ CHICAGO, IL ■ OAKLAND, CA ■
TUCSON, AZ ■ WASHINGTON, DC ■ WOODLAWN, MD
Mathematica ® is a registered trademark
of Mathematica Policy Research, Inc.
| File Type | application/pdf |
| File Title | Impact Study of Feedback for Teachers Based on Classroom Videos: Part A, Supporting Statement for Paperwork Reduction Act Submis |
| Subject | OMB |
| Author | MATHEMATICA |
| File Modified | 2017-05-04 |
| File Created | 2017-05-01 |