Impact Evaluation of Teacher Residency Programs

Appendix F

Program interview

Appendix F.1

Program interview protocol

Impact Evaluation of Teacher Residency Programs

Residency Program Interview Protocol

Last Updated June 25, 2021

Introduction

The U.S. Department of Education’s Institute of Education Sciences (IES) is conducting a large-scale evaluation of teacher residency programs. The evaluation will provide comprehensive information about the characteristics, effectiveness, and retention of teachers from different types of teacher preparation programs. It will also aim to shed light on effective approaches to teacher preparation used by teacher residency programs that could be implemented more broadly.

As part of this evaluation we are asking residency programs to participate in an interview. The in-depth information collected through this interview will provide a better understanding of the structure of residency programs and the instruction and support they provide to their participants.

Your participation in the study is voluntary and you may refuse to answer any questions you are not comfortable answering. The interview will pose minimal risk to you and will take about 60 minutes to complete. Some of the questions may require additional follow-up. As a token of appreciation for your time, you will receive $50 for completing this interview. Before we proceed, I want to confirm that this is still a good time for you.

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for research purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific residency program, district, school, or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony. Would you like to proceed?

Paperwork Reduction Act Statement: Public reporting burden for this collection of information is estimated to average 75 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the needed data, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The valid OMB control number for this information collection is 1850-0960 which expires 04/30/2024. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Mathematica, 600 Alexander Park, Suite 100 Princeton, NJ 08540 ATTN: Melissa Clark. Do not return the completed form to this address.

A. Background

First, let’s talk broadly about your residency program.

1. How long has your residency program been operating?

2. In what year did the first program graduates begin their first teaching jobs? By program graduate, we mean someone who has completed the program requirements and is eligible to become a teacher of record.

3. What is the overall length of the program for a full-time student, from the time the student initially enters the program and begins taking coursework until the time the participant completes the program requirements and can begin a job as a teacher of record?

1. Length of program for typical participant in calendar time: ________

2. Length of program for typical participant in number of academic terms: ________

Make sure to record how programs define academic term, either as a quarter, a semester, a full school year (fall to spring), or other definition.

4. Are the program’s residents working toward a certification or licensure? Please specify.

5. Are residents working toward an undergraduate or graduate degree? Please specify.

6. Does your residency program have different pathways or requirements based on resident type? For instance, residents focused on STEM or English Learners topics may have different program requirements than non-STEM or non-English Learner residents.

   1. If yes, what are the different types of residents or pathways in your program?

7. Residency programs may include a variety of different organizations as partners. For each of the following types of organizations, please list the program’s partner organizations.

1. Colleges or universities (institutes of higher education)?

2. Foundations and nonprofit organizations?

3. Public school districts?

4. Charter school networks?

5. Other organizations?

B. Recruiting, selecting, and retaining participants

Next, I’d like to ask about how your program recruits new applicants.

1. When recruiting new resident applicants, does your program tailor its recruiting efforts to the needs of the district(s) and/or charter school networks you serve, in terms of the types of teachers they need?

   1. How does your program determine district or charter school network staffing needs?

      1. Do you meet with district or network staff to discuss these needs? If yes, how frequently do you typically meet?

      2. Does your program receive written information on districts’ needs?

      3. Other ways?

2. Now I’d like to ask about your program’s recruiting goals for the 2021–2022 school year.

1. How many residents did you aim to recruit overall for the 2021–2022 school year?

1. How many applications did you receive for the 2021–2022 school year?

2. How many applicants were accepted or admitted to the program for the 2021–2022 school year?

3. How many enrolled for the 2021–2022 school year?

   1. What proportion (or number) of the enrollees are male? Female? Other gender?

      Gender	Proportion of 2021-2022 enrollees	OR Number of 2021-2022 enrollees
      Male
      Female
      Other gender

   2. What proportion (or number) of the enrollees are white? Black or African American? Asian? Native Hawaiian or Other Pacific Islander? American Indian or Alaska Native?

      Race	Proportion of 2021-2022 enrollees	OR Number of 2021-2022 enrollees
      White
      Black or African American
      Asian
      Native Hawaiian or Other Pacific Islander
      American Indian or Alaska Native

   3. What proportion (or number) of the enrollees are of Hispanic, Latino, or Spanish origin?

   4. Which field of studies are the 2021–2022 enrollees planning on entering? For each field of study, what proportion (or number) of the enrollees are planning on entering that field of study?

Field of study	Proportion of 2021-2022 enrollees	OR Number of 2021-2022 enrollees
Field of study 1 _____________________
Field of study 2 _____________________
Field of study 3 _____________________
Field of study 4 _____________________
Field of study 5 _____________________

4. How did enrollment in the 2021–2022 school year compare to enrollment in the year prior to the pandemic (2019–2020)? Enrollment in 2021–2022 was ____ than in 2019–2020. [Options: More than 25 percent lower, 5 to 25 percent lower, About the same, 5 to 25 percent higher, or More than 25 percent higher?

Now I would like to ask about the types of residents your program recruits.

2. Did the program try to recruit diverse residents who more closely reflect the diversity of districts’ (or charter school networks’) student populations? If yes, what types of diverse residents? If yes and B1=yes, was this done to meet specific needs of your partners and/or charter school networks?

3. Did the program try to recruit residents who plan on teaching specific subjects or grade levels? If yes, which subjects and grade levels? If yes and B1=yes, was this done to meet specific needs of your partner districts and/or charter school networks?

4. Did the program try to recruit residents who plan to become special education teachers or English learner/bilingual education teachers If yes, for which types of teachers? If yes and B1=yes, was this done to meet specific needs of your partners and/or charter school networks? Programs may use other terms for English learner students, such as emergent bilingual students.

5. Did the program target recruitment for any other categories of residents? If yes, which ones? If yes and B1=yes, was this to address district (or charter school network) needs?

3. What particular strategies to recruit residents of color or those who plan to become English learner teachers or special education teachers did your program use? Only ask about the types of teachers that programs reported above in 2b and 2d.

1. Residents of color?

2. English learner teachers?

3. Special education teachers?

3. What challenges, if any, does your program currently face when recruiting residents?

Now let’s talk about the screening and admission process.

3. I’m going to list several factors that you may or may not consider when deciding which applicants to admit to your program. First, tell me if you do or do not consider it, then tell me whether it is very important, moderately important, or slightly or not important to the admission decision.

		SKIP IF NOT CONSIDERED OR NOT USED
Factor	Considered?	Very important	Moderately important	Slightly or not important
High school or undergraduate GPA IF CONSIDERED HIGH SCHOOL GPA: What is the minimum GPA required for program admission? ____________ IF CONSIDERED UNDERGRAD GPA: What is the minimum GPA required for program admission? ____________
College major or types of courses taken in high school or as an undergraduate
Quality of undergraduate institution
Score on a standardized test (such as Praxis, Praxis II, GRE, or a state test)
Performance in interview
Performance in sample teaching
Proficiency in languages other than English
Previous work experience in schools (for example, working as a paraprofessional or substitute teacher)
Applicant demographic characteristics, such as age, sex, or race and ethnicity
Any other factors you consider when deciding to admit applicants? (specify) ______________________ ______________________ ______________________

3. Does your program employ any strategies to support participants struggling with the program or to help participants pass exams required for certification? If yes, please describe.

C. Incentives

Next, I have some questions about financial supports for residents.

1. I’m going to list several types of financial support that residents may receive during their residency. First, please tell me the percentage of recipients who receive each type of support. Then please tell me the average amount among those who receive that type of support. Please consider financial support provided by the program or support only available to program participants, not general financial aid from the university. Record any notes about requirements for receiving support or amounts of support that vary greatly across recipients.

Type of financial support	Percentage of participants who receive each form of support	Average value per participant per specified time period (per semester, per year, etc.)	Special notes about any of these supports
Living stipend or salary while in residency
Tuition reduction
Loans or grants
Loan forgiveness
Other (specify) ______________________ ______________________ ______________________

2. After residents become full-time teachers of record, do participants in your program routinely receive any additional financial support above and beyond their regular district-provided salary? Please consider financial support provided by the program or support only available to program participants, not general financial aid from the university.

1. IF YES: What is the average amount of the financial support that participants receive per year, while fulfilling their commitment to this program and the hiring district? How many years do they receive this support?

D. Coursework

Next I have some questions about the coursework residents complete and how it intersects with their clinical experience.

1. First I’d like to ask about the amount and timing of the coursework residents complete.

   1. Do residents complete the coursework before the clinical experience or concurrently?

   2. How many days per week and hours per day is a student taking courses in each academic term? (Ask only for the number of terms reported in A2b.) If A6 is yes (or D2 in the spreadsheet is yes), ask if these numbers vary by resident type. If yes, collect these numbers for each type of resident reported at A6a. Also, if it is easier for the respondent to report credit hours per term, collect those and make a note so we can convert to hours per week.

	Academic term
	First	Second	Third (if applicable)	Fourth (if applicable)	Fifth (if applicable)	Sixth (if applicable)
Days per week
Hours per day
Credit hours

2. Are the coursework and clinical experience designed to complement or reinforce one another?

   1. Is there any specific course that allows students to discuss or analyze their clinical experience with instructors or peers? Please describe.

   2. Do instructors tell participants what to do or watch for during their clinical experience? Please describe.

   3. Other ways?

1. Do district (or network) staff…:

   1. Provide input on coursework needs? If yes, how often do they provide input?

   2. Help design the coursework? If yes, what is the extent of their involvement in the design?

   3. Help deliver the coursework? If yes, what is the extent of their involvement in coursework delivery?

   4. Are district (or network) staff involved in the coursework in other ways? Please describe.

4. Does the coursework focus on teaching practices aligned with:

1. District’s student populations, such as English learner students, students with disabilities, or students of color?

2. District’s curricula or common instructional approaches used in the district? If you do not think that the coursework is specifically designed to aligned with district needs for their curricula or instruction approaches, just let me know.

3. Other district and/or charter network needs? Please describe.

5. Who are the instructors for the coursework (for example, full-time college faculty at a partner university, adjunct or part-time college faculty, district-level employees such as curriculum or assessment director)?

1. If college faculty at a university partner, does the university offer a traditional teacher preparation program in addition to the residency program?

   1. If YES: How does the coursework for the residency program differ from the coursework that the university offers for its traditional teacher preparation program? (Order, quantity, focus, instructors, etc.?)

6. Let’s discuss the full set of courses that residents must complete to receive their certification or degree. We want to know how often each of the following topic areas were emphasized in the coursework. Could you please let me know if you think the following topics are never emphasized, infrequently emphasized, sometimes/occassionally emphasized, or frequently emphasized? Record notes about any differences in course emphasis based on resident type (A6).

	Select one per row
	never	infrequently	sometimes or occasionally	Frequently	don’t know
a. Classroom management	1 	2 	3 	4 	d 
b. Lesson planning	1 	2 	3 	4 	d 
c. General pedagogy	1 	2 	3 	4 	d 
d. Pedagogical content	1 	2 	3 	4 	d 
e. Teaching English learner students	1 	2 	3 	4 	d 
f. Teaching students with disabilities	1 	2 	3 	4 	d 
g. Student engagement	1 	2 	3 	4 	d 
h. Family engagement	1 	2 	3 	4 	d 
Teaching students from diverse economic backgrounds
Cultural awareness, racial equity, and/or racial diversity	1 	2 	3 	4 	d 
Differentiated instruction across students	1 	2 	3 	4 	d 
Child development	1 	2 	3 	4 	d 
Collaborating with other teachers	1 	2 	3 	4 	d 
n. Socio-emotional learning	1 	2 	3 	4 	d 
o. Other aspect of teaching (SPECIFY) (STRING (NUM))	1 	2 	3 	4 	d 

E. Clinical experience

Now I have some questions about the clinical experience your program offers.

1. In each academic term, during a typical five-day school week, how many hours per week is the resident expected to spend in the mentor’s classroom? If needed: The mentor or cooperating teacher maintains ultimate responsibility for the classroom of the resident. A mentor or cooperating teacher has teaching experience or expertise and typically provides information, advice, support, coaching, and feedback to residents.

1. In each term, what is the minimum number of hours per week that the resident is expected to be fully in charge of the classroom?

If A6 is yes (or E3 in the spreadsheet is yes), ask if E1 or E1a vary by resident type. If yes, collect E1 and E1a for each type of resident reported at A6a.

	Academic term
	First	Second	Third (if applicable)	Fourth (if applicable)	Fifth (if applicable)	Sixth (if applicable)
Hours per week that resident is expected to be in the mentor’s classroom
Hours per week that the resident is expected to be fully in charge of the classroom

2. How many mentors do residents have? If residents have more than one mentor, what is the rationale for this approach?

3. How are residents matched to mentors and host schools? Are efforts made to match residents with…:

   1. Mentors who teach the grades and subjects or student populations that residents are interested in teaching?

   2. Mentors who have the same gender, race, or ethnicity as the resident?

   3. Host schools that serve particular student populations?

2. How often are mentor teachers expected to provide residents with the following types of support…:

   1. At the beginning of the first term that the residents are in the mentor’s classroom?

   2. At the end of the last term that the residents are in the mentor’s classroom?

   3. For a particular type of support, if its frequency changes over time, can you please describe, generally, how it changes? For instance, is the change gradual or does it occur at certain milestones?

If there is no expectation that mentor teachers provide residents with a type of support, just let me know.

	Frequency Every day or nearly every day At least once a week, but less than every day At least once a month, but less than once a week Less than once a month Not expected to provide this type of support
	Beginning of the first academic term	End of the last academic term	If change, describe how changes are implemented
Observe the resident’s teaching and provide feedback to the resident
Model a specific instructional practice for the resident in his or her classroom
Have the resident rehearse a specific instructional practice before class or implementation
Co-teach with the resident
Coach the resident on reviewing data to plan future instruction
Help integrate the resident’s coursework with his or her teaching experience
Other types of coaching or support (specify)

5. Are mentor teachers expected to work with residents on the following aspects of teaching? If so, does the program provide training to support mentor teachers in this aspect?

	Are mentor teachers expected to work with residents on this aspect? (Y/N)	If yes, does the program provide training to support mentor teachers in this aspect? (Y/N)
Classroom management
Lesson planning
General pedagogy
Pedagogical content
Teaching English learner students
Teaching students with disabilities
Student engagement
Family engagement
Teaching students from diverse economic backgrounds
Cultural awareness, racial equity, and/or racial diversity
Differentiated instruction across students
Collaborating with other teachers
Other aspects of teaching (specify) ____________________ ____________________ ____________________

6. Are residents required or encouraged to observe any teachers other than their mentors, or to do any other sort of fieldwork? Please describe the nature, amount, and timing and frequency of any such activities.

7. Are residents observed in action by program or district staff (other than their mentor teacher) during their residency? Please describe the nature, purpose, amount, and timing and frequency of these observations, and who conducts them. Do these observers provide feedback directly to the resident, program staff, or both?

F. Selecting, training, and compensating mentor teachers

Now I would like to ask a few more questions about the mentor teachers.

1. Is there a formal application form for potential mentors?

2. Is the residency program involved in selecting mentor teachers? If yes, are mentor teachers selected solely by the residency program or is the school district also involved?

1. If the residency program participates in selecting mentor teachers, I’m going to list several qualities that you may or may not consider when selecting mentor teachers. Please let me know whether you consider each quality.

Quality	Considered? (Y/N)
Strong coaching skills (specify what specific evidence or measures of coaching skills, if any, are used) ____________________ ____________________ ____________________
Teaching effectiveness (specify what specific measures of teaching effectiveness are used) ____________________ ____________________ ____________________
Prior experience as a mentor teacher
Years of teaching experience
Vision/values that align with the program (specify what vision/values) ____________________ ____________________ ____________________
Any other qualities you consider when selecting mentors? (specify) ____________________ ____________________ ____________________

3. Can you describe the training process for new mentors? When and how are they prepared for their duties?

1. What topics are covered? (Coaching, adult learning skills, co-teaching, etc.)

2. Length of time? (Days, hours per day?)

3. Are they provided any tools such as an observation or feedback guide?

4. How, if at all, do you assess mentors’ performance as a mentor? (Formally or informally, approaches and methods used? When, how often?)

5. Are mentors compensated in any of the following ways?

1. A stipend or additional pay for the mentor teacher role? If so, how much?

2. Reduction in expected classroom instruction time?

3. Something else?

5. What proportion of first year mentor teachers return for a second year?

G. Post-residency placement and support

Now I have a few questions about how program graduates are placed in teaching positions.

1. About how many residents graduated from the program in spring or summer 2021?

   1. About how many of these graduates passed their licensure or certification exam?

2. How do participants obtain teaching assignments for their first year as a full-time teacher of record? (Roles and responsibilities of program, district, and school staff versus participants themselves? What factors or goals does the program consider? School characteristics? Participant clustering in schools?)

3. How many years does the program require participants to commit to teaching in a partner district after completing their residency?

4. Considering all the resident graduates who became a full-time teacher of record in the 2021–2022 school year, could you please let me know how many were placed into schools in the following grade levels and subject areas?

Grade level or subject	Elementary	Middle	High
Total
Multiple subjects, teaching self-contained classes for general education students
Multiple subjects, teaching self-contained classes for students with disabilities
English, reading, writing, or language arts
English learner or bilingual programs
Social studies or history
Science
Mathematics
Foreign language(s)
Something else? (specify) _______________ _______________ _______________

5. Does your program make available any of the following supports or benefits to participants after they become full-time teachers of record? Please describe supports or benefits that the program provides to its graduates, not supports or benefits that the district may provide to all new teachers.

		IF YES:
Special support or benefit	Offered?	Which participants receive this support or benefit? All participants Those who want it (volunteers) A limited number (first-come, first-served) Other	How many years do participants receive the support or benefit?
a. Formal mentoring
b. Content area professional development
c. Pedagogical professional development
d. Some other support or benefit

H. Changes to the program

Finally, I have a few questions about recent and upcoming changes to the program.

1. How did the COVID-19 pandemic affect the clinical experience in 2019–2020 and 2020–2021 (for example, was the clinical experience entirely virtual)? Collect responses for each year separately.

1. 2019–2020

2. 2020–2021

2. Beyond the changes due to the pandemic, have there been other changes in the amount of clinical experience that residents received in recent years?

3. Do you anticipate any challenges to program sustainability over the next one to two years? Please describe.