Instrument 21: Implementation fidelity observation protocol (teacher burden)

VIQI Implementation Fidelity Observation Protocol

November 2020

IMPLEMENTATION FIDELITY OBSERVATION PROTOCOL

The purpose of the Implementation Fidelity Observation Protocol is to collect information about implementation of classroom activities and teacher practices aligned with the selected intervention curricula in a subsample of both treatment and control classrooms. The pre-observation protocol also serves to orient the observer to the activities that will be delivered during the observation.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0508 and the expiration date is 05/31/2021.

1. Pre-Observation Protocol

Overview: During the pilot and full-scale implementation years, the VIQI research team will contact grantee administrators, center administrators, and teachers to schedule classroom observations. They will communicate with participants primarily via email and phone to schedule fidelity observations.

OBSERVATION SCHEDULING

Grantee/Agency Program Administrator Communications: Program Administrators will receive a call or email informing them of plans for site visits, technical assistance, and classroom observations. The following information will be included in such communications:

• Purpose for site visit

• Purpose for classroom observation measuring intervention fidelity

• Proposed timeline for site visit (including classroom observation)

• Request for information about scheduling conflicts

• Team contact information

Center Administrator Communications: Center Administrators will receive calls or emails providing information about the plans for site visits and classroom observations. During the phone call, Center Administrators will be asked to provide days and times during which members of the team can plan to conduct fidelity observations. The following points will be included in communications with Center Administrators to schedule observations:

• Reminder about participation in the VIQI Study

• Purpose for site visit and classroom observation measuring intervention fidelity

• Proposed timeline for site visit (including classroom observation)

• Request for information about scheduling conflicts and to identify target weeks during which they can block their schedule for observations

• Request for information about daily classroom schedules (if not already obtained)

• Team contact information

• Request that they inform teachers of the observation

Teacher Communications: Observers will email or call teachers to inform them about the classroom observation. During the call or email, they will ask teachers to confirm their availability for dates scheduled. The following points will be included in their communications with teachers to schedule observations:

• Purpose for classroom observation measuring intervention fidelity

• Description of data collection procedures in each teacher’s classroom

• Confirm availability on dates for site visit

• Team contact information

OBSERVATION PROTOCOL

Overview: On the day of observations, observers will meet with teachers to go over information about the observation and answer any questions the teacher may have.

Talking Points: The observer will use the following talking points to convey information about the observation to the teacher, or hand them a pamphlet with the following information, and answer any questions the teacher may have:

• Overview: We will conduct observations in your classroom for the VIQI Study. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0508 and the expiration date is XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contractor Contact Name]; [Contractor Contact Address].

• Observation Purpose: These observations will be used to measure how curriculum is being used in classrooms across the study.

• Observation Experience: I have a few questions to ask you before and after the observation. During the observation, I will be a silent observer and will stay out of your way as much as possible.

• Pre- and Post-Observation Questions Length: The questions I will ask you before and after the observation will take 18 minutes.

• Privacy: The research team will do everything possible to keep your identity private, to the extent permitted by law. Only the research team will view the observation data. Teacher and classroom names will never be named in reports or other publications or presentations.

• Voluntary: The observations and questions afterwards are voluntary. You may choose whether to participate or not, and may opt out at any point in the study. Opting out will have no implications for your job.

2. PRE-OBSERVATION PROTOCOL

ASK LEAD TEACHER THESE QUESTIONS PRIOR TO STARTING THE OBSERVATION

1. I have a few questions about the staff assigned to the classroom. For each position:

1. How many [staff position] are assigned to the classroom on a typical day?

2. How many days per week does [person in position] work in this classroom?

3. Does [staff position] work in the classroom for the full day or part of the day?

4. Is [name of person in position] permanently assigned to this classroom?

5. Has [name of person in position] been working in this classroom since September? If no, In what month were they assigned to this classroom?

6. Is the [staff position] present today? If no, Is there a substitute?

NOTE: ASK TEACHER TO POINT OUT WHO IN THE ROOM HOLDS EACH POSITION.

Staff Position	# staff assigned	# days a week	Full-day?	Permanent?	Month assigned	Absent?	Sub present?
Teacher							
Assistant Teacher(s)							
Aide(s)							
Volunteer(s)							
Floater teacher							
Other							

2. I have a few questions about the lessons you have planned for today. Where do the lessons come from?

1. What is the name of the curriculum? Are there multiple curricula used to plan today’s lessons?

2. What unit and/or week of the curriculum do the lessons come from? NOTE: If multiple curricula, ask about each one.

3. If not from a curriculum, did you create the lesson?

4. Is the lesson from another source other than a curriculum (e.g., website; book; etc.)? Please name the source of the lesson.

Curriculum or Source Name*	Unit #/Name	Week #
1)
2)
3)
4)
5)

*For lessons created by teachers, write: “Teacher-created.”

3. POST-OBSERVATION PROTOCOL

ASK LEAD TEACHER THESE QUESTIONS AT END OF OBSERVATION

1. Aside from my presence in the classroom, on a scale of 1 to 5 with 1 being not typical and 5 being very typical, would you say that today was a typical day in the classroom?

Not typical								Very typical
O1		O2		O3		O4		O5

2. Did today’s activities or lessons incorporate a theme or focal question?

Yes [If yes, ask the following:]

No [If no, go to #3:]

1. Please describe the theme(s) or focal question(s):

2. Which of the following was most influential in determining what content you taught today? (Pick one.)

• What your children know and are able to do

• Your curriculum or lesson plans

• What you think the children should know

3. Other than the curriculum(a) you mentioned earlier today, are there any other curricula that you use on an ongoing basis? (check all that apply)

LIST OF CURRICULUM TO BE FINALIZED BASED ON INFORMATION LEARNED IN RECRUITMENT

Creative Curriculum Focus Curriculum Connect4Learning High/Scope DLM Early Childhood Express Opening the World of Learning Frog Street Preschool Pathways to Science (PrePS)

Handwriting without Tears Fundations Let’s Begin with the Letter People Building Blocks Everyday Mathematics Number Worlds Number Talks Pyramid Model for Supporting Social Emotional Competence

Preschool PATHS Tools of the Mind Second Step Open Circle Locally-developed or teacher or center-developed curriculum Other: ______________________________

4. What assessments are you using to track child progress?

(ASK TEACHER TO RESPOND WITH A YES OR NO)

Galileo Ages and Stages Questionnaires Learning Accomplishment Profile Screening (E-LAP, LAP-R, LAP-D) Hawaii Early Learning Profile (HELP) Brigance Preschool Screen Locally-designed assessment or teacher or center-created assessment

Work Sampling System Creative Curriculum Teaching Strategies GOLD High/Scope Child Observation Record (COR) Building Blocks Small Group Record Sheets Child Portfolios (hardcopy) Teacher observations of specific objectives

5. During a typical week in this classroom, how often are the following components incorporated into the school day? (USE A CHECKMARK √ )

	(Typical) Number of days per week						Occasionally	Not enough info/ don’t know
Component	5	4	3	2	1	0
Centers
Lang/Lit activities delivered in whole groups
Lang/Lit activities delivered in small groups – with groups of about 3 to 7 children
Read Aloud
Math activities delivered in whole groups
Math activities delivered in small groups – with groups of about 3 to 7 children
Science activities delivered in whole groups
Science activities delivered in small groups – with groups of about 3 to 7 children
Social-emotional activities delivered in whole groups
Social-emotional activities delivered in small groups – with groups of about 3 to 7 children

IF TEACHER DELIVERED SMALL GROUP LESSONS OR REPORTS DELIVERING SMALL GROUP LESSONS, ASK THE FOLLOWING:

6. Now I want to learn how you select children for small groups.

a) How do you decide who goes into each group?

• Based on skill level

• Based on behavior/ personality

• Other (specify:__________________________________________________________)

b) How often do you change group assignments?

• Daily

• Weekly

• Monthly

• Quarterly

c) Is the content and structure of small group activities the same or different across groups?

• The same

• Different

Why? ___________________________________________________________________

3