Study on Sustaining the Positive Effects of Preschool

Appendix I. Evaluator Protocol

Study on Sustaining the

Positive Effects of Preschool

Evaluator Protocol

Interviewer_____________________________________ Interviewee ID #_________________

Site __________________________________________ Date/Time_____________________

Introduction

My name is ______, and I work for American Institutes for Research (AIR). We’ve been contracted by the U.S. Department of Education to conduct case studies on sites that are implementing particular programs to support early elementary students.

As you may know, the purpose of the study is to document characteristics of programs and policies in early elementary school that support student growth and development, such as PK–3 alignment, differentiated instruction, and other programs designed to sustain the advantages of preschool. We are interested in learning more about the theoretical or practical background of these programs, how they are implemented, how they are sustained and funded, and their outcomes.

I want to reassure you that your responses to this interview will be used only for research purposes. Though the district will be identified, no part of the study involves evaluation of any individual, and we will not provide information that identifies you as an individual in any reporting. I will now give you a consent form that provides more details about the study and your rights as a participant.

Do you have any questions about the study?

I plan to audio record this session, solely for our note-taking purposes. Only research staff will use the audio recording, and we’ll destroy it when the project is done. Is that okay with you?

Role and Perspective

1. I’d like to start by asking you to tell me a bit about yourself. I understand that you are the evaluator of [PROGRAM]. Briefly, can you tell me what your role has been in relation to [PROGRAM]?

1. How did you come to be involved with [PROGRAM]?

1. Can you tell me more about the organization that you work for and about the kind of work they typically engage in?

2. Why did [ORGANIZATION] become interested in partnering with [or evaluating] [PROGRAM]?

   1. Can you describe when [ORGANIZATION] got involved? Was it before or after [PROGRAM] began?

Evaluation Questions

4. When did your work evaluating [PROGRAM] begin?

5. Can you tell me about the basic study design of the evaluation?

6. What types of information have you collected over the course of the evaluation work?

   1. Why did you select this information to collect?

7. How does the evaluation measure students’ progress?

8. Has the evaluation documented gains in student outcomes in kindergarten through third grade?

If interviewee reports effect:

1. For which outcomes have you seen gains?

2. How are these gains measured or documented?

3. Have you seen gains for any particular student groups?

4. Have these gains been consistent over time?

5. Can you share or point us to the data or report illustrating these patterns?

If no effect:

1. What do you think are the barriers to seeing more student gains?

9. Has the evaluation documented changes in teacher practices? Teacher turnover?

If yes:

1. Have you seen any patterns?

2. Can you share any data with us?

If no:

1. What do you think are the barriers to seeing changes in teacher practice?

10. What else have you learned from the evaluation so far?

11. How have you shared the results of your evaluation with district and school staff?

12. In what ways and for how long will you continue evaluating the program?

13. Do you expect your work to help the district improve the program? If so, how?

Success, Challenges, and Closing

14. What challenges have you encountered working with a district on this type of initiative? How have you overcome those challenges?

15. Overall, what is working well with [PROGRAM]? How do you know?

16. In your opinion, what are the most important aspects of this program?

17. What aspect of [PROGRAM] needs strengthening? Why?

18. What challenges have you noticed in the district(s) and schools in implementing [PROGRAM], and what steps may be needed to address them?

19. [We already have (XX DOCUMENT)] OR [We do not have any written documents on (PROGRAM)]. Are there [other] written resources that document the program and its evaluation?

    1. Could we get a copy of those?

20. Those are all the questions I have. Are there any questions that I should ask about [PROGRAM] that I didn’t? Is there anything else you’d like to share with me today in regard to [PROGRAM]?

American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—I-1

1875_06/15