Form 9 Process Evaluation Key Informant Interviews

Appendix G. Key Informant Interview Guides

Appendix G.1 Facilitator Key Informant Interview Guide

Form Approved

OMB No. 0990-

Exp. Date XX/XX/20XX

SMARTool Pilot Replication Project

Facilitators Key Informant Interview Guide

Introduction

Thank you for meeting with us today.

Before we begin, I’d like to go over some information about your participation in this study. If you have any questions, let me know.

• As you know, the Department of Health and Human Services (DHHS) is sponsoring this [SMARTool Pilot Replication Project]. This interview is part of the project’s external evaluation and is designed to get your impressions of the intervention. You have been asked to participate because you are a facilitator, with experience teaching this curriculum. [NOTETAKER’S NAME] and I work for RTI International, which has been tasked to help conduct this study, under a subcontract to the MITRE Corporation. RTI is a not-for-profit research organization headquartered in North Carolina. MITRE has a contract with DHHS to conduct the study. MITRE is a not-for-profit company based in Bedford, Massachusetts and McLean, Virginia.

• There are no risks or benefits to you personally for participating in this interview. Your participation is voluntary. You may decline to answer a question or stop the interview at any time. Any information you share is confidential; we will never use your name in our reports or associate you with statements you make or information that you provide during this interview.

• The interview is designed to last about 60 minutes. I will ask you questions primarily about your experience implementing [SRA curriculum].

• If you have any questions or concerns about this study, please contact the RTI study director, Ellen Wilson (919-316-3337), the MITRE study leader, [NAME (PHONE NUMBER)] or the DHHS study leader, [NAME] ([PHONE).

• [NOTETAKER’S NAME] will be taking notes. We would also like to record the interview, so we do not miss anything you say. We will refer back to the recording to be sure our notes are complete and accurate; the recording will be destroyed after we analyze the information from the interviews.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-xxxx . The time required to complete this information collection is estimated to average 1 hour per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

The curriculum

1. To start, I’d like to ask you about the curriculum you used. I understand that you used the [CURRICULUM NAME] for this [SMARTool Pilot Replication Project]. Had you ever taught this curriculum before?

POSSIBLE PROBES:

• IF YES: How many classes had you taught using this curriculum before this [SMARTool Pilot Replication Project]? [AN APPROXIMATE NUMBER IS FINE]

2. How many classes did you teach this Spring under this [SMARTool Pilot Replication Project]?

3. Is facilitator your primary job? [Yes/No]

[IF YES]

1. Do you work full- or part-time? [Full / part]

2. How many years’ experience did you have facilitating or teaching classes for youth prior to this [project]?

3. How many of those years involved facilitating classes in sex education?

[IF NO] Besides your role as a facilitator, what is your usual job?

POSSIBLE PROBES:

• IF TEACHER: What grade(s) do you usually teach? What classes do you teach? How many years teaching experience do you have?

• IF OTHER: Prior to this [project], did you have any experience teaching, either in the school system or in programs like this? What subjects, or curricula, had you taught? How many years’ experience did you have prior to this [project]?

4. Now that you have facilitated the curriculum, how appropriate do you think the topics were for your [youth/students]?

POSSIBLE PROBES:

• Do you think the curriculum was appropriate for this age group (9^th or 10^th graders)?

• Was it too sexually explicit?

• Did it provide enough information, in the right level of detail?

• Were the teaching methods appropriate for the students? How could you tell?

• Was it practical and skills-based?

5. We know that curricula and the manuals that go with them can differ in the level of detailed guidance they offer to facilitators. Some may be very detailed and directive, while others may provide higher degrees of latitude about, for example, the teaching methods or how content is presented.

1. How specific was the curriculum (Was there a script to follow? Or just general guidance? Or no guidance?)

2. Are there any parts of the curriculum that seemed unclear or less organized than you would have liked?

3. What parts of the curriculum allowed you any degree of latitude?

6. Sometimes a curriculum is changed or adapted by a facilitator to better fit students’ needs or for other purposes. What changes or adaptations did you make to the curriculum? [IF FACILITATOR IS AN IO STAFF MEMBER: This does not include any adaptations [IO NAME] made to align the curriculum with the SmartTool. It only refers to changes you may have made yourself, for your classes.]

[IF CHANGED] POSSIBLE PROBES:

• What prompted that change?

• What kind of material did you add? What did you change?

• Could you tell me about reasons for that [adaptation/addition]?

7. Let’s turn to the teaching methods used. These include things like lectures, role playing, small group discussions, quizzes, class projects, and so forth.

1. Based on your experience, which teaching methods included in your curricula were most effective? [PROBE: Why?]

2. Which were least effective?

PROBES:

• Why did they seem less effective?

• How do you think that could be improved?

3. Did you experience any challenges implementing any of the teaching methods? [PROBE: Which ones? Why were these techniques challenging?]

Stakeholder support

Now I’d like to ask you about the community’s perceptions of this curriculum.

8. In general, how receptive were parents to this curriculum? [POSSIBLE PROBES: How did you become aware of how parents felt about the curriculum? Were the parents of the students involved with homework assignments, e.g., parent-child discussions? (IF YES: Could you tell me more about that?)]

9. How receptive were other members of the community to the curriculum? [POSSIBLE PROBE: How did you become aware of how other members of the community felt about the curriculum?]

10. How receptive was the school administration to the curriculum?

11. Has there been any discussion in the media or elsewhere in this community about sexual health education in recent years?

PROBE:

• How did these [discussions/debates] affect this curriculum or how you facilitated it?

12. What prompted you personally to serve as a facilitator for this curriculum?

Program context

13. Sometimes external events can have an impact on how a curriculum is implemented, or its outcomes -- e.g., teacher demonstrations or strikes and other public events or circumstances. Did any external events or circumstances occur during the implementation of this curriculum that had an impact?

POSSIBLE PROBES:

• What impact did this [event/situation/circumstance] have on your classes?
  
  

14. What other sexual health curricula or programming have been offered to [your students] in the past few years?

POSSIBLE PROBES:

• Did the [CURRICULUM NAME] appear to reinforce the prior programming, or did it have messages that were inconsistent with that prior programming? (Could you tell me more about that?)
  
  

Students

15. Let’s talk about the students.

1. Now that you have implemented the curriculum, do you think that it met the needs of your target population?

2. In what ways was the content [relevant/not relevant] to the students’ daily lives?

POSSIBLE PROBES:

• Could you tell me more about that? Why do you think that is the case?

16. How receptive were the students to the curriculum?

POSSIBLE PROBES:

• How engaged were youth/students in curriculum activities?

• How comfortable were youth/students in discussing the curriculum’s various topics? How could you tell? Did you perceive any discomfort? (IF SO: Please tell me more about that.)

• Did the students appear to accept all the messages conveyed in the curriculum? [IF NOT: Could you tell me more about that?] What topics seemed to resonate most with students? Did students want to talk to you about any of the topics after the class?

• Did students make any comparative comments about the differences/similarities between this [CURRICULUM] and other programming they have had in the past?

• Did you notice any differences in the overall engagement of the youth/students based on things like their background, culture, or other factors?

17. Sometimes curricula are better suited to some groups of students than others. Has it been more difficult to implement this curriculum with any particular student group? Has it been easier to implement this curriculum with any particular student group?

POSSIBLE PROBES:

• For which groups has it been [easier/more difficult]?

• Why do you think that is the case?

• In what way? (for example, was [the issue/problem] related to the curriculum content, the recommended instructional methods, or something else?)

• [IF DIFFICULT FOR SOME GROUPS] How did you address these challenges?

18. Do you think the curriculum is likely to change the students’ attitudes or behavior?

POSSIBLE PROBES:

• IF LIKELY: In what ways?

• IF NOT LIKELY: Could you help us to understand why that is the case?

• Do you think students who are taught this curriculum are likely to wait until they are married to have sex? [IF NO: Do you think students who are taught this curriculum are likely to delay having sex?]

19. Have you detected any unintended negative consequences from this curriculum?

POSSIBLE PROBES:

• IF YES: What are those? Why do you think that is the case?

Facilitators

20. I’d like to know about the guidance you received about this curriculum. Prior to starting this [project], did you attend a formal training program?

POSSIBLE PROBES:

• [FOR FACILITATORS WHO HAD ALREADY CONDUCTED THIS CURRICULUM] Had you been trained on this curriculum in another, earlier context? Did you receive any refresher training or other guidance before starting this [project]?

• [FOR ALL FACILITATORS] Were you given a manual or other guidance materials in addition to the curriculum? Was the training in person (IF NOT, WHAT WAS THE FORMAT?).

21. [IF FACILITATOR RECEIVED FORMAL TRAINING FOR THIS PROJECT] How useful was that training for getting you prepared for teaching this curriculum? Could you tell me more about that?

22. During the [SMARTool Pilot Replication Project], some of the sessions were observed by [IO NAME] staff. Did you ever receive any feedback based from those observations?

POSSIBLE PROBES:

• How often would you say you received feedback?

• How useful was that feedback? Was it relevant to your role as a facilitator?

• Did you change any of your techniques as a result of that feedback? How? Could you tell me more about that?

23. In addition to feedback from those observations, did you receive any other guidance during the course?

POSSIBLE PROBES:

• Who provided that guidance, and how?

• Was it useful?

Challenges / Keys to Success

Now I’d like to ask about the challenges you faced implementing this curriculum, but that were not about the curriculum itself. These might have come from sources external to the organization and staff involved in delivering the curriculum (e.g., local administrative requirements, federal policy) or from internal ones--that is, features of the organization or staff such as differences of opinion, technology or resource challenges.

24. First, what were the greatest challenges you experienced from external sources in facilitating this curriculum?

POSSIBLE PROBES:

• How did you address these challenges?

25. What were the greatest challenges you experienced from internal sources in implementing this curriculum?

POSSIBLE PROBES:

• How did you address these challenges?

• What could [the site] have done to help you be more successful?

26. What challenges, if any, did you encounter in fulfilling the evaluation requirements (e.g., completing the session logs or other activities related to monitoring the [project])?

27. I’d also like to hear about keys to success in implementing this curriculum. What factors would you say contributed most to the implementation process?

POSSIBLE PROBES:

• What kinds of support did you receive from [the site]? How did that help?

Lessons learned

28. Ιn hindsight, what, if anything, would you change or do differently implementing this curriculum?

POSSIBLE PROBES:

• If you were to implement this curriculum again, what would you do to improve the delivery?

• What was the most surprising in your interaction with students?

29. Would you like to teach this curriculum again? Would you teach another SRA curriculum? [Why or why not?]

Background characteristics

30. Lastly, I need to record your demographic characteristics. What is your gender? Are you Hispanic? What is your race? And, finally, what is your highest level of education completed?

Thank you for your time today; your input is very helpful.

-END OF INTERVIEW-

Appendix G.2 Site Representative Key Informant Interview Guide

Form Approved

OMB No. 0990-

Exp. Date XX/XX/20XX

SMARTool Pilot Replication Project

Site Representative

Key Informant Interview Guide

Introduction

Thank you for meeting with us today.

Before we begin, I’d like to go over some information about your participation in this study. If you have any questions, let me know.

• Ask you know, the Department of Health and Human Services (DHHS) is sponsoring this [SMARTool Pilot Replication Project]. This interview is part of the project’s external evaluation and is designed to get your impressions of the intervention. You have been asked to participate because you are a representative of a site that implemented this curriculum. [NOTETAKER’S NAME] and I work for RTI International, which has been tasked to help conduct this study, under a subcontract to the MITRE Corporation. RTI is a not-for-profit research organization headquartered in North Carolina. MITRE has a contract with DHHS to conduct the study. MITRE is a not-for-profit company based in Bedford, Massachusetts and McLean, Virginia.

• There are no risks or benefits to you personally for participating in this interview. Your participation is voluntary. You may decline to answer a question or stop the interview at any time. Any information you share is confidential; we will never use your name in our reports or associate you with statements or information that you provide.

• The interview is designed to last about 60 minutes. I will ask you questions primarily about your experience implementing the [SMARTool Pilot Replication Project].

• If you have any questions or concerns about this study, please contact the RTI study director, Ellen Wilson (919-316-3337), the MITRE study leader, [NAME (PHONE NUMBER)] or the DHSS study leader, [NAME] ([PHONE).

• [NOTETAKER’S NAME] will be taking notes. We would also like to record the interview so we do not miss anything you say. We will refer back to the recording to be sure our notes are complete and accurate; the recording will be destroyed after we analyze the information from the interviews.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-xxxx . The time required to complete this information collection is estimated to average 1 hour per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

The curriculum

1. To start, I’d like to ask you about the [CURRICULUM NAME] the facilitators used.

[IF SITE HAD OFFERED THIS CURRICULUM PREVIOUSLY]

I understand that you have been offering it at this school/organization for the past [#] years.

[IF SITE HAD NEVER OFFERED THIS CURRICULUM PREVIOUSLY]

I understand that this was the first time your school/organization has offered this particular curriculum.

[ALL SITES]

Now that you have implemented the curriculum, how appropriate do you think the topics were for your [youth/students]?

2. What other sexual health curricula or programming, if any, are you currently offering [your students/youth in your organization]?

POSSIBLE PROBE:

• What grade levels [is that/are those] curriculum(a) offered to?

• How is the content different from what you are offering in this course?

3. What other sexual health curricula or programming, if any, have been offered to [your students/youth in your organization] in the past few years?

POSSIBLE PROBES:

• What grade levels [is that/are those] curriculum(a) offered to? [INTERVIEWER: IF MULTIPLE, OBTAIN A CHRONOLOGY OF CURRICULA, BY GRADE LEVEL]

• Where [is that/are those] curriculum(a) taught? Does the course take place on school premises? What percentage of the students take that/those course(s)?

1. How has sex education programming at your school/site changed in the past 5 years?

2. Did the [CURRICULUM NAME] appear to reinforce that prior programming?

• [IF YES] In what way did that other programming reinforce the curriculum?

• [IF NO] Why do you think the [other curricula/programming/activities] didn’t reinforce [THE CURRICULUM]?

4. I’d like to hear about your [school’s/organization’s] decision to be part of the [SMARTool Pilot Replication Project].

1. What led you to select this [project]?

2. How was the decision made?

POSSIBLE PROBE:

1. Did you consult with other people or organizations about this choice?

3. What other options did you consider?

POSSIBLE PROBE:

1. Why did you select this [CURRICULUM NAME] over those?

4. In retrospect, what do you think [schools/organizations] need to take into account when selecting the best sex education curriculum for their students?

Stakeholder support

Now I’d like to ask you about the community’s overall perceptions of this curriculum.

5. In general, how receptive were parents to this curriculum?

1. Did many parents stop by the school/organization to review the questionnaires or curriculum before signing the parental consent form? Or after signing? POSSIBLE PROBE: About how many?

2. Did you do any outreach activities for parents about the [SmartTool Pilot Replication Project] or the curriculum in general, e.g., an informational meeting? [Yes/No]

IF YES:

• How well attended was the meeting?

• What kind of reaction did you get?

• Do you get the sense that the opinions represented views held by a majority or a minority of parents?

6. How receptive were other members of the community to the curriculum?

POSSIBLE PROBES:

• How did you become aware of this?

• What leads you to think that?

7. Has there been any discussion in the media or elsewhere in this community about sexual health education in recent years? [Yes/No]

POSSIBLE PROBE:

• Could you tell me about these discussions?

• What viewpoints were expressed?

• What concerns were raised?

IF YES:

1. Do you get the sense that the opinions represented views held by a majority or a minority of stakeholders/others?

2. Did it seem to be an organic discussion, or one prompted by entities or forces outside the community?

3. How did these [discussions/debates] affect how this curriculum was implemented?

8. [IF CONCERNS WERE RAISED] How did you address the concerns raised?

Program context

9. As I’m sure you know, in addition to media attention and community discussions sometimes other external events can have an impact on how a curriculum is implemented or on its outcomes -- e.g., teacher demonstrations or strikes and other public events. Did any external events or circumstances occur during the implementation of this [project] that had an impact?

POSSIBLE PROBES:

• What impact did this [event/situation] have on the way you implemented the [project] here?

• How did you address this issue? Did that help?

• Are there local or state requirements regarding sex education? [IF YES: What are those requirements? Did the [SMARTTOOL CURRICULUM] meet those requirements?]

10. Things can also happen within [a school / site] that affect how a curriculum is implemented. Were there any internal issues here that affected your ability to implement this curriculum?

POSSIBLE PROBES:

• For example, how satisfied were you with communication with [the IO]?

  • Were you able to get answers to your questions in a timely way? Could you always count on them do provide the help you needed, when it was needed?

  • Could you tell me more about that?

• How was classroom discipline taken into account in this program? e.g., did the classroom teacher remain in the classroom during the sessions? Were the facilitators briefed about school discipline policy?
  
  

Students

11. Let’s talk about the students, that is, the participants in this [project]. Was this curriculum taught as part of a required class, or did the sites need to recruit the students to enroll them?

[IF RECRUITMENT WAS INVOLVED] What recruitment strategies were used to recruit the students?

POSSIBLE PROBES:

• Who was responsible for recruitment and what were their assigned roles?

• Did you experience any challenges during the recruitment process? How did you address these challenges?

12. Now that you have implemented the curriculum, do you think it met the needs of your target population?

POSSIBLE PROBES:

• Could you tell me more about that? Why do you think that is the case?

13. Do you think the curriculum is likely to change the students’ attitudes or behavior in the ways the curriculum intends? For example, do you think it changed the way students think about sex, relationships and personal choices?

POSSIBLE PROBES:

• IF YES: In what ways? What makes you think this way?

• IF NO: Could you help us to understand why that is the case?

Facilitators

14. Let’s turn now to the facilitators (the teachers). Did [SITE NAME] have a role in identifying and recruiting the staff who facilitated the classes? [Yes / No]

[IF YES] How were the facilitators recruited and selected?

POSSIBLE PROBES:

• Did [SITE NAME] have a role in recruitment and selection? What was that role?

15. I’d like to know about the guidance the facilitators received. What role, if any, did [SITE NAME] have in preparing facilitators to implement this curriculum?

16. During the [project], trained observers observed some of the sessions. During the [project], did you personally receive any feedback based on those observations?

PROBE:

• How did you use the feedback?

17. Did [SITE NAME] have a role in monitoring the facilitators or providing them with feedback?

POSSIBLE PROBES:

• Did you have a role in encouraging facilitators to adhere to the curriculum or to [SMARTool Pilot Replication Project] expectations regarding the evaluation?

18. Some educational programs train regular classroom teachers to facilitate the sessions, while others—like this project—use outside staff for facilitation. Thinking about this choice from your perspective as a [school administrator], what are the advantages and disadvantages of these two approaches?

POSSIBLE PROMPTS:

• For example, are there any dis/advantages regarding scheduling the sessions? Teacher workload issues? Privacy issues? Expertise?

Challenges / Keys to Success

19. Now I’d like to ask about the challenges you faced implementing this curriculum. These might have come from sources external to the organization and staff involved in delivering the curriculum (e.g., local administrative requirements, federal policy) or from internal ones that is, features of the organization or staff such as differences of opinion, technology or resource challenges. What challenges did [SITE NAME] experience in implementing this curriculum?

POSSIBLE PROBES:

• What challenges did you experience when it came to parental, school, or community support for this curriculum? How did you address these challenges?

• What challenges, if any, did you encounter in fulfilling the evaluation requirements? (e.g., Were there any challenges in obtaining parental consent? providing the rosters of students? scheduling time for the baseline and follow-up surveys?)

• Were there any other challenges that you experienced?

• How did you address these challenges?

20. Sometimes curricula are better suited to some groups of students than others. Has it been more difficult to implement this curriculum with any particular student group?

POSSIBLE PROBES:

• Why do you think that’s the case?

• In what way? (for example, was the problem related to the curriculum content, the recommended instructional methods, or something else?)

• For which groups has it been harder? How did you address these challenges?

21. Has it been easier to implement this curricula with any particular student group?

POSSIBLE PROBES:

• For which groups has it been easier?

• Why do you think that is the case?

22. I’d also like to hear about keys to success in implementing this curriculum. What factors would you say contributed to the implementation process?

POSSIBLE PROBES:

• What factors helped to secure support from parents, the school, and community?

• Was there anything in particular about the way [SITE NAME] managed the [project], for example?

Lessons learned

23. Ιn hindsight, what, if anything, would you change or do differently implementing this curriculum?

POSSIBLE PROBES:

• If you were to implement this curriculum again, what would you do to improve the recruitment process? The instructional techniques? Other factors?

24. Let’s step back and think about the [CURRICULUM] and its goals and messages. What additional programming, activities or specific curriculum features would you recommend that might reinforce the impact this [CURRICULUM] has on youth in this age group?

25. Is [SITE NAME] likely to use this curriculum again in the future, or do you think another curriculum would meet the needs of your (youth/students) better? Why? POSSIBLE PROBE: Is [SITE NAME] likely to implement ANY SRA approach again?

Thank you for your time today; your input is very helpful.

-END OF INTERVIEW-

18