Discussion Guide for Use with Program Staff

INSTRUMENT #3

Discussion Guide for Use with Program Staff

Personal Responsibility Education Program (PREP) Evaluation

Discussion Guide for Use with Program Staff

[Introduction.] As you may know, the Patient Protection and Affordable Care Act, signed into law in March of 2010, established the Personal Responsibility Education Program (PREP) which funds programs designed to educate adolescents on both abstinence and contraception for the prevention of pregnancy and sexually transmitted infections and at least three adulthood preparation subjects. We are conducting a study of the PREP program on behalf of the U.S. Department of Health and Human Services. The goal of the study is to document how programs funded through PREP are operationalized in the field and to experimentally test a number of selected PREP-funded programs’ effectiveness using random assignment methods.

We are beginning an exploratory phase to obtain feedback from program administrators and staff, researchers, policy experts, and school administrators. We are speaking with you specifically because we are interested in learning more about your program. Eventually, we will discuss involvement in the study with specific programs and, only with their consent, will include them in the evaluation. In addition to any broad strategies or promising approaches you may know of, please be sure to let us know of existing PREP-funded programs that might be appropriate evaluation sites.

Lastly, please note that your responses will be discussed internally among the research team and the funding agency (the Administration for Children and Families) but that, to the extent allowable by law, individual identifying information will not be disseminated publicly.

Program Name:

Program Location:

Sponsoring Organization:

Individual(s) Interviewed: (names and titles)

Contract Staff (as appropriate):

Date of Communication:

Module A: Background

A1. [Current Role] Please describe your current role in this program.

Probes: What is your current role/position in this program? Are you full-time or part-time; paid or volunteer; adult(s) or youth/peer leader(s) or counselor(s)?

A2. [Personal Involvement] Please tell me how you personally became involved in this program.

Probes: What are your reasons for working with this program? What is your motivation for working with this program? What are your reasons for being involved?

A3. [Attitudes and Beliefs] Please describe your personal beliefs about youth sexual risk behaviors and pregnancy prevention.

Probes: Does the program mirror your beliefs? In what ways is the program similar/different?

A4. [Qualifications and Experience] Please tell me about your qualification and experience in this area, and type of program.

Probes: Do you have any education in this area? Do you have experience facilitating activities like this? Any other qualifications or experience?

A5. [Training] Please tell me about training you have received in this program.

Probes: Have your received training while working in the program? What has this training consisted of? What is the frequency, topics, and providers of the training? What has been most helpful from this training? Are there any gaps/training needs?

A6. [Additional Background] Please tell me any other background information that we should know about this program, and your involvement in it.

Module B: Component Description

B1. [Description of the Component] Please describe the activity you are in charge of.

Probes:

1. Content. i.e.:

• curriculum/curricula used

• curriculum used in entirety or in parts used

• tailoring to local needs?

2. Frequency, duration, intensity of activity

3. Strategies used to deliver content (probe for innovative strategies)

4. Completion of program (i.e. how defined)

5. Target population (race/ethnicity, gender, grades/ages, SES, urbanicity, etc.)

• differences by grade level, population group

• assessment of age-appropriateness

6. Voluntary or mandatory participation

7. Location/setting where component is delivered

8. (If program uses student/peer teachers or counselors) Student leadership involvement, i.e.:

• ages of students involved

• roles and responsibilities

• time commitment

• educational background

• previous knowledge/experience prior to volunteering with program

• training received by students

B2. [Description of Special Occasions] Please describe any special occasions that occur with your activity.

Probes: Do you ever do anything out of the ordinary with your activity? For example, classes for parents, parent-youth workshops, holiday or other celebrations, newsletters?

B3. [Goals of Component] Please tell me about the goals for this program.

Probes: What do your supervisors see as the goals of this activity? What do you see as the goals of this activity? Are there any differences between the goals as you see them and as your supervisors see them?

B4. [Additional Description of Component] Please tell me anything else that is important to know about the activity(ies) you facilitate.

Module C: Recruitment, Participation, and Retention

C1. [Recruitment] Please describe participant recruitment for this program.

Probes: How are participants recruited for this program? Who recruits them? Are you involved with recruitment? Are there any problems with recruitment? How have those problems been addressed? Does your program have targets for recruitment? Are you successful? Is your program successful?

C2. [Participation] Please tell me about participation in this program, specifically how many participants are involved.

Probes: How many participants are involved in your program? How many participants do you see on average (i.e. what is your “caseload”)? Does participation vary from activity to activity/class to class/etc.? Does your program have any “participation” targets? If so, how do you track them? Are you successful? Is your program successful?

C3. [Retention and Completion] Please describe how you retain participants for this program, and whether students complete.

Probes: Are there many “drop-outs” for your program? What strategies have you (or others in your program) employed to increase participation? Does your program have any “drop-out” targets? If so, how do you track them? Are you successful? Is your program successful? How successful has your program been at maintaining participation? What proportion of participants actually complete the program? Is there a target for completion? Are you successful? Is your program successful?

C4. [Additional Information about Recruitment, Participation, and Retention] Please tell me other important things to know about recruitment, participation, and retention.

Module D: Participant Engagement

D1. [Participant Engagement] Please tell me how engaged participants are in this program, and in your specific activity.

Probes: Do participants in your activity seem engaged? How do participants respond to the program? Is engagement easy, or do participants seem reluctant to engage?

D2. [Strategies to Increase Engagement] Please describe any strategies you (or others in your program) employ to engage participants.

Probes: Do you have to employ methods to engage participants? What strategies do you use to engage your participants in the activities, or to increase engagement? Do you have any anecdotal stories of success or challenge? Does participant response change over time?

D3. [Additional Information about Participant Engagement] Please describe other important things to know about participant engagement.

Module E: Strengths, Weaknesses, and Perceived Effectiveness of Program

E1. [Overall Strengths and Weaknesses of Program] Please tell me the overall strengths and weaknesses of the program, as you see them.

Probes: What do you perceive to be the program features or factors most important to its success? What people or factors have led to your program’s success? Are there any areas of disappointment? What people or factors have led to these disappointments?

E2. [Challenges] Please tell me the challenges this program has faced, is facing, and/or will face.

Probes: Were there any challenges that you faced when you first started, as compared to currently? Are there any unique challenges of serving this population or community? What future challenges do you foresee?

E3. [Perceived Effectiveness] Please describe the effectiveness of the activities you facilitate.

Probes: What effect do the activities you facilitate seem to have on participants? Do you think they are effective? How effective do you think they are – what is the extent of change you see in your participants? Which activities that you facilitate seem to be most effective?

E4. [Evaluations and Evidence] Please tell me about any data or evidence you have about this program.

Probes: Are you aware of any efforts to evaluate this program (e.g. track attendance, in-house surveys of participants, use of external evaluator, other)? Are there any written documents/reports available? Are there any anecdotal stories you have heard or witnessed as to the outcomes of this program? What are the perceived impacts and outcomes of this program? What are its major accomplishments?

E5. [Additional Information Program Effectiveness] Please tell me anything else we should know about the effectiveness of this program.

Module F: Community Context

F1. [Community Characteristics] Please tell me about the characteristics of the community in which the program operates.

Probes: What are urbanicity, population size, SES, race/ethnicity, major industries? Are there major religious affiliations, practices, or influences? Are there major issues or challenges facing the community?

F2. [Community Attitudes toward Problem] Please tell me about the attitudes in the community towards the problem of teen pregnancy.

Probes: What are the prevailing attitudes towards adolescent sexual and other risk behaviors by members of community? What are the beliefs about teen pregnancy (i.e. a large problem, a manageable problem)? Are teen sexual behavior and pregnancy perceived as problems by members of community?

F3. [Visibility of Program] Please describe the visibility of this program.

Probes: Is this program (highly) visible in this community? What about outside of this community?

F4. [Community Attitudes toward Program] Please tell me about the attitudes in the community towards your program.

Probes: What is the level of community support and/or opposition for program from schools/school supervisors/community? What are the sources of support and/or opposition for program from schools/schools supervisors/community? Have you received any positive or negative messages about your program? Are there particular components of your program that are perceived more positively or negatively by the community?

F5: [Community Partners] Please provide a brief description of key partners.

Probes: Please describe the type(s) of organization, history of partnership(s), etc. Do you have a formal or informal arrangement(s) with this/these partner(s)? Do they have a role in the program (e.g., funding, staffing, speakers)? What is the frequency and type of communication or collaboration you have with these partners?

F6. [Community Influence on Program] Please describe how, if at all, the community has influenced your program.

Probes: Have there been any effects of community characteristics and attitudes on your program?

1. Has the community affected the structure of your program (e.g. number and type of schools, public & private, grade-levels)?

2. Have school characteristics affected your program’s design or structure (e.g. level of violence, drug use, gang activity, truancy, school failure and drop-out rates in schools participating in the program)?

3. Has the school climate affected your program’s design or structure (e.g. prevailing attitudes towards adolescent sexual and other risk behaviors by school staff and students, extent to which sexual behavior and teen pregnancy is perceived as a problem by school staff and students)?

F7. [Interactions with State Law and School Board] Please describe how the program has interacted with, and been affected by, state law and (as necessary) the school board(s).

Probes: Have you had to change parts of the program due to either state law or the school board? Have you attempted to work with/influence legislators or the school board?

F8. [Additional Community Information] Please tell me anything else we should know about this community and its interaction with the program.

Module G: Other Programs and Services in Community

G1. [Other Programs] Please tell me about other programs related to pregnancy prevention in this community.

Probes: Are there other (a) sexual risk and pregnancy prevention; (b) youth development; or (c) health programs and services available? Please provide their name, location, school or community sponsorship, services provided, ages served, characteristics of population served, needs of youth addressed by program, perceived impact and effectiveness of other programs on target population and community

G2. [Unmet Needs in Community] Considering your program and the other programs you have mentioned, please tell me about unmet youth-related sexual risk and pregnancy prevention needs in this community (as well as youth development or youth health needs in the community).

Module H: Lessons Learned and Other Programs

H1. [Lessons Learned] Please share any lessons learned.

Probes: Do you have any advice you would give other programs?

H2. [Other Programs] Are there other programs you know of that might be interested in participating in a study like this? How would be able to get in contact with these programs?

Program Materials to Obtain

Curriculum and curriculum materials

Activity plans

Data collection instruments (e.g. attendance lists)