INSTRUMENT #2
Personal Responsibility Education Program (PREP) Evaluation
[Introduction.] As you know, the Patient Protection and Affordable Care Act, signed into law in March of 2010, established the Personal Responsibility Education Program (PREP) which funds programs designed to educate adolescents on both abstinence and contraception for the prevention of pregnancy and sexually transmitted infections and at least three adulthood preparation subjects. We are conducting a study of the PREP program on behalf of the U.S. Department of Health and Human Services. The goal of the study is to document how programs funded through PREP are operationalized in the field and to experimentally test a number of selected PREP-funded programs’ effectiveness using random assignment methods.
We are beginning an exploratory phase to obtain feedback from program administrators and staff, researchers, policy experts, and school administrators. We are speaking with you specifically because we are interested in learning more about your program. Eventually, we will discuss involvement in the study with specific programs and, only with their consent, will include them in the evaluation. In addition to any broad strategies or promising approaches you may know of, please be sure to let us know of existing PREP-funded programs that might be appropriate evaluation sites.
Lastly, please note that your responses will be discussed internally among the research team and the funding agency (the Administration for Children and Families) but that, to the extent allowable by law, individual identifying information will not be disseminated publicly.
Program Name:
Program Location:
Sponsoring Organization:
Individual(s) Interviewed: (names and titles)
Contract Staff (as appropriate):
Date of Communication:
Module A: Background
A1. [General Framework] Please provide the general framework by which you view the problem of pregnancy prevention.
Probes: If you were to put the problem of sexual risk behavior and pregnancy among adolescents into a framework, what would that framework look like? What are the primary causes and effects? How do you think of the problem of sexual risk behavior and pregnancy among youth? How do you conceptualize the problem?
A2. [Intervention Points/Approaches] Please indicate the primary points of intervention (i.e. approaches) for preventing pregnancy, from your perspective.
Probes: What are the primary points of intervention, from your perspective? Which are the most critical points of intervention, from your perspective? How would you rank these intervention points, from most to least critical?
A3. [Programs/Strategies] Please describe the programs (or strategies) you know of for targeting these points of intervention.
Probes: Considering your list of critical points of intervention …
What strategies do you know of for targeting these intervention points? Please include any strategies you have heard of, especially including those which may be new, innovative, or heretofore untested.
Of these strategies, which do you think hold the most promise for reducing sexual risk behavior and pregnancy among adolescents?
A4. [Problem] Please tell me about the problems in this community that inspired this program.
Probes: What was the perceived problem or impetus for the program? Did/do you have data to demonstrate this problem?
A5. [Mission] Please tell me about the mission of this program.
Probes: Do you have a mission statement; a vision statement; a strategic plan; a logic model? What are the primary goals of your program?
A6. [Commencement of Activity] Please tell me about how your program began.
Probes: When did your program begin? Was there an organization or entity responsible for getting it off the ground; if so, who? What was the source of initial funding?
A7. [Length of Service] Please tell me about the length of service of this program.
Probes: How long has your program been in existence? Does your program have a set end-date?
A8. [Personal Involvement] Please tell me how you personally became involved in this program.
Probes: What are your reasons for working with this program? What is your motivation for working with this program?
A9. [Qualifications and Experience] Tell us about your qualification and experience in this area, and type of program.
Probes: Do you have any education in this area? Do you have experience facilitating a program like this? Any other qualifications or experience?
A10. [Additional Background] Please tell me any other background information that we should know about this program, and your involvement in it.
Module B: Program Design and Operation
B1. [Initial Program Goals and Design] Please describe the initial program goals, and the initial design, of the program.
Probes: What were the main goals/objectives of the program? How was the program initially designed? Who was involved in the original design of the program (e.g. were school administrators/staff involved)?
B2. [Current Program Goals] Please describe the current goals of the program.
Probes: What are the primary goals of your program? Do you have a strategic plan or a logic model?
B3: [Current Program Design] Please tell me how the program is currently designed.
Probes:
What are the activities/components of the program?
For each component, please elaborate, as appropriate, on:
Content. i.e.:
curriculum/curricula used
commercial or home-made curriculum, or hybrid/mix/in-between
curriculum used in entirety or in parts used
tailoring to local needs?
Frequency, duration, intensity of activity
Completion of program (i.e. how defined)
Target population (race/ethnicity, gender, grades/ages, SES, urbanicity, etc.)
differences by grade level, population group
assessment of age-appropriateness
Voluntary or mandatory participation
Location/setting where component is delivered
Staff responsible for delivering content or overseeing activities, i.e.:
number and type of staff (full-time or part-time; paid or volunteer; teachers, medical personnel, community members, etc.)
background and qualifications of staff
(If program uses student/peer teachers or counselors) Student leadership involvement, i.e.:
ages of students involved
roles and responsibilities
time commitment
educational background
previous knowledge/experience prior to volunteering with program
training received by students
What about special occasion? Does your program ever do anything out of the ordinary? For example, classes for parents, parent-youth workshops, holiday or other celebrations, newsletters?
Are there impacts of the school on the program’s design and/or implementation? Do the logistics of the school affect the program?
Are there any gaps in the program design? Is the program (and the components of the program) adequate to address program goals? Are there missing elements that would strengthen program? What are the reasons for gaps (e.g. financial resources, appropriate staff, space constraints, local opposition, etc.)?
B4. [Changes from Initial Design] Please tell me how the program has changed from the original design and goals to the current arrangements.
Probes: Have there been any shifts from original design (changes in targeting, number or type of program components, strategies, etc.)? Have there been any key events or developments that prompted these changes?
B5. [Service Population] Please tell me about the population you serve.
Probes: What are the demographics of this population (race/ethnicity, gender, grades/ages, SES, urbanicity, etc.)? What is the program catchment area (area in which services are provided, neighborhood(s), communities, schools)?
B6. [Receptiveness of Target Population(s)] Please tell me how you would characterize the receptiveness of the target population(s) to this program?
Probes: Do you have any anecdotal or statistical data showing this receptiveness?
B7. [Organizational Structure] Please describe how this program’s staffing is structured.
Probes: How is staffing in your program organized? What is the organizational structure? Do you have an org chart?
B8. [Implementation and Service Levels] Please tell me about your service strategy and levels of service, i.e. your recruitment, participation, and completion.
Probes:
For each component (listed in B3b), please elaborate, as appropriate, on:
Participant recruitment, i.e.:
strategies for recruitment and retention
problems with recruitment and retention
average number of recruitments per attempted recruitments
targets for recruitment, achievement of targets, and how progress is tracked
Participation, i.e.
target number of participants served
actual number of participants served
average hours of service per individual
targets for participation, achievement of targets, and how progress is tracked
Completion, i.e.
target number of completions
actual number of completions
completion rate (and rate across time, if available)
targets for completion, achievement of targets, and how progress is tracked
Do you have an MIS (Management Information System) in place to track success? What does this look like?
Does your program follow-up with drop-out or non-attendees? How? Does your program have targets for drop-outs, and are you successful?
Where does your program operate? What are the number and location of program offices/sites?
B9. [Advisory Board] Tell me about any community board(s) or advisory group(s)?
Probes: What is this group’s composition? What are the roles and responsibilities of this group?
B10. [Support Services] Please describe any support services your program provides.
Probes: Does your program offer support services for participants, e.g. referrals; peer counseling; transportation; child care; health care, or mental health care; meals; translation; other services?
B11. [Overall Strengths and Weaknesses of Program] Please tell me about the program features or factors that you believe are most important to its success.
Probes: What people or factors have led to your program’s development or success? Are there components of the program that are more successful than others? Do you have any anecdotal or statistical data to demonstrate this success? Are there any areas of disappointment? What people or factors have led to these disappointments?
B12. [Challenges] Please describe the largest challenges this program has faced.
Probes: Were there any challenges that you faced during start up, as compared to currently? Are there any unique challenges of serving this population or community? What future challenges do you foresee?
B13. [Additional Information on Program Design and Operation] Please tell me anything else we should know about this program and its operations.
Module C: Current Evaluation
C1. [Evaluations] Please tell me about any evaluations associated with this program.
Probes: Are there any efforts to evaluate this program (e.g. track attendance, in-house surveys of participants, documentation of how the program is implemented, use of external evaluator, other)? Are there any written documents/reports available?
C2. [Anecdotal Evidence] Please tell me any anecdotal stories you have heard about the outcomes of this program, and perceived impacts.
Probes: What are the perceived impacts and outcomes of this program? What are its major accomplishments?
C3. [Additional Information on Evaluation] Please provide any additional information you may have on research and/or evaluation associated with this program.
Module D: Staffing
D1. [Staff Responsibilities] Please describe the responsibilities of the program’s staff.
Probes: Who has responsibility for what? For what is the average instructor responsible? What does the schedule of the average instructor look like? What about other staff members?
D2. [Recruitment and Retention of Staff] Please tell me about how you have recruited and retained staff.
Probes: How have you located staff? Has recruitment been easy/difficult? What about retention?
D3. [Staff Qualifications and Qualities] Please describe the qualifications for staff for this program, as well as the qualities you look for in staff.
Probes: Are these qualifications written down? Are they clear to applicants? Is it difficult to find people that meet these qualifications? What about the qualities of potential staff: what do you look for? Is there an ideological or philosophical commitment to certain beliefs that you look for? Are these qualities written down? Is it difficult to meet people that have these qualities?
D4. [Staff Buy-In] Tell me about the buy-in that staff have to the program.
Probes: Have staff bought into the program’s mission? What about the program’s design? What about the program’s implementation? Have there been disagreements between staff regarding buy-in to specific components of the program (e.g. mission, design, implementation)?
D5. [Staff Training] Tell me about training staff have received.
Probes: Have staff been trained for this program? If so, how have they been trained, and how many have been trained? What is the frequency of training? What topics have been covered in training? Who has provided training? Are there any gaps in training, or training needs? Has staff been trained outside of this program?
D6. [Turnover] Please tell be about the turnover of staff for this program.
Probes: Have you experienced problems with staff turnover? Are there underlying issues related to staff turnover? How have you addressed these problems?
D7. [Additional Information on Staff] Please tell me anything else we should know about staffing for this program.
Module E: Funding
E1. [Sponsoring Organization] Please tell me about this name(s) and type(s) of sponsoring organization(s).
E2. [Sources of Funding] Please tell me about the program’s sources of funding.
Probes: What are your current sources of funding? Have any of these sources changed? What do you anticipate are future funding sources? What strategies have you implemented to pursue and secure funding?
E3. [Current Budget] Please describe for me your current budget.
Probes: What is a ballpark estimate of your program’s annual budget? Do you have an idea of the breakdown of costs (i.e. for staff, materials, training, etc.)?
E4. [Additional Funding Information] Please tell me any other information we should know about funding for this program.
Module F: Community Context
F1. [Community Characteristics] Please tell me about the characteristics of the community in which the program operates.
Probes: What are urbanicity, population size, SES, race/ethnicity, major industries? Are there major religious affiliations, practices, or influences? Are there major issues or challenges facing the community?
F2. [Community Attitudes toward Problem] Please tell me about the attitudes in the community towards the problem of teen pregnancy.
Probes: What are the prevailing attitudes towards adolescent sexual and other risk behaviors by members of community? What are the beliefs about teen pregnancy (i.e. a large problem, a manageable problem)? Are teen sexual behavior and pregnancy perceived as problems by members of community?
F3. [Visibility of Program] Please describe the visibility of this program.
Probes: Is this program (highly) visible in this community? What about outside of this community?
F4. [Community Attitudes toward Program] Please tell me about the attitudes in the community towards your program.
Probes: What is the level of community support and/or opposition for program from schools/school supervisors/community? What are the sources of support and/or opposition for program from schools/schools supervisors/community? Have you received any positive or negative messages about your program? Are there particular components of your program that are perceived more positively or negatively by the community?
F5: [Community Partners] Please provide a brief description of key partners.
Probes: Please describe the type(s) of organization, history of partnership(s), etc. Do you have a formal or informal arrangement(s) with this/these partner(s)? Do they have a role in the program (e.g., funding, staffing, speakers)? What is the frequency and type of communication or collaboration you have with these partners?
F6. [Community Influence on Program] Please describe how, if at all, the community has influenced your program.
Probes: Have there been any effects of community characteristics and attitudes on your program?
Has the community affected the structure of your program (e.g. number and type of schools, public & private, grade-levels)?
Have school characteristics affected your program’s design or structure (e.g. level of violence, drug use, gang activity, truancy, school failure and drop-out rates in schools participating in the program)?
Has the school climate affected your program’s design or structure (e.g. prevailing attitudes towards adolescent sexual and other risk behaviors by school staff and students, extent to which sexual behavior and teen pregnancy is perceived as a problem by school staff and students)?
F7. [Interactions with State Law and School Board] Please describe how the program has interacted with, and been affected by, state law and (as necessary) the school board(s).
Probes: Have you had to change parts of the program due to either state law or the school board? Have you attempted to work with/influence legislators or the school board?
F8. [Additional Community Information] Please tell me anything else we should know about this community and its interaction with the program.
Module G: Other Programs and Services in Community
G1. [Other Sexual Risk and Pregnancy Prevention Programs] Please tell me about other sexual risk and pregnancy prevention programs and services available in this community.
Probes: Please provide their name, location, school or community sponsorship, services provided, ages served, characteristics of population served, needs of youth addressed by program, perceived impact and effectiveness of other programs on target population and community
G2. [Youth Development Programs] Please tell me about other youth development programs and services available in the community.
Probes: Please provide their name, location, school or community sponsorship, services provided, ages served, characteristics of population served, needs of youth addressed by program, perceived impact and effectiveness of other programs on target population and community.
G3. [Health Programs] Please tell me about other school or community-based health programs and services targeting youth in the community.
Probes: Please provide the services provided, ages served, characteristics of population served, needs of youth addressed by program, perceived impact and effectiveness of other programs on target population and community.
G4. [Additional Information on Programs in Community] Please tell me about any other programs or efforts aimed at pregnancy prevention, that we should know about in this community.
G5. [Unmet Needs in Community] Considering your program and the other programs you have mentioned, please tell me about unmet youth-related sexual risk and pregnancy prevention needs in this community (as well as youth development or youth health needs in the community).
Module H: Lessons Learned, Participation in Evaluation, and Other Programs
H1. [Lessons Learned] Please share any lessons learned.
Probes: Do you have any advice you would give other programs?
H2. [Participation in Evaluation] We will be reviewing responses from a number of programs, after which we may want to follow-up with you about participating in this evaluation. I can explain more about the evaluation if you would like. Would your program be interested in participating in an evaluation like this? If so, what would be the best may to involve your program?
H3. [Other Programs] Are there other programs you know of that might be interested in participating in a study like this? How would be able to get in contact with these programs?
Program Materials to Obtain
Program Plans/Reports
Organization chart
Logic model
Outreach or recruitment materials
Program newsletters
Evaluation reports or survey results
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | APPENDIX B |
| Author | Seth F. Chamberlain |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |