Design Survey Discussion Guide for use with Researchers, Policy Experts, and Macro-Level Coordinators

INSTRUMENT #1

Interview Guide for Use with PREP State-Level Coordinators

Interview Guide for Use with PREP State-Level Coordinators

Possible Data Collection Modalities:

• Telephone interview

• In-person interview

Department Name:

Location:

Individual Interviewed (name and title):

Individual’s Contact Information (phone and email):

Contractor Staff Member Administering Survey:

Date of Communication:

Introduction

Thank you for taking the time to talk with me about the Personal Responsibility Education Program (PREP) in your state!

As background, as you know, PREP is one of multiple efforts at the Federal level to reduce teen pregnancy through the use of evidence-based practices. PREP has provided funding to states (which is most often distributed to sub-awardees) for the replication of evidence-based effective programs or substantial incorporation of elements of effective programs.

The Administration for Children and Families (ACF) (within the federal Department of Health and Human Services), which oversees PREP, is interested in learning about how PREP-funded programs are operationalized in the field and how selected PREP-funded programs impact behavior and health outcomes for youth. Therefore, ACF has hired [INSERT CONTRACTOR NAME] to conduct a multi-component evaluation of the PREP program.

The evaluation of PREP will involve three components - a Design and Implementation Study, a Performance Analysis Study, and an Impact and Implementation Study. All three components are vitally important for informing future efforts to implement evidence-based programs to prevent teen pregnancy.

This interview will contribute to the Design and Implementation Study, which will be a broad descriptive study of how states designed and implemented their PREP programs. The data collected through the interviews we are conducting will be reported in two ways. First, a summary profile will be created for each state that will contain PREP program design decision facts – for example, the selected program models, populations to be served, number and location of sub-awardees, and number of total youth to be served. This information will be reported for each state receiving PREP funding. While we will not attribute this factual information to a specific respondent, it will be attributed in aggregate to a specific state. Second, the reports we produce will discuss themes emerging from responses regarding “how” and “why” PREP program decisions were made. Such responses will not be attributed to any one state, but will instead be analyzed and reported as part of overall trends across all states.

Before we begin, I want to inform you of several important things about the interview:

• Your state has agreed to participate in the PREP national evaluation as a requirement of receiving a PREP grant. You are being asked to participate in this interview because it is a component of the national evaluation.

• The interview should take an hour or less.

• I’m going to ask you questions in three areas: (1) background information on existing teen pregnancy prevention programs in your state and how the state became aware of and decided to apply for PREP funding, (2) information on how PREP is being administrated in your state, and (3) information on the particular program models your state is implementing.

• The information you provide will be used only for research purposes to describe how PREP is being implemented in your state. Your responses will be used only to characterize the general experiences of your state, without attribution to you or any other specific individual.

• Your answers to questions will not affect your grant—now or in the future—and will not be shared with anyone else other than the researchers.

• An IRB has reviewed and approved this study. If you have any questions about your rights as a research participant, you can contact Melissia Billarial at Public/Private Ventures IRB, toll free at 800-755-4778 x4482. Questions about the overall study can be directed to Susan Goerlich Zief of Mathematica Policy Research at 609-275-2291.

Before we begin, do you have any questions, comments, or concerns?

Module A: Background Information

[Context]

1. Please describe your background (with regard to the PREP program and teen pregnancy prevention programming) and current position.

Probes: What are your current responsibilities? How long have you been in your current position (# of months)? Is your position newly created to support the PREP program in your state? What certifications, degrees, or credentials do you hold that are relevant to your current position? How long have you been working on the issue of teen pregnancy?

2. Thinking broadly about the issues of sexual risk behavior and pregnancy among adolescents, in your opinion what do you think are the causes of these problems? What are the effects? If you were to put the problems of sexual risk behavior and pregnancy among adolescents into a framework that explains the causes and effects, what would that framework look like?

Probes: How do you think of the problem of sexual risk behavior and pregnancy among youth? Are there any established frameworks with which you are familiar that accurately represent your perspective on the causes and effects of teen pregnancy?

3. In your opinion, what problems does your state face in terms of teen pregnancy and its associated risk behaviors? What specific issues are you attempting to address through PREP and related teen pregnancy prevention programs?

Probes: What is the prevalence of sexual risk behaviors among adolescents in your state? What is the teen pregnancy rate for your state? What is the teen birth rate? What communities or populations (racial, economic, geographical, vulnerable) are most at-risk?

4. In your opinion, what are the general attitudes in the state toward the problem of teen pregnancy?

Probes: What are the beliefs about teen pregnancy (ie, a large problem, a manageable problem)? How would you describe the political environment in your state with regard to teen pregnancy prevention programs?

5. Are there specific policies in place in this state that directly impact teen pregnancy prevention programming? For example, has the legislature passed laws which either mandate or restrict specific approaches to teen pregnancy prevention or HIV/STD programming?

Probes: How have these laws influenced the PREP programming decisions that have been made in your state.

6. Please describe programs and/or activities aimed at decreasing teen pregnancy and its associated sexual risk behaviors currently operating in your state, other than the PREP program.

1. What % of the teen population do you think are receiving these services?

   2. Does the state collect these data? Is your estimate based on these state data, or some other means, such as observation or personal experience?What are the characteristics of the population (racial, ethnic, economic, etc) receiving services?

2. In what settings do the programs operate? (Probes: For example, are they provided in religious-based organizations, community-based organizations, university and schools, etc?)

3. How and at what levels are these programs funded? What programs are funded with federal dollars? State dollars? Non-profits, foundations, and other organizations?

4. To what degree is there coordination in the state around teen pregnancy prevention service delivery?

5. How well do you think these programs/activities address the causes of teen pregnancy as you explained them earlier?

7. Is there any other basic background information about teenage pregnancy, teenage pregnancy prevention, or the PREP program in your state that you think we should know?

[Approach/Strategy]

1. How did your state become aware of the availability of PREP funds?

2. Why did your state decide to apply for PREP funds?

3. Did you seek assistance in developing your proposal for PREP funds? If so, who provided this assistance and what type of assistance did they provide?

4. Does your state have an overall approach/strategy to reduce teen pregnancy and associated risk behaviors?

1. If so, could you please discuss this overall approach/strategy?

2. How did your state develop this overall approach/strategy?

3. Did your state develop a formal logic model to describe this approach/strategy?

4. How does PREP fit into this approach/strategy?

5. Is there any other background information related to your state’s PREP program approach/strategy that you think we should know?

[Misc.]

6. This study will give us the opportunity to learn a great deal about how states design and implement their PREP programs. From your perspective, what do you think are the most valuable questions that we could answer through this study?

Module B: Program Administration

[State Agency Structure/Administration]

1. What state agency is administering the PREP program?

1. Can you describe the following:

   1. Agency structure

   2. Line(s) of authority and agency/reporting responsibilities

   3. Primary contact with sub-awardees/local entities

   4. Program staff

2. Please identify:

1. PREP program settings

2. Specific target population(s) (if applicable)

3. Number of youth intended to receive PREP-funded services (per session? per year? per the entire grant period?)

4. Average award amount (if applicable) to sub-awardees

5. Total number of sub-awards made (if applicable)

3. What led the state to shape the program in this way? Please describe how and why the state arrived at the decisions you just described regarding the following:

1. PREP program settings

2. Specific target population(s) (if applicable)

3. Number of youth intended to receive PREP-funded services

4. The appropriate award amount (if applicable)

5. The number of awards (if applicable)

4. What other funding streams, if any, are being integrated with PREP funds?

1. If other streams are integrated with PREP funds,

   1. What other government entities (state, county, city, or federal) are involved?

   2. How are the roles defined?

5. How has PREP influenced current efforts to address teen pregnancy and its associated risk behaviors in your state? Has PREP been integrated into pre-existing programs? Have other programs been eliminated or integrated into PREP?

6. Please tell me about the cost of administering the PREP program in this state.

Probes: What is the total amount of funding that is dedicated to the PREP program? (Note: We will likely know this from grant documents, but this question will confirm that the respondent understands this.) Of this total, how much of the funding goes to administrative costs? How much is awarded in the form of grants to sub-awardees?

7. Considering the costs of the PREP program and the number of youth expected to be served, it appears that the program will cost about $X per youth participant (not including administrative costs). Is this about right? Did the cost per participant figure into any of your plans for the PREP program?

8. Is there any other information related to the state agency or administering structure that you think we should know?

[Award Process]

1. Please identify:

1. Entities eligible for sub-award

2. Award process

3. Criteria used to award/allocate PREP funds to sub-awardees/local entities

2. What led the state to shape the program in this way? Please describe how and why the state arrived at the decisions you just described regarding the following:

1. Entities eligible for sub-award

2. Award/selection process

3. Award criteria/eligibility requirements

3. Is there any other information related to the award process that you think we should know?

[State and Sub-Awardee Relationship]

1. Please describe the relationship between the state agency and the sub-awardees/local administering entities.

Probes: How closely is the state agency overseeing the PREP program working with its sub-awardees/local administering entities? Is there regular communication between the state and local entities? What process has been established for monitoring and oversight by the state agency?

2. Please identify:

1. State requirements that sub-awardees/ local administering entities must fulfill

2. Assurances they are required to certify as sub awardees (for example, having to do with separation of religious activities, Americans with Disabilities Act (ADA), non-discrimination, anti-bullying/harassment, and/or anti-stigma regulations)

3. The guidance provided by the state to sub-awardees regarding the PREP Program

4. Support and/or training and technical assistance provided to sub-awardees/ local administering entities

5. Data (e.g. performance data), if any, that sub-awardees/local administering entities are required to report to the state administering agency

6. Monitoring surrounding performance expectations of sub-awardees (e.g. recruitment/retention targets; fidelity to program model).

   1. If this type of monitoring is taking place:

      1. How is it being administered?

      2. What type of feedback mechanisms are in place?

      3. Will the data that you collect via monitoring inform decision-making (either on the state or the sub-awardee level)? If so, what type of actions do you expect to take based on what you learn through monitoring?

3. What led the state to shape the program in this way? Please describe how and why the state arrived at the decisions you just described.

4. Did the state develop a formal plan for the implementation of the PREP program? For example, does a formal plan exist which outlines activities such as delivering services, monitoring program implementation, and providing training and technical assistance?

5. Is there a specific mechanism to ensure that gay, lesbian, bisexual, and transgender (GLBT) youths’ concerns may be adequately addressed? Can you describe it?

6. Will the state’s PREP program – either on a state-wide or a sub-awardee level – be formally evaluated in any way? If so, how?

7. Is there any other information related to the state/sub-awardee relationship that you think we should know?

[Miscellaneous]

1. Did your state establish any formal community partnerships for the PREP program?

2. How, if at all, are local community groups or other agencies/organizations involved in the administration of the PREP program? To your knowledge, how has the PREP program been received by local community stakeholders?

Module C: Program Design

[Guiding Considerations]

1. Please describe broadly, from the beginning, the process that your state went through in order to (1) select or design a program model or program models to be implemented with PREP funds and/or (2) to provide guidance to sub-awardees related to selecting or designing a program model or program models to be implemented with PREP funds.

2. Did your state identify and assess potential needs, or ask sub-awardees to do so? If so:

   1. Who was involved in this process? How were they involved?

   2. What data did you use?

   3. How did the needs assessment influence your program design? Did it influence your selection of (or guidance related to) program models? If so, how?

3. In designing the overall PREP program in your state, did you:

1. Assess the current organizational infrastructure and capacity in your state? If so, how did these considerations influence your program design?

2. Secure buy-in from key stakeholders (for example, school district administrators, community leaders, elected representatives, parent organizations)? If so, who were these stakeholders and how did you engage with them?

3. Consider the sustainability of PREP-funded programs in the future if federal funds do not continue? If so, what are your plans to sustain the programs?

4. Program leader (or “champion”): Was there someone who saw this PREP opportunity and really led the charge (e.g. held meetings, influenced decision-makers) to formulate it and make it a reality?

[Program Model Selection and Design]

1. Please describe how your state plans to meet federal guidelines regarding the PREP program. Specifically, how does the state plan to ensure that PREP-funded programs:

1. Replicate evidence-based effective programs or substantially incorporate elements of evidence-based effective programs

   1. Did your state identify one or more evidence-based program model(s) for replication? Why or why not? If your state identified one or more evidence-based models, which program models were selected and why? (Probes: How do you define replication as it relates to the programs you selected? For example, will grantees be required to meet only “core components” of the program or are they expected to replicate every aspect of the program?)

   2. Alternately, did your state deem any program model(s) or characteristics of program models inappropriate for your state? Why or why not?

   3. Did your state choose to substantially incorporate elements of various programs? Why or why not? If you chose to substantially incorporate elements, what elements did you choose? How did you choose these elements?

   4. Did your state leave the selection of program models up to sub-awardees? Why or why not? If so, did your state provide any guidance related to program model selection? What guidance did you provide? Why did you develop this guidance?

2. Place substantial emphasis on both abstinence and contraception

   1. Did your state develop requirements or provide guidance around placing substantial emphasis on abstinence and contraception? Why or why not? (Probes: How did you define substantial? Was there a list of requirements?)

   2. If so, what requirements or guidance did you provide? Why did you develop these requirements or provide this guidance?

3. Incorporate at least three adulthood preparation subjects

   1. Did your state develop requirements or provide guidance around the incorporation of adulthood preparation subjects? (Probes: At what point is a component sufficiently incorporated? How was this determined?)

   2. If so, what requirements or guidance did you provide? Why did you develop these requirements or provide this guidance?

2. How has your state addressed the medical accuracy of programming?

3. Will your state offer training or technical assistance support to incorporate adult preparation material or other adaptations into evidence-based program models? If so, please describe the training and/or technical assistance that will be offered.

4. How, and by whom, were final decisions related to the curriculum of PREP program made?

5. To what extent do you believe the PREP program, as it is currently being implemented in your state, has the potential to meet the needs of your state outlined earlier? Why?

6. Is there any other information related to program model selection or design that you think we should know?

[Adaptations]

1. Did the state create a process for approving/rejecting program model adaptations? If so, what is this process?

2. Specifically, did the state create standards regarding adaptations having to do with:

1. Cultural appropriateness

   1. What aspects of culturally-appropriateness were addressed? For example, were linguistic adaptations addressed? Were program content adaptations addressed?

2. Age appropriateness

3. Other aspects of the program model, including dosage (length of sessions, overall program), intensity (time spent receiving intervention and completing any associated components), components, setting, etc.

3. Based on the previous question, could you please identify:

1. The standards created regarding adaptations having to do with:

   1. Cultural appropriateness;

   2. Age appropriateness;

   3. Other aspects of the program, including dosage, intensity, components, length, setting, etc.

2. How these standards were developed and what they were based upon.

3. How these standards are ultimately enforced.

4. Is there any other information related to program adaptations that you think we should know?

[Fidelity]

1. How does your state understand program fidelity (i.e., fidelity to the evidence-based program model or to planned program implementation)?

Probes: Do you view fidelity as the extent to which a program replicates the core components of an evidence-based model, the extent to which one implements a program as planned, or in some other way?

2. What are your state’s expectations of PREP-funded programs with respect to program fidelity?

3. Have you made your expectations about fidelity explicit? If so, how?

4. Does your state plan to monitor program fidelity? If so, how?

5. Does your state plan to provide training and technical assistance to the sub-awardees to ensure fidelity? If so, how?

6. Are you aware of any other existing efforts to monitor fidelity in the implementation of evidence-based programs targeting school-aged youth (including adolescents who are in non-school settings) in your state? If so, please describe these efforts.

7. Is there any other information related to program fidelity that you think we should know?

[Additional Information, Informants, and Follow-up]

1. Is there anything else that we should know about your state’s PREP program?

2. Is there anyone else who you think would be a good informant for this study (i.e. a descriptive study on how PREP programs were designed and implemented)? Could we have that person’s title and contact information?

3. We may want to follow-up with you in a year or so to see how the state’s PREP programs are being implemented. What is the best way to reach you? If we can’t reach you via this method, are there other people who will know how to reach you? Could we have their contact information?