Interview Guide for SRAE Grantees

Form approved

OMB Control No:

Expiration Date:

SRAE National Descriptive Study
Early Implementation Study

Grantee Telephone Interview Protocol

DRAFT

April 2019

THE PAPERWORK REDUCTION ACT OF 1995

Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The information collected will help ACF understand decisions grantees make regarding the design of their SRAE-funded programs and how the programs are being implemented. The collection of this information is voluntary and responses will be kept private to the extent allowed by law. The OMB number for this information collection is 0970-XXXX and the expiration date is XX/XX/XXXX.

Grantee Telephone Interview Protocol

Note to reviewers: This guide is a pool of potential questions that will be drawn upon to tailor each interview. The questions will be selected and adapted based on the information that emerges from the document review and web survey. Each interview will be limited to one-and-a-half hours.

Introduction

Thank you for taking the time to answer questions about the [Title V] Sexual Risk Avoidance Education (SRAE) Program in your state/community!

As background, ACF’s SRAE program is a set of Federal grant programs to enable States or other local organizations to provide education exclusively on sexual risk avoidance, meaning voluntarily refraining from non-marital sexual activity and other risky behaviors.

The Administration for Children and Families (ACF), within the federal Department of Health and Human Services, oversees SRAE programs and is interested in learning about how SRAE-funded programs are implemented in the field and how selected SRAE-funded programs impact behavior and health outcomes for youth. Therefore, ACF has hired Mathematica Policy Research to conduct a national descriptive study of the SRAE program.

This interview will contribute to an Early Implementation Study, which will be a broad descriptive study of how SRAE grantees designed and implemented their programs. The data collected through this interview, along with the web survey you recently completed, will be reported in two ways. First, a summary profile will be created for each SRAE grantee that will contain facts about the SRAE grantee’s program design – for example, the characteristics of youth targeted to receive programming, the range of direct services youth receive, partnerships with community organizations, plans to address A-F requirements, and the outcomes anticipated for youth at program exit. While we will not attribute this factual information to a specific respondent, it will be attributed in aggregate to a specific state or competitive grantee. Second, the reports we produce will discuss cross-grantee themes emerging from responses regarding “how” and “why” SRAE grantees made decisions: for example, how grantees include parents in programming and why they chose particular curricula or program strategies.

Just so you know, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it has a currently valid OMB control number. The OMB number for this information collection is 0970-XXXX and the expiration date is XX/XX/XXXX.

Before we begin, I want to inform you of several important things about the interview:

• [ONLY APPLICABLE FOR STATE TITLE V SRAE GRANTEES: Your state/organization has agreed to participate in the SRAE national evaluation as a requirement of receiving a SRAE grant.] You are being asked to participate because it is a component of the national evaluation.

• These questions should take no more than one-and-a-half hours.

• I’m going to ask you questions in [five] areas [number and specific areas to be tailored for each interview]: (1) background information on existing sexual risk avoidance education programs in your state/community and how your state/agency became aware of and decided to apply for SRAE funding; (2) information on how your SRAE grant is being or is planning to be administered in your state/community; (3) information on how you designed your programming; (4) how you intend to evaluate your program, if at all; and (5) additional information, to help us if/when we want to follow-up with you.

• The information you provide will be used for research purposes to describe how SRAE is being implemented by your state/organization. Your responses will be used only to characterize the general experiences of your state/organization, without attribution to you or any other specific individual. If you do not know the answers to any questions and want to circle back with us, let us know. At the end of the interview, we’ll work with you to identify a time to speak again. If you want to defer to someone else on any question, let us know to whom we should reach out for those answers.

• An IRB has reviewed and approved this study. If you have any questions about your rights as a research participant, you can contact (name) at (name) IRB, toll free at (number). Questions about the overall study can be directed to (name, number).

Also, to help us capture all the information, we will be taking notes as we talk. And, if you’re comfortable, we would like to tape record the discussion. This recording will only be used for note-taking purpose and will not be shared beyond the site visit team, will be stored on a secure drive, and will be destroyed once the project is completed. Would that be ok?

Before we begin, do you have any questions, comments, or concerns?

So, first, I want to share definitions for a few common terms that can help with these questions:

• When I ask about a program, I mean the entire program funded by your SRAE grant. This includes not just the programming that youth receive, but also the staff and staffing structure, the administration, the processes to enroll youth and keep youth engaged, any performance measurement or research that occurs, and so forth.

• If I ask about programming, I am also referring to all of the activities and processes of the entire program funded by the SRAE grant.

• Then, if I talk about a program component, it would be any smaller portion of this program, such as the activities in which youth engage, the ways that classes are facilitated, the processes to enroll youth and keep them engaged, and so forth.

Module A: Background Information

[Context]

A1. Please describe your background/experience (with regard to the SRAE program and other similar programming) and current position.

Probes: What are your current responsibilities? How long have you been in your current position (# of months)? Is your position newly created to support the SRAE program in your state/organization? What certifications, degrees, or credentials do you hold that are relevant to your current position? How long have you been working on SRAE?

A2. Your Grantee Web Survey indicated that [FILL FROM B10] are the programs or activities aimed at educating youth about avoiding sexual risk that are currently operating in your state/community. Is that correct?

1. What proportion of the teen population, would you estimate, are receiving these services in your state/community? (Please respond, to the extent you can, for the state and/or community populations with which you are familiar).

1. Does your state/organization collect data on the number or proportion of youth who receive education on sexual risk avoidance? If so, who collects these data (for example, school)? Is your estimate based on these state/organization data, or some other means, such as observation or personal experience?

2. What are the characteristics of the population (for example, racial, ethnic, economic) receiving services?

2. In what settings do the programs operate? (Probe: For example, are they provided in faith-based organizations, community-based organizations, universities, and schools, and so forth?)

3. How are these (SRAE and similar) programs funded? Which programs are funded with federal dollars? State dollars? Non-profits, foundations, and other organizations?

4. How are these programs perceived by the communities they serve? (Probe: Are communities receptive/resistant to these programs?)

5. How well do you think these programs are implemented (for example, with regard to fidelity, achieving outcomes)? (Probe: Are there programs that you think are particularly well-implemented?)

6. How well do you think these programs and their activities achieve outcomes of youth avoiding non-marital sexual activity? (Probe: Are there programs that you think are particularly effective at helping youth achieve these outcomes? Are there particular program activities/strategies that you think are particularly effective at helping youth achieve these outcomes?)

7. To what extent do you think your perspective on these topics reflects the perspective of your state/organization in general?

[Approach/Strategy]

A3. How did your state/organization become aware of the SRAE funding and why did it decide to apply for SRAE funds? In what ways does the SRAE funding support or fill in gaps in the programming already operating in your state and/or communities?

A4. Was there one individual who really “led the charge” (for example, influenced decision makers) in seeking your SRAE grant and making it a reality? Who was this individual?

A5. How would you define the mission of the [Title V] SRAE grant program? What do you understand to be unique about this program’s purpose?

A6. In general, I’m interested in your perspective on teen sexual behavior and its associated risks, such as pregnancy and STIs, and other risky behavior among youth. Does your state/agency have an overall approach/strategy to reduce non-marital sexual activity and associated risk behaviors?

1. If so, could you please discuss your state/agency’s overall approach/strategy?

2. How does SRAE inform this approach/strategy?

3. How did your state/organization develop this overall approach/strategy? PROBE: Who makes decisions related to overall approach/strategy: you, your subgrantees, or someone else?

4. To what extent have you or are you drawing on particular frameworks or theories of behavior change to inform your program plans? Please describe. Did your state/organization develop a formal logic model to describe this approach/strategy?

Did your state/organization engage with any partners who played a key role in developing this approach/strategy? If yes, who and how were they chosen?

A7. Is there any other background information related to SRAE or your state’s/organization’s SRAE program approach/strategy that you think we should know?

Module B: Program Administration

[State Agency Structure/Administration]

B1. What is your SRAE program’s structure, and what led your state/organization to structure your SRAE program that way? Please describe how and why the state/organization arrived at the decisions you just described regarding the following:

1. SRAE program settings (number and type)

2. Specific target population(s) (if applicable)

3. The specific focus of the services or strategies provided

4. Number of youth intended to receive SRAE-funded services

5. The appropriate award amount to sub-awardees, or intended award amount (if applicable)

6. The number of sub-awards, or intended number of sub-awards (if applicable). [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] PROBE: Are you working with the same sub-awardees that you worked with under the Title V Abstinence Education program?

7. The stage of implementation in each setting/site (when did or when will program implementation begin)?

B2. Do you supplement with in-kind support or donated resources? If so, in what ways and for which part of your program do you supplement? What are the sources of these additional resources? (Do partner organizations provide any of the in-kind support or donated resources?)

B3. Is there any other information related to your state agency, administering structure, or the grant award process that you think we should know?

[State/Organization and Subawardee Relationship, if applicable]

B4. What is the relationship between your state/organization and the subawardees/local administering entities?

1. How were the subawardees identified and selected? [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] PROBE: Have you worked with subawardee in the past?

2. What challenges, if any, did you face in identifying and recruiting subawardees/local administering entities?

PROBE: What steps, if any, did you take to market the SRAE program to potential subawardees and recruit them to participate? Did you explain the unique mission and requirements of the SRAE program? [May be particularly relevant for states/communities with experience implementing comprehensive sex education programs.]

3. What process have you established for monitoring and overseeing the subawardees?

PROBE: How closely is your state/organization overseeing the SRAE program working with its sub-awardees/local administering entities? How often do your state/organization and local entities communicate?

B5. Please identify:

1. The requirements from your state/organization that sub-awardees/local administering entities must fulfill

2. Assurances they are required to certify as sub-awardees (for example, having to do with separation of religious activities, Americans with Disabilities Act (ADA), non-discrimination, anti-bullying/harassment, and/or anti-stigma regulations)

3. The guidance provided by your state/organization to sub-awardees regarding the SRAE Program and its federally mandated requirements, and specifically the A-F requirements.

4. Data (for example performance data), if any, that sub-awardees/local administering entities are required to report to the state administering agency. How, if at all, have delays in finalizing the federal performance measures affected the collection of data required from sub-awardees/local administering entities?

B6. Are your state/organization or any of the sub-awardees serving vulnerable youth (for example, racial minorities, religious minorities, homeless and runaway youth, LGBT youth)?

1. If so, what types of vulnerable youth?

2. Is there a specific mechanism to ensure that needs of the vulnerable youth are adequately addressed? Can you describe it?

B7. Is there any other information related to your state/organization and sub-awardee relationship that you think we should know?

[Organization Monitoring of Program Implementation Sites, if applicable]

B8. Please describe the relationship between your organization and the implementation site(s).

PROBE: Who (job title) at your organization is responsible for maintaining this relationship? How closely is your organization working with the implementation site(s)? How often do your organization and the implementation sites communicate?

B9. Please identify:

1. The expectations of implementation sites. What have they agreed to provide? (ex: health educators, staff to monitor, space, transportation, contact information)

2. Whether the roles and responsibilities of the implementation site(s) are outlined in an MOU or letter of commitment?

3. Who the key stakeholders are at your implementation sites?

4. The main point of contact at the implementation site(s) for your program. What is their role in the project?

5. Other persons you work with at the implementation site(s)? What are their roles in the project?

6. Whether anyone at the implementation site(s) assists with youth recruitment and retention activities?

7. Whether anyone at the implementation site(s) assists with attendance, data collection, or providing information to help track youth? If so, please describe their role.

8. Whether anyone at the implementation site(s) make decisions related to program goals, objectives, and primary messages, including alignment with SRAE program requirements? If so, please describe.

9. How do you monitor implementation site(s) to ensure they are meeting expectations regarding roles and responsibilities?

10. Whether you monitor implementation site(s) to ensure programming fulfills the federal requirements and mission of SRAE?

11. Do you have a process in place if there is an issue with a site that is not meeting expectations? If so, please describe.

B10. Is there any other information related to your organization and the implementation site(s) relationship that you think we should know? PROBE: What are the most common challenges you have faced in working with the implementation site(s)?

[Partnerships and Collaborations]

B11. How are local community groups, partners, or other agencies/organizations involved in the SRAE program (for example, in its administration, implementation, or in some other way)? To your knowledge, how has the SRAE program been received by local community stakeholders?

PROBE: [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] Has the involvement with community groups, partners, and local agencies changed since the Title V Abstinence Education program?

PROBE: [For states/communities with experience implementing comprehensive sex education programs] What challenges, if any, did you face in familiarizing community groups, partners, and local agencies with the SRAE program? What steps did you take to market the program to them and gain their support and/or promote their involvement?

B12. Are there collaborations and workgroups that support the SRAE program that you are already involved with or are looking to become involved with over the course of this grant? Through this grant, do you have plans to start a collaboration or workgroup in your area?

PROBE: [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] Have your community partners changed since the Title V Abstinence Education program?

Module C: Program Design

[Guiding Considerations]

C1. Please describe broadly, from the beginning, the process that your state/organization went through in order to (1) select or design programming to be implemented with SRAE funds and/or (2) to provide guidance to sub-awardees related to selecting or designing programming to be implemented with SRAE funds.

PROBE: What factors influenced your state’s/organization’s and/or sub-awardees’ selection of particular program interventions, curricula, or specific program components or activities?

[Program Model Selection and Design]

C2. Can you tell me what you know about the existing evidence of effectiveness for your intervention or for its curricula, strategies, or approach?

1. PROBES: That is, do you know which research studies or sources have shown that the intervention or its approach lead to positive impacts or outcomes for youth, families, or communities? If so, what types of positive impacts or outcomes did the research show? Do you know which curricula, components, or aspects of your program have evidence of effectiveness and which do not?

2. Did your state/organization deem any program component inappropriate for your program, based on the research? Why or why not?

C3. [ASK BASED ON RESPONSE TO WEB SURVEY C82] Your Grantee Web Survey indicated that you are supplementing your program plan with some supplemental materials, such as modules or activities. Is that right?

1. What supplemental modules or lessons are being added and what do they address above and beyond the original program?

2. What supplemental activities are being added and what do they address above and beyond the original program?

3. Are these supplemental materials being added to address the A-F requirements, and if so, in what ways?

C4. How are you addressing each of the A-F requirements through your program components or supplemental materials? [Interviewer will ask and record answer for each of the A-F topics] Who is making these decisions; are you making these decisions, or are your sub-grantees making these decisions?

If the respondent is unsure about the requirements, the A-F topics are:

1. The holistic individual and societal benefits associated with personal responsibility, self-regulation, goal setting, healthy decision-making, and a focus on the future

2. The advantage of refraining from non-marital sexual activity in order to improve the future prospects and physical and emotional health of youth

3. The increased likelihood of avoiding poverty when youth attain self-sufficiency and emotional maturity before engaging in sexual activity

4. The foundational components of healthy relationships and their impact on the formation of healthy marriages and safe and stable families

5. How other youth risk behaviors, such as drug and alcohol usage, increase the risk for teen sex

6. How to resist and avoid, and receive help regarding, sexual coercion and dating violence, recognizing that even with consent teen sex remains a youth risk behavior

C5. How do your program components or supplemental materials incorporate elements of Positive Youth Development (PYD) into their approach?

C6. [ASK BASED ON RESPONSE TO WEB SURVEY C102/103] [grantees implementing the option to include information on contraception.] How do your program components or supplemental materials discuss information about contraception? How did your organization make decisions about what type of information to provide related to contraception?

C7. How do your program components involve parents, if at all? How did your organization make decisions about how to involve parents?

C8. How, and by whom, were final decisions related to the program/curriculum of SRAE program made?

C9. [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] How have you changed or adapted your program components in response to the requirements of the Title V SRAE grant (specifically the A-F requirements)?

C10. Is there any other information related to program component selection or design that you think we should know?

[Adaptations]

C11. Did you work with your providers to make any adaptations to ensure that your program addresses the grant requirements and expectations? Did you give your subawardee providers the opportunity to make additional adaptations on their own?

C12. Did your state/organization create a process for approving/rejecting program component adaptations? If so, what is this process? When did this process get put into place, and [for State Title V SRAE grantees who were former Title V Abstinence Education grantees] how has it changed in response to the Title V SRAE grant requirements?

C13. Specifically, did the state/organization create standards regarding adaptations having to do with:

1. A-F topics

As a reminder if helpful, the A-F topics are:

1. The holistic individual and societal benefits associated with personal responsibility, self-regulation, goal setting, healthy decision-making, and a focus on the future

2. The advantage of refraining from non-marital sexual activity in order to improve the future prospects and physical and emotional health of youth

3. The increased likelihood of avoiding poverty when youth attain self-sufficiency and emotional maturity before engaging in sexual activity

4. The foundational components of healthy relationships and their impact on the formation of healthy marriages and safe and stable families

5. How other youth risk behaviors, such as drug and alcohol usage, increase the risk for teen sex

6. How to resist and avoid, and receive help regarding, sexual coercion and dating violence, recognizing that even with consent teen sex remains a youth risk behavior

2. Cultural appropriateness

1. Which aspects of culturally-appropriateness were addressed? For example, were linguistic adaptations addressed? Were program content adaptations addressed?

3. Age appropriateness

4. Medical accuracy and completeness. Only after respondent responds, then probe:

   1. Have you balanced discussions about medical accuracy with discussions about the advantage of refraining from nonmarital sexual activity in order to improve the future prospects and physical and emotional health of youth? If so, what standards have you developed regarding adaptations to address refraining from nonmarital sexual activity?

   2. [for those who have chosen the option of discussing contraception] What standards have you developed to assess whether the programs provide the complete and medically accurate information that contraception reduces risk and does not eliminate risk?

5. Other aspects of the program, including dosage (length of sessions, overall program), intensity (time spent receiving intervention and completing any associated components), specific program components, setting, and so forth.

C14. Based on the previous question, could you please identify:

1. The standards created regarding adaptations having to do with:

1. A-F topics

2. Cultural appropriateness

3. Age appropriateness

4. Medical accuracy

5. Other aspects of the program, including dosage, length, setting, and so forth

2. How and when these standards were developed and what they were based upon.

3. How these standards are ultimately enforced.

C15. Is there any other information related to program adaptations that you think we should know?

[Staff Hiring and Training]

C16. What qualities and characteristics do you require and look for when hiring staff (specifically program facilitators)?

1. [State Title V SRAE grantees who were former Title V Abstinence Education grantees] How have you changed or adapted your staff hiring approach in response to the requirements of the Title V SRAE grant?

C17. What training and ongoing assistance do you provide to staff? Please describe training certifications, supplemental trainings, and ongoing support provided. How do you ensure that staff are prepared to deliver the program to meet the SRAE expectations and meet other job requirements?

1. [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] How have you changed or adapted your staff training approach in response to the requirements of the Title V SRAE grant?

[Fidelity]

C18. How is program fidelity defined in your state/organization (for example, fidelity to the program curriculum, or to planned implementation of a program component or activity)?

PROBE: Do you view fidelity as the extent to which your program replicates the program curriculum, the extent to which it implements a program component or activity as planned, the extent to which it meets the federal requirements and mission of SRAE, or in some other way?

C19. How do you measure program fidelity? Please describe who is involved, what the methods are, what you specifically look for, and the frequency of performance monitoring and/or evaluation. (Probe for whether the federal A-F requirements are part of the monitoring plan.)

1. [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] How, if at all, did your fidelity monitoring process change based on the Title V SRAE grant requirements?

C20. What are your state’s/organization’s expectations of your SRAE program with respect to program fidelity?

1. Have you made your expectations about fidelity explicit to subawardees, implementation sites, and/or program providers? If so, how?

C21. Is there any other information related to program fidelity that you think we should know?

Module D: Evaluation

[Tracking Outcomes]

D1. What outcomes do you expect to achieve through your SRAE program (and in particular, through this grant)? In other words, to what kinds of outcomes would you hold your organization accountable?

1. [For State Title V SRAE grantees who were former Title V Abstinence Education grantees] How, if at all, did your program outcomes change based on the Title V SRAE grant requirements?

D2. Do you plan to track your program’s outcomes, to determine whether you have achieved your outcomes? If so, please broadly describe how you intend to track these outcomes and collect the data needed to measure them. How often will you check to determine achievement of outcomes?

D3. Do you plan to implement a continuous quality improvement (CQI) system? If so, please broadly describe this system.

[Local Evaluation]

D4. Do you intend to conduct any evaluation activities or a grantee-specific evaluation, sometimes called a local evaluation?

If no: PROBE: Why did you decide not to conduct an evaluation? How, if at all, did the 20 percent set aside rule influence your decision (for example, did the rule act as a disincentive to conducting an evaluation?)

D5. If yes, please broadly describe what type of research activities you plan on conducting.

D6. What are the primary research questions you would like to answer through these evaluation activities?

D7. Who are the staff or partners that will assist you with evaluation activities? How did you select these staff or partners?

D8. Please broadly describe your research design. (PROBE: Will this be an impact evaluation, or a descriptive evaluation, or a needs assessment, or something else?)

D9. What is the rough timeline for your evaluation?

D10. Why did you choose to conduct an evaluation? What influenced your decision to conduct these evaluation activities?

PROBE: What proportion of your grant funds are allocated to evaluation activities? How, if at all, did the 20 percent set aside rule influence your decisions related to whether to conduct an evaluation and how to structure an evaluation (for example, did the rule act as either an incentive or disincentive?)

D11. Do you have outside support in helping to fund evaluation activities?

D12. Is there anything else we should know about your local evaluation?

[Federally-led Evaluation]

D13. Is there anything interesting in your program that you think could be evaluated as part of another evaluation, such as one led by the federal government?

D14. Would you be interested in participating in a federally-led evaluation? Note that federally-led evaluations are not just trials of program effectiveness, but may also involve implementation documentation, qualitative interviews, testing smaller components of your overall program, or other kinds of evaluation.

Module E: Additional Information, Informants, and Follow-up

E1. Is there anything else that we should know about your state’s/organization’s SRAE program (or evaluation)?

E2. [IF NEEDED] You mentioned you wanted to look into [TOPIC] some more and circle back to us. When should we reach out to you again?

E3. Is there anyone else who you think would be a good informant for this study? Could we have that person’s title and contact information?

E4. We may want to follow-up with you in a year or so to understand your SRAE implementation experiences. What is the best way to reach you?