SSB - APS Study Formative GenIC Prog Support

Alternative Supporting Statement Instructions for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

The Adulthood Preparation Subjects (APSs) Study of Dosage and Cultural Relevance

Formative Data Collections for ACF Program Support

0970 - 0531

Supporting Statement

Part B

May 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, DC 20201

Project Officers:

Caryn Blitz

Tia Brown

Part B

B1. Objectives

Study Objectives

The objective of this proposed data collection is to gather information from Personal Responsibility Education Program (PREP) grantees and providers about their experiences implementing Adulthood Preparation Subjects (APSs) topics. Specifically, the APS Study aims to gather information on what APS topics and content grantees and providers cover with youth and whether the APS topics and content align with the needs of youth served and are culturally relevant to youth. This formative evaluation seeks to address gaps in knowledge about the PREP grantees’ and providers’ experiences with delivering the APSs. The information collected through this study will be used to inform technical assistance planning and resources for PREP grantees related to the design and delivery of APS topics and content, as well as future learning agendas and research priorities related to on the PREP APSs.

The study research questions are described in Supporting Statement Part A, Section A2.

Generalizability of Results

This study is intended to present an internally valid description of the PREP APSs and is not meant to promote statistical generalization to other programs, sites, or service populations. Data collected under this generic information collection request will be used to refine program delivery strategies for the PREP APSs and assess the appropriateness of APS topics for PREP grantees.

Appropriateness of Study Design and Methods for Planned Uses

We will conduct multiple data collection activities for this study. We will conduct one-on-one telephone interviews with up to 20 grantees and providers and site visits with three grantees. The site visits will include interviews with program leaders and managers; interviews with program facilitators; focus groups with community members, including parents; and focus groups with youth participants in PREP programming.

• Telephone interviews: We will schedule the virtual interviews at times convenient for the grantee and provider respondents, recognizing the many competing demands that these respondents may face. Each interview will have one respondent, who will be the program director, grant administrator, or another staff member whom the grantee or provider feels could best answer the questions. The topic guide for the interviews (Instrument 1) will collect in-depth, qualitative data on the APSs that each grantee provides, focusing particularly on the dosage of APS topics and content provided, the degree to which these APS topics and content meet the needs of the youth served, the degree to which these APS topics and content are culturally relevant to the youth served, and any suggested improvements to the APSs. This design will enable grantees to provide nuanced feedback specific to the population of youth that they serve. This is important for the study, as the grantees and providers serve a wide variety of youth who may have unique needs and cultural backgrounds. Collecting qualitative feedback from a range of grantees and providers will enable the study team to better understand how well APS topics meet the needs of, and have cultural relevance for, the wide range of youth served by PREP programming.

• Site visits: We will conduct in-person site visits with three grantees. We will schedule the in-person interviews and focus groups at times convenient to the respondents. The topic guide for the site visits (Instrument 2) will guide the collection of in-depth qualitative data from various respondents.

  • We will conduct interviews with program leaders and managers (such as PREP grant administrators or program directors) and with facilitators of PREP programming. We will conduct the interviews with individual staff or in small groups, depending on the availability of staff. The interviews allow for collecting detailed information on the dosage of APS topics and content in their PREP programming, as well as on information on each APS topic (including those the grantee does not implement), how well the APS topics and content meet the needs of the specific youth served by the program, whether the APS topics and content are culturally relevant, and suggest improvements to the APS topics and content. Collecting detailed qualitative information from multiple respondents, across different program roles, at each grantee site will provide a fuller understanding of APS implementation at each site. Including staff with different perspectives is valuable to the study. For instance, the program director may know more about the grantee’s procedures for identifying that some APSs or APS topics do not meet the needs of the youth they serve (and thus the grantee does not implement them), while the facilitators may know more about the dosage of APS topics and content delivered and how well youth feel the APS topics and content meet their needs and are culturally relevant to them.

  • We will conduct focus groups with community members and youth. Focus groups provide a unique opportunity to collect information about respondent opinions and reflections, capturing respondent voices in a way that other data collection methods (such as survey instruments) cannot. For example, unlike surveys, the focus groups will enable us to delve deeper into community members’ or parents’ opinions on each APS and how well it meets the needs of the youth in their community. In addition, the focus group format will enable the study team to probe for in-depth information on whether APS topics and content are culturally relevant, and if not, ask for more details on potential revisions that could make the APS topics and content more culturally relevant for the youth served by PREP programming. During the focus groups, moderators will ask questions in a semi-structured way, providing opportunities for participants to ask clarifying questions and respond as appropriate. Moderators will also encourage participants to engage with others in the group.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The population of interest for this study is a subset of all organizations funded through the PREP grant program—including some Competitive PREP, State PREP, Tribal PREP, and Personal Responsibility Education Innovative Strategies (PREIS) grantees and their providers. We will prioritize organizations that are implementing innovative APS programming, have adapted APS for cultural relevance, or are serving subgroups of youth that may have particularly useful insights on cultural relevance—for example, those serving tribal and American Indian and Alaska Native (AI/AN) youth, Latino youth, youth living in rural areas, and youth living in child welfare or juvenile justice settings. Respondents of interest for the telephone interviews are the project directors, grantee administrators, or another staff member that the grantee or provider identifies as the most knowledgeable respondent.

For each of the three grantees selected for the site visits, we will collect information from program leaders and managers; facilitators of PREP programming; community members, including parents; and youth. The sampling frame for each site will be the staff that oversee the PREP programming; the roster of all facilitators providing PREP programming for the grantee; parents of youth participants in PREP programming and other community members who helped develop or adapt APS materials or collaborate with the PREP program in other ways; and youth that have participated in PREP programming.

The study team will use non-probability, purposive sampling to identify potential respondents who can provide information on the study’s key constructs. Because participants will be purposively selected, they will not be representative of the staff providing PREP programming or populations of families that the programs serve. Instead, we aim to obtain variation in experiences to understand how APS topics are viewed by various relevant parties.

Sampling and Site Selection

Telephone interviews: The study team – including the contractor, Mathematica, and ACF staff – will identify up to 20 grantees and providers to participate in the telephone interviews. This will be a combination of Competitive PREP, State PREP, Tribal PREP, and PREIS grantees. We will purposively select grantees and providers that are diverse in terms of how they address APSs and other grantee characteristics, such as the target population of youth served, the setting of program implementation, and geographic location. We will use data collected under the PREP performance measures (OMB #0970-0497; Exp Date 06/30/2023¹) to gather information on the grantees’ characteristics. Once the grantees and providers have been selected , we will invite them to complete the interview. (Appendix A includes recruitment and outreach materials for the telephone interviews.) Respondents involved in telephone interview data collection will be from a convenience sample; they may not be representative of all PREP grantee or provider staff.

Site visits: The study team will identify three grantees to participate in the site visits. We will purposively select grantees for the site visits that have experience or insights related to adapting APS materials or implementing innovative APS-related content. In so doing, we will also aim to select grantees that serve youth from different cultural backgrounds in order to gain a deeper understanding of the APS topics’ cultural relevance for various youth. We will select grantees from two or more of the PREP funding streams (Competitive PREP, State PREP, Tribal PREP, and PREIS grant programs) with at least one Tribal PREP grantee represented. We will also draw on information from the telephone interviews to identify potential grantees for the site visits, based on their current work related to the APS topics. Respondents involved in site visit data collection will be from a convenience sample; they may not be representative of all PREP grantees.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The study team developed two topic guides for this study (Instruments 1 and 2) to address four research questions that reflect the information needs for ACF regarding PREP grantees’ programming on APSs. Through the performance measures, PREP grantees answer two questions on APSs: (1) which APSs their programs cover; and (2) whether they cover APSs through their existing curriculum or supplement with additional content. These performance measures data provide high-level information on the prevalence of each APS and how it is covered, but they do not provide in-depth, contextual data on APS dosage nor on whether the APSs are culturally relevant and meet the needs of youth served. The topic guides therefore aim to collect data on four main topics: (1) dosage of APS topics and content delivered, (2) alignment of APS content with youths’ needs, (3) cultural relevance of APS topics and content to youth served (and whether this differs by subgroup of youth), and (4) potential adaptations to the APSs to improve them. The topic guide for the telephone interviews (Instrument 1) focuses on the APSs that the respondent is currently implementing, while the topic guide for the site visits covers all six APSs, whether or not the grantee is currently implementing them.

Through the topic guides, we will ask open-ended questions so respondents can provide rich qualitative responses about their experiences. Both topic guides were developed by Mathematica staff, in collaboration with ACF, and were designed to be concise and streamline data collection to only collect the information necessary to answer the research questions.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection.

Telephone interviews: A trained member of the study team will conduct an interview over the phone with up to 20 respondents, at a time convenient for the respondent. Data collection will occur on a rolling basis. Before data collection, the study team will send an email inviting grantee or provider staff to participate in an interview. (Appendix A includes outreach materials for the telephone interviews.) The invitation will explain the purpose of the interview and detail the data collection mode. After the potential respondent indicates their interest in participating in the study, we will send a second email with at least three potential times for the interview and ask the respondent to provide their availability. The contractor will follow up with those that do not respond to the outreach email to encourage their participation by highlighting the importance of the interviews. Federal project officers might also contact nonrespondents to encourage their participation.

Site visits: The study team will conduct three site visits in-person to the selected grantee organizations. We expect each site visit to last no longer than three days.

The site visits include four data collection activities: (1) interviews with program leaders and managers; (2) interviews with program facilitators; (3) focus groups with community members, including parents of youth participants in PREP; and (4) focus groups with youth participants in PREP. At each of the three site visits, we will interview up to three program leaders or managers (total of nine respondents), and we will interview up to five program facilitators (total of 15 respondents). We may have small groups of respondents complete the interviews at the same time, if that works better for scheduling purposes. At each site visit, we will hold a focus group with up to 10 community members (total of 30 respondents) and a focus group with up to 10 youth (total of 30 respondents). We will work with the grantee project director to identify the appropriate respondents for each data collection activity and will ask the project director to help schedule the interviews with the program leaders and managers, and program facilitators. We will also ask the project director to share information about the study with potential community member and youth respondents for the focus group. (Appendix B includes recruitment materials for focus groups.) Individuals interested in participating in the focus group can reach out to the grantee staff or the study team directly. Each youth that participates in a focus group will need to have consent form signed by their parent or guardian and to sign the youth assent form. (Appendices C and D include the parental consent form and youth assent form, respectively.) Parents and guardians will complete the form using the secure and private QuestionPro GovCloud application.

Table B.1. shows the total expected number of participants, the mode, and time commitment for each data collection activity. All five activities will be a one-time data collection request of the participants.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The telephone interviews and site visit interviews and focus groups are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias will not be calculated. Respondent type will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

Table B.1. APS Study: Data collection activities

Data collection	Administration plans
Telephone interview with PREP grantees and providers	Total participants	20
	Mode	Telephone
	Time	60 minutes
	Frequency	1
Site visit interview with program leaders and managers	Total participants	9
	Mode	In-person
	Time	90 minutes
	Frequency	1
Site visit interviews with program facilitators	Total participants	15
	Mode	In-person
	Time	60 minutes
	Frequency	1
Site visit focus groups with community members	Total participants	30
	Mode	In-person
	Time	60 minutes
	Frequency	1
Site visit focus groups with youth participants in PREP programming	Total participants	30
	Mode	In-person
	Time	60 minutes
	Frequency	1

B7. Data Handling and Analysis

Data Handling

Parents and guardians will complete the consent form electronically via QuestionPro GovCloud. The study team will send each parent or guardian an individualized email with a unique link, which they will follow to complete the consent form. The consent form will include PII (youth and parent/guardian names, and parent/guardian email address and phone number). QuestionPro GovCloud has protections for the security and privacy of all information entered into it. It is hosted in a FedRAMP compliant Gov Cloud environment, and it is also compliant with the General Data Protection Regulation (GDPR), Section 508, ISO 27001, California Consumer Privacy Act (CCPA), Payment Card Industry’s Data Security Standards (PCI - DSS), and FERPA. The PII entered into the form will be securely transmitted to the study team with encryption. The PII will be saved on a secure drive, separate from the data collected through the focus groups. For each telephone interview and site visit data collection activity, we will record each interview or focus group if we have respondents’ permission to do so.

We will create transcripts of each interview and focus group based on the recording (or if a recording is not available, based on the interviewer’s notes). Interviewers will review transcripts to fix spelling and grammar issues, fill in missing words, and explain unclear terms or phrases in preparation for qualitative coding and analysis. The recording and transcript from each interview and focus group will be saved on a secure drive accessible only to study team members.

Data Analysis

Qualitative data from the transcripts will be reviewed for overarching themes and lessons on each of the key topics explored through the virtual discussions. The study team will develop a coding scheme based on the research questions and topic guides. Moderators will apply the coding scheme to the transcripts and conduct a thematic analysis of the responses under each topic. The task leads will monitor coding and thematic analysis across the team to ensure accuracy and consistency.

Data Use

The data collected will be used to gather information on the dosage of APS topics and content delivered to youth enrolled in PREP programming as well as on whether the APS topics and content meet the needs of youth and are culturally relevant. The data will also be used to identify potential changes to improve the APS topics.

The findings from this formative data collection will inform the development of future PREP technical assistance planning and resources for grantees providing programming to youth on the APSs, as well as future ACF learning agendas and research priorities related to the PREP APSs. The primary purpose of the information collected is not publication, though findings may be incorporated into materials that are made publicly available. For example, to contextualize potential plans for a future study of APSs, ACF may reference the findings from this formative phase.

B8. Contact Persons

In Table B.2, we list the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Individuals Responsible for Study

Name	Affiliation	Email address
Caryn Blitz	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Caryn.Blitz@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Lauren Murphy	Mathematica	LMurphy@mathematica-mpr.com
Alicia Meckstroth	Mathematica	AMeckstroth@mathematica-mpr.com
Betsy Keating	Mathematica	EKeating@mathematica-mpr.com
Diana Gates	Mathematica	DGates@mathematica-mpr.com

Attachments

Appendices

Appendix A: Telephone Interview Outreach Materials

Appendix B: Recruitment Materials for Site Visit Focus Groups with Community Members and Youth

Appendix C: Parent and Guardian Consent Form for Youth Focus Groups

Appendix D: Youth Assent Form for Youth Focus Groups

Instruments

Instrument 1. Telephone Interview Topic Guide for PREP Grantees and Providers

Instrument 2. Site Visit Topic Guide for PREP Grantees

1 An extension request is currently in process for these measures with the 60-day comment period currently underway (88 FR 17576)

10