SSA - Measuring Self- and Co-Regulation in SRAE Programs – Part 2

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs

Pre-testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part A

August 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Tia Brown

Kathleen McCoy

MeGan Hill

Nakia Martin-Wright

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (0970-0355).

• Progress to Date: This request builds on part 1 of the Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs approved under the same umbrella generic Pre-testing of Evaluation Data Collection Activities (0970-0355) in April 2023. During that initial pilot, the study team developed and conducted a pretest of a Youth Self-Assessment Survey (titled “How I Feel and What I Do”). The instrument was piloted with a total of 66 youth participating in five Sexual Risk Avoidance Education (SRAE) programs, with the goals to (1) assess validity and reliability of the survey questions, (2) determine whether the survey questions could detect change between pre- and post- SRAE program, (3) examine which survey items worked best, and (4) assess whether the survey items measured one construct or multiple constructs. Based on findings from part 1 of the pre-test and additional internal discussions about the goals of the survey, we refined the survey to revise the wording of items and add domains covering knowledge of SRAE topics; perceptions of the classroom environment; perceptions of the facilitator-youth relationship; and knowledge of the skills reinforced through the classroom co-regulation strategies. These topic areas were selected because they cover the components of programming that may Influence youth’s outcomes in areas of self-regulation and sexual risk avoidance.¹

• Description of Request: This information collection request is for further piloting of the revised and expanded survey items. The proposed pretest includes three phases: 1) a series of small group cognitive interviews in which youth will take the Youth Self-Assessment Survey to explore how youth understand the survey items and the relevance of the language and examples used; 2) revising the Youth Self-Assessment Survey, based on findings from the cognitive interviews to then administer the updated version of the survey to a large, diverse sample to examine the survey’s reliability and validity, and 3) revising the Youth Self-Assessment Survey based on findings from the phase 2 administration to conduct a pre-post administration of the survey among SRAE program participants to evaluate the sensitivity of the survey to detect change between the two data collection points. The findings from these data collections will inform revisions and finalization of a new survey instrument on youth perceptions of their self-regulation and the co-regulation in their SRAE classroom.

A new survey containing these measures is critical to ACF’s self- and co-regulation learning agenda² and will contribute to the body of knowledge on ACF program design and intended outcomes. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

In 2013, the Office of Planning, Research, and Evaluation (OPRE) of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), began developing an important research agenda focused on Self-Regulation. This agenda generated a set of foundational resources and contributed to theory and practice.³ Self-Regulation, defined as a developmental process that includes executive function, self-control, and regulation of emotions, is an important skill that impacts the quality of life for children and their families. This work has grown over the past decade encompassing the idea that youth develop self-regulation through co-regulation, “the interactive process by which caring adults (1) provide warm, supportive relationships, (2) promote self-regulation through coaching, modeling, and feedback, and (3) structure supportive environments.”¹ Concepts related to self- and co-regulation are found in many programs that ACF administers and evaluates.

This generic information collection request supports the development and testing of a survey instrument to assess youth’s perceptions of co-regulation in the classroom and their self-regulation skills as well as youth’s understanding of SRAE topics. This pre-test data collection would support ACF’s goal of prioritizing the development of a reliable and valid survey instrument. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency. ACF has contracted with Mathematica to conduct this work.

A2. Purpose

Purpose and Use

This proposed information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures. Data collected under this generic information collection request will be used to develop a survey that captures youth perceptions of self-regulation and co-regulation and their understanding of SRAE topics, and if found to be valid and reliable through pre-testing, would advance ACF’s learning agenda. In addition to broad use in the field, the final Youth Self-Assessment Survey may be used in ACF evaluations of self- and co-regulation strategies.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

ACF proposes the following guiding questions for this pre-test:

• Do the youth respondents seem to easily understand the questions and response options?

• Are the survey items valid and reliable for measuring youth’s perceptions of self- and co-regulation?

• How long does it take youth to complete the survey items?

• Are the survey items sensitive in detecting change over time?

Study Design

This survey pre-test includes three phases. Each phase involves recruitment, as described below.

• Phase 1: small-group cognitive interviews that include administering the Youth Self-Assessment Survey Version A

• Phase 2: administrating the revised Youth Self-Assessment Survey Version B at one point in time

• Phase 3: administering the revised Youth Self-Assessment Survey Version C in a pre- and post-program data collection design

Phase 1. The study team will recruit study participants for phase 1 through two sources. The first is a market research vendor. The vendor has a large group of panel members across the U.S., from various geographical areas and with a wide range of demographic characteristics. To recruit, we will use a screening instrument called the Youth Screener for Cognitive Interviews and Pilot Survey (Instrument 1). To be eligible to participate, youth need to be in grades 9-12 and have participated in a sexual education and/or healthy relationship class within the current or prior school year. We will aim to recruit a mix of students – including variation in gender; younger (9th and 10th graders) and older students (11th and 12th graders); and students from a diverse range of racial and ethnic backgrounds. We will also recruit youth who have participated in a Sexual Risk Avoidance Education (SRAE) program. We will work directly with the Family and Youth Services Bureau (FYSB) to recruit these youth.

The study team will conduct up to 10 small-group cognitive interviews (using the Cognitive Interview Protocol, Instrument 2), with each group, comprised of 3-4 youth. We will use the cognitive interviews to explore how youth are interpreting the items on the Youth Self-Assessment Survey (Instrument 3, version A), including how they understand and operationalize self-regulation -and co-regulation. This information will help to further refine items by ensuring we include terminology that youth relate to and, when appropriate, situate the skills we are asking about in salient social situations for youth. We aim to interview up to 32 youth across the 10 groups.

Phase 2. The study team will revise the Youth Self-Assessment Survey (Instrument 3, version A), as needed, based on the cognitive interviews. Then, as with the phase 1 recruitment, we will use the Youth Screener for Cognitive Interviews and Pilot Survey (Instrument 1) to determine eligibility for the survey. We will prioritize recruiting students that vary based on geographic location, age/grade level, gender, race, and ethnicity. The study team will again work with the market research vendor to administer the Youth Self-Assessment Survey (Instrument 3, version B), at one point in time with a diverse sample of 350 high-school students who currently attend a public or private high school. This will allow the study team to assess the reliability and validity of the survey items quickly and efficiently, with a diverse sample of youth.

Phase 3. In the third, and final, pre-testing phase, the study team will analyze data from the second phase and revise the Youth Self-Assessment Survey (Instrument 3, version B) as needed, such as dropping poorly performing items. We will then administer the revised Youth Self-Assessment Survey (Instrument 3, version C) to youth who participate in an ACF-funded SRAE program that teaches Love Notes. This phase will include up to three SRAE programs who are teaching Love Notes in high schools. The study team will recruit up to 250 youth participating in SRAE offered by the participating programs. We will work with each of programs to recruit at the classroom level, requiring that all programs are compliant with administering the performance measures as entrance and exit tickets and that the data collection activities for this study are acceptable to the IRBs’ overseeing program-level youth data collection. All youth in all selected classrooms will be eligible to take the revised Youth Self-Assessment Survey (Instrument 3, version C).

The purpose of this phase is to assess whether the survey instrument can detect changes in youth’s perceptions of their self-regulation skills and the co-regulation in their classroom. The survey items will be administered before the program and again after the program. To minimize the burden on staff and youth, the items will be administered at the same time youth complete the required SRAE entrance and exit performance measure surveys.⁴ The study team will use results from the final phase of the pre-test to further refine the Youth Self-Assessment Survey (Instrument 3, version C); results are not intended to be representative of, or generalizable to, a given subpopulation.

Table A.1 summarizes the study design, including the data collection instruments, their content and respondent types, and the mode and duration of each data collection activity.

Table A.1. Study design summary

Data collection activity	Instruments	Respondent, content, purpose of collection	Mode and duration
Participant screener for Phase 1 and Phase 2	Instrument 1: Youth Screener for Cognitive Interviews and Pilot Survey	Respondents: 450 youth, ages 14–19. Content: Questions about respondents’ demographic characteristics, such as age/grade level, gender, race, ethnicity, and state of residence Purpose: To assess eligibility and interest in participating in phases 1 and 2 of the pre-test; to ensure participants meet a range of demographic characteristics, and to provide demographic information about the study participants.	Mode: Web-based survey Duration: 5 minutes
Phase 1. Cognitive interviews	Instrument 2: Cognitive Interview Protocol Instrument 3, version A:5 Youth Self-Assessment Survey	Respondents: 32 youth, ages 14–19. Content: Questions about self-regulation (e.g., goal setting, inhibition, and controlling emotions) and about youth perceptions of the co-regulation in their classroom and understanding of SRAE topics. Purpose: To inform revisions to a survey to assess youth’s self-regulation skills and their perceptions of co-regulation in their classroom and understanding of SRAE topics.	Mode: In-person or virtual small groups Duration: 90 minutes
Phase 2 and 3. Youth survey	Instrument 3, version B: Youth Self-Assessment Survey Instrument 3, version C: Youth Self-Assessment Survey	Respondents: 600 youth (350 youth for phase 2; 250 youth for phase 3), ages 14–19. Content: Questions about self-regulation (e.g., goal setting, inhibition, and controlling emotions) and about youth perceptions of the co-regulation in their classroom and understanding of SRAE topics. Purpose: To gather information about the psychometrics and sensitivity to change of a survey of youth perceptions of self-regulation skills and the co-regulation in their classroom and understanding of SRAE topics.	Mode: Paper or web-based Duration:10 minutes

Other Data Sources and Uses of Information

Findings from the pre-test data collection will inform the development of a new survey instrument for youth perception of their self-regulation skills and co-regulation in the classroom, pre- and post-program. This work builds on pre-testing started under an earlier information collection approved under this umbrella clearance⁶.

A3. Use of Information Technology to Reduce Burden

In phase 1, the study team will conduct the cognitive interviews in-person or via a video conferencing software, such as Zoom. After obtaining permission from each participant, the Mathematica study team will audio-record all in-person cognitive interview groups and video and audio-record all virtual groups. These recordings ensure that information is accurately captured. The Mathematica study team will use the recordings to supplement the notes taken during the cognitive interviews. The video recordings will also allow for efficient transcript production, as draft transcripts can be automatically produced following the recording of these groups, which the Mathematica study team will review for accuracy. For phase 2 of the pre-testing, the revised version of the Youth Self-Assessment Survey (Instrument 3, version B) will be a web-based survey administered through the market research vendor.

In phase 3, the Youth Self-Assessment Survey (Instrument 3, version C), again revised based on findings from phase 2, will be administered either by web or on paper, depending on the preferences and practices of each grant recipient. This flexibility in survey delivery methods accommodates different administrative contexts, thereby reducing burden on grant recipients.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team has scanned available measures of self- and co-regulation and identified a set of items aligned with ACF’s research priorities. The Youth Self-Assessment Survey (Instrument 3, version A) incorporates items from measures used in other contexts. Some items will be adapted to meet the needs of this study, and the full survey instrument developed for this pre-test will be new.

A5. Impact on Small Businesses

The programs participating in phase 3 of the study will be small, nonprofit organizations. The study team will request information required only for the intended use. The pre- and post-test data collection will be scheduled to be administered in conjunction with the SRAE performance measures entrance and exit surveys⁷ for youth to help reduce burden. We will also take a flexible approach in case some grant recipients prefer a different process. For example, in some cases, grant recipients might prefer for their facilitators to administer the survey outside of this performance measures time and instead oversee the data collection during a normal course period. Facilitators will not be asked to work additional time to complete data collection activities.

A6. Consequences of Less Frequent Collection

This is a one-time data collection with three phases.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published on January 5, 2021; Volume 86, Number 2, page 308, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announcing the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published on May 21, 2021; Volume 86, Number 97, page 27624, and provided a thirty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

During work on the initial Youth Self-Assessment Survey (information collection approved under this umbrella generic), the study team engaged with two external experts to inform development of the survey items: Desiree Murray (senior research scientist at the Center for Health Promotion and Disease Prevention at the University of North Carolina) and Velma McBride Murray (professor, Peabody College and School of Medicine, Vanderbilt University). During work on these current phases of data collection, we will continue to consult with experts who specialize in measurement, youth self-regulation, and co-regulation⁸.

A9. Tokens of Appreciation

To recognize the value of the input provided by participants, help to remove barriers to participation, and affirm that contributions from those with lived experience are as valuable as those from other experts, we plan to provide honoraria to youth participants in phase 1 and 2, as described in section A13.

In addition, in phase 1, the parents/caregivers of cognitive interview participants will receive tokens of appreciation of $50 for providing transportation to and from the focus group facility for in-person groups, aimed at increasing youth participation and mitigating financial and time-constraint barriers that socioeconomically disadvantaged families may face.⁹ The token of appreciation offered is necessary to attract a diverse group of participants, critical to obtaining feedback from youth with a range of experiences and opinions related to self-regulation. Without this token of appreciation there is a risk of excluding those whose families might find it financially challenging to participate, skewing the study participants and undermining the objectives of the pre-test. Thus, the token of appreciation also serves as a strategic approach to ensuring inclusivity and increasing the robustness of the analytic findings.

We do not propose offering tokens of appreciation or honoraria to youth in phase 3. These youth will take the Youth Self-Assessment Survey (Instrument 3, version C) in the classroom at the same time they complete surveys for the performance measures. This will occur during their regular school day, as part of the youth’s SRAE class, and it is standard practice not to provide tokens of appreciation or honoraria for data collection that take place during regular school hours.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

In phases 1 and 2, for the participants recruited from the market research vendor, the vendor maintains all personally identifiable information (PII) for their panel members. This information will not be shared directly with the study team. For phase 1, the vendor will share the demographics collected from the Youth Screener for Cognitive Interviews and Pilot Survey (Instrument 1) needed for recruitment with the study team, but this will not include any other identifying information about participants. At the time of the cognitive interview groups, the vendor will provide a list of first names of participants, without any other identifying information. In phase 1, we will also recruit teens in collaboration with FYSB who have participated in a SRAE program. To conduct that recruitment, FYSB may need to provide first names and email addresses to the contractor so that they can contact youth for cognitive interview scheduling. Following the interviews, all names and contact information will be destroyed. Data collected from the interviews will not be linkable by the PII used to schedule the interviews.

For phase 2, the market research vendor will send the study-eligible and selected panel participants an email with the link to the web-based Youth Self-Assessment Survey (Instrument 3, version B). When data collection is complete, the vendor will provide the study team with de-identified demographic information on each youth that completes the Youth Self-Assessment Survey (Instrument 3, version B), which may allow us to run sub-group analyses if we have sufficient sub-group sample sizes.

For phase 3, the youth self-assessment will not collect PII. To track responses between the pre-program and post-program Youth Self-Assessment Survey (Instrument 3, version C), where possible, we will use pre-labeled paper surveys with an identification number. If this is not possible for sites to manage (due to needing to track who gets which survey at post-test), we will use an alternate approach. As was done in the initial pilot, we will use a series of questions at the start of the pre-program survey to develop unique self-generated identification codes. The same questions will be asked in the post-program survey to link responses without using PII. This self-generated matching method is not completely reliable, so will be used as a backup.

In programs that require active or passive consent, the study team will work with the site to adapt their site consent form (as needed) to also include the Youth Self-Assessment Survey (Instrument 3, version C) for this pilot data collection. Sites will manage the consent form process, and youth and parent names will not be linkable to the pilot data set.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

For the cognitive interviews, youth, and their parents or guardians, will be informed that groups will be audio-recorded for notetaking prior to the start of the group, and that they may opt out of these recordings (see Appendix B. Participant Consent). Groups will only be recorded if all participants in the group agree to being recorded. The cognitive interview consent forms also include language explaining the unique confidentiality risks associated with participating in a group interview. We will wait to begin recording the discussion until after everyone has introduced themselves. The transcribed notes will not include any names. All notes and audio recordings will be stored on Mathematica’s secure network. No one outside the study team will have access to the data. Only Mathematica staff working directly on this project have access to the project folder on the network where recordings will be saved. All audio recordings will be destroyed as soon as they have been transcribed and notes will be destroyed per contract requirements.

The study will be reviewed by Mathematica’s Institutional Review Board (IRB), the Health Media Lab, and the IRBs overseeing the data collection of the selected programs. Outreach and data collection will not begin until the project has received IRB approval.

Data Security and Monitoring

As specified in the contract, the contractor shall protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. The contractor has developed a data security plan that assesses all protections of respondents’ PII. The contractor will ensure all employees and subcontractors (at all tiers), including employees of each subcontractor who perform work under this contract and subcontract, receive training on data privacy issues and comply with all requirements. All Mathematica staff must sign an agreement to (1) maintain the privacy of any information from individuals, businesses, organizations, or families participating in any projects conducted by Mathematica; (2) complete online security awareness training when they are hired; and (3) participate in a refresher training annually.

As specified in the evaluator’s contract, the contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the standard. The contractor will incorporate the standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing risk, to the extent possible including sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive information to ensure secure storage and limits on access.

No information will be given to anyone outside the study team and ACF.

A11. Sensitive Information ¹⁰

A key goal of SRAE programs is to provide youth with age-appropriate strategies, skills, messages, and discussion on topics of sexual risk avoidance and teen pregnancy prevention education. As such, the survey contains a minimal number of items pertaining to youth’s personal opinions regarding scenarios that they think are appropriate for having sex or waiting to have sex, as well as a question on their comfort level speaking with a parent or caregiver about sex, pressure they feel from their friends to have sex, and questions on their perceived ability to say no to having sex or using alcohol or drugs if they didn’t want to. This feedback will allow ACF to develop a survey that is relevant to this population and designed in a way that meets the needs of both the SRAE program and the youth that are served.

All pre-test participants will be informed that they can refuse to answer any questions they feel uncomfortable answering. In phases 1 and 2, this will be noted on the parent consent form, the youth assent form, and at the start of the Youth Self-Assessment Survey (Instrument 3, versions A and B). In phase 1, the discussion group moderator will reiterate this verbally at different points throughout the session, as scripted in the Youth Self-Assessment Survey (Instrument 3, version A) and in the Cognitive Interview Protocol (Instrument 2). Additionally, phase 1 participants will be informed that the goal of the pre-test is about their understanding of the questions, not to capture their specific answers.

As mentioned in Section A10, respondents in all phases of the pre-test will be instructed to not record their name or any other identifying information on the survey instrument, which will prevent sensitive information in the survey from being linked to individual respondents.

All data collection instruments, and study materials will be reviewed by the contractor’s IRB, Health Media Lab.

A12. Burden

Explanation of Burden Estimates

In Table A.2 we summarize the estimated reporting burden and costs for each instrument. The survey estimates include time for respondents to review the instructions and to complete and review their responses. The following provides an overview of burden estimates:

• Instrument 1: Youth Screener for Cognitive Interviews and Pilot Survey. For phases 1 and 2 of the pre-test, a 5-minute web-based demographic survey will be administered to the market research vendor’s panelists. The survey will be completed once for potential cognitive interview participants and once for potential survey participants in phase 2, to determine study eligibility. We estimate it will be completed with up to 450 youth total, assuming about 80% are determined eligible.

• Instrument 2: Cognitive Interview Protocol. This protocol will be used during the cognitive interviews with up to 32 youth recruited from the market research vendor and SRAE programs. Each cognitive interview group will last up to 90 minutes.

• Instrument 3: Youth Self-Assessment Survey. Using version A, we will administer the survey along with the cognitive interview protocol as part of cognitive interviews with up to 32 youth recruited from the market research vendor and SRAE programs. Each cognitive interview group will last up to 90 minutes. Then, using version B, we will administer the survey one-time to up to 350 youth recruited through the market research vendor’s panel. The survey will be completed by web and is expected to take approximately 10 minutes. Finally, using version C, we will administer the survey with up to 250 youth participating in SRAE programming. This survey will be administered just before youth start the SRAE program (pre), and then again at the end of the SRAE program (post). The time between the pre- and post-survey ranges between 2 and 10 weeks. We anticipate that overall, 80 percent of the youth will complete version C at both time points. We will work with each site to ensure that absent students participate, thereby maximizing the completion rates, however we assume that 20 percent of the youth will be missing either a pre- or post-survey. The survey will be completed on paper or by web and is expected to take approximately 10 minutes.

Estimated Annualized Cost to Respondents

We estimate the average hourly wage for youth respondents at $7.25, based on the federal minimum wage from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics (OES), 2023.¹¹ For each instrument listed in Table A.2, the study team calculated the total annual cost by multiplying the annual burden hours by the current federal minimum wage ($7.25), as listed in Table A2.

Table A.2. Total burden requested under this information collection

Instrument	Number of respondents (total over request period)	Number of responses per respondent (total over request period)	Average burden per response (hours)	Total annual burden (hours)	Average hourly wage rate	Total annual respondent cost
Instrument 1. Youth Screener for Cognitive Interviews and Pilot Survey	450a	1	0.08	36	$7.25	$261.00
Phase 1. Instrument 2. Cognitive Interview Protocol and Instrument 3, version A. Youth Self-Assessment Survey	32	1	1.5	48	$7.25	$348.00
Phase 2. Instrument 3, version B. Youth Self-Assessment Survey	350	1	0.17	59.5	$7.25	$431.38
Phase 3. Instrument 3, version C. Youth Self-Assessment Survey	250	1.8b	0.17	76.5	$7.25	$554.63
Total	700			220	$7.25	$1,595.01

^a We anticipate that to recruit 32 youth for phase 1 and 350 youth for phase 2 a total of 450 youth will complete the screener, instrument 1.

^b In phase 3, we anticipate 250 administrations of Instrument 3, version C at the pre-program data collection point, and 200 at the post-program data collection point due to an estimated 20 percent attrition rate.

A13. Costs

Directly engaging the communities ACF serves and including these individuals in ACF research is in line with the following priorities of the current Administration and HHS:

• Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (EO 13985)

• Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government

• ASPE’s Methods and Emerging Strategies to Engage People with Lived Experience (2021)

• ASPE’s Recruiting Individuals with Lived Experience (2022)

Consistent with the guidance documents referenced, we propose to offer youth participating in phase 1 cognitive interviews and phase 2 survey administration an honorarium for their time spent providing their expertise and experience. Specifically, for phase 1, we propose to offer $100 honorarium for participation in the 90-minute cognitive interview to ensure a diverse representation of youth. In phase 2, we propose to offer an honorarium equivalent to $5.50 in points from the vendor for completing the 10-minute survey. Both honoraria are standard amounts that the vendor provides to panel members as part of their agreement to participate in vendor activities. These amounts are set by the vendor and vary based on the estimated time and anticipated engagement for the type of activity. Phase 1 cognitive interviews require 90 minutes of active engagement in person with others and travel time, whereas completing the survey in phase 2 involves 10 minutes virtually.

Equitable compensation is in line with leading practices for ethical engagement of those with lived expertise and advancing equity for populations who have been historically underserved. Providing equitable compensation recognizes the value of the time provided by participants, helps to remove barriers to participation, and affirms that the contributions from those with lived experience are as valuable as those from other experts.

As noted in the 2022 report by ASPE this “helps ensure a diverse population with varied views can participate”.¹² Additionally, a 2021 brief by ASPE says that “Providing [those with lived experience] with compensation commensurate with the rates that other experts—i.e., experts engaged based on their expertise as practitioners or researchers, rather than lived experience—receive helped recognize the valuable and unique expertise that people with lived experience lend, which promoted meaningful engagement.” The report goes on to specify that not doing so could result in “unintended consequences…when lived experience engagements have scarce resources and experts are undercompensated, which can undermine, disregard, and/or marginalize people with lived experience”.¹³

A14. Estimated Annualized Costs to the Federal Government

The estimated total cost to the federal government for this study is $273,000 (Table A.3). This includes costs for collecting, processing, and analyzing the data, and preparing memorandums.

Table A.3. Estimated total cost by category

Cost category	Estimated costs
Field work	$176,000
Data analysis and initial dissemination	$97,000
Annual total	$273,000

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella clearance for pre-testing (0970-0355).

A16. Timeline

Table A.4 contains the timeline for collecting, analyzing, and reporting data. The study team expects to conduct the cognitive interviews in early fall 2024. The study team will conduct preliminary analysis on the data collected to inform revisions to the Youth Self-Assessment Survey (Version B), which will be administered in Phase 2 in fall 2024. Subsequent analysis will be conducted in late fall 2024-early winter 2025. Phase 3 of the pilot, using the revised Youth Self-Assessment Survey (Version C) based on data from phase 2, will begin in early winter 2025. The study will be completed by late spring 2025.

Table A.4. Schedule for pre-test data collection and reporting

Activity	Timinga
Data collectiona and analysis, Phase 1
Cognitive interviews	Early fall 2024
Data Analysis Phase 1	Fall 2024
Data collection and analysis, Phase 2
Single administration to market research vendor panel	Late fall 2024
Data Analysis Phase 2	Fall 2024
Data collection and analysis, Phase 3
Pre-post administration	Winter 2025
Data Analysis Phase 3	Winter-early spring 2025
Reporting	Late spring 2025

^a After obtaining OMB approval

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instruments

Instrument 1: Youth Screener for Cognitive Interviews and Pilot Survey

Instrument 2: Cognitive Interview Protocol

Instrument 3: Youth Self-Assessment Survey

Appendices

Appendix A: Outreach to participants

Appendix B: Participant consent

1 Tingey, L., R. Piatt, A. Hennigar, C. O’Callahan, S. Weaver, and H. Zaveri. (2023). The Sexual Risk Avoidance Education National Evaluation: Using Co-regulation in Youth Programs. OPRE Report 2023-281, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

2 McKenzie, K.J., Meyer, A., and OPRE Self-Regulation Learning Agenda Team. “Co-Regulation and Connection in Human Services: Developing a Learning Agenda.” U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation. https://www.acf.hhs.gov/opre/blog/2022/03/co-regulation-connection-human-services-developing-learning-agenda.

3 Office of Planning, Research, and Evaluation. “Self-Regulation and Toxic Stress Series.” Administration for Children and Families, U.S. Department of Health and Human Services, n.d. https://www.acf.hhs.gov/opre/project/self-regulation-and-toxic-stress-series. Accessed December 20, 2022.

4 As approved under OMB Control Number 0970-0536, expiration date 1/31/2025.

5 Given the iterative refinement of the Youth Self-Assessment Survey, the package refers to versions A-C of the instrument that align with planned phases 1-3.

6 Part 1 of the Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs pretesting activities was approved under this umbrella generic for Pre-testing of Evaluation Data Collection Activities in April 2023.

7 OMB Control Number 0970-0536.

8 Through these consultations, the same information will not be requested from more than 9 respondents.

9 Hinojosa, M.S., Kadivar, H., Fernandez-Baca, D., Chisholm, T., Thompson, L.A., Stanford, J., and E. Shenkman. “Recruiting Low Income and Racially/Ethnically Diverse Adolescents for Focus Groups.” Maternal and Child Health Journal, vol. 18, 2014, pp. 1912-1918.

10 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

11 Characteristics of minimum wage workers, 2023: BLS Reports: U.S. Bureau of Labor Statistics

12 U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. “Recruiting Individuals with Lived Experience,” by Jasmine Forde, Sonia Alves, Lauren Amos, Ryan Ruggiero, Annalisa Mastri, Kate Bradley, Nkemdiri Wheatley, Tonyka McKinney, Dana Jean-Baptiste, Jeremiah Donier, Madison Sandoval-Lunn, Wilnisha Sutton, Roger De Leon, Kataney Prior, Laura Erickson, Amanda Benton, and the HHS staff peer learning community on equitably engaging people with lived experience. Washington, District of Columbia: 2022. https://aspe.hhs.gov/reports/recruiting-individuals-lived-experience.

13 U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. “Methods and Emerging Strategies to Engage People with Lived Experience. Office of the Assistant Secretary for Planning and Evaluation,” by Syreeta Skelton-Wilson, Madison Sandoval-Lunn, Xiaodong Zhang, Francesca Stern, and Jessica Kendall. Washington, District of Columbia: 2021. https://aspe.hhs.gov/reports/lived-experience-brief.

10