SSA Res Eval GenIC - Sexual Risk Avoidance Education National Evaluation: Co-regulation Guidance Pilot Study

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education National Evaluation: Co-regulation Guidance Pilot Study

Formative Data Collections for ACF Research

0970 – 0356

Supporting Statement

Part A

January 2025

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

Tia Brown

Kathleen McCoy

MeGan Hill

Nakia Martin-Wright

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for the Administration for Children and Families (ACF) Research (0970-0356).

• Progress to Date: The Sexual Risk Avoidance Education National Evaluation (SRAENE) provides information on the implementation of Sexual Risk Avoidance Education (SRAE) programs, the effectiveness of selected program components, and the ways grant recipients can use data and evidence to improve SRAE programming. As a first step in understanding the effectiveness of program components, ACF conducted the program components impact study (CIS) proof of concept pilot study that explored the use of three co-regulation strategies with three SRAE programs (OMB #: 0970-0531). Next, from August 2022 through May 2023, ACF conducted a formative evaluation of an in-depth implementation study that assessed the use of co-regulation strategies with nine SRAE programs (OMB #0970-0356). Through this work, ACF learned that facilitators were able to be trained on and implement co-regulation strategies in SRAE classrooms, but facilitators varied widely in the frequency and quality with which they adopted and used the strategies.

Description of Request: As part of SRAENE, ACF seeks to test the implementation of six co-regulation strategies to improve facilitation and to promote youth self-regulation during SRAE classes. This information collection request builds on the proof of concept pilot and formative evaluation by collecting data from facilitators to understand the feasibility of using newly developed guidance to assist facilitators with understanding where and how to integrate co-regulation strategies into a SRAE curriculum. Data will be collected through a facilitator log, a post-training pulse check survey, a post-coaching pulse check survey, and facilitator mini-interviews. Data collection will occur with facilitators at two sites. Data collected under this generic information collection request will inform key areas outlined in ACF’s co-regulation learning agenda¹, a potential large-scale impact study to test the effectiveness of the strategies on facilitation and youth outcomes, and the provision of technical assistance planning and resources. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

In February 2018, Congress reauthorized Title V, Section 510 of the Social Security Act to fund the Sexual Risk Avoidance Education (SRAE) grant program. SRAE grants fund programs that teach adolescents to refrain from sexual activity. The Family and Youth Services Bureau (FYSB), Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services (HHS), administers the program. SRAE programs also provide education on personal responsibility, self-regulation, goal setting, healthy relationships, a focus on the future, and preventing drug and alcohol use. The reauthorization included a requirement to evaluate the SRAE grant program. ACF awarded an initial contract for the SRAE National Evaluation (SRAENE) in 2018 and a subsequent contract (SRAENE 2.0) in 2023.

This request pertains to continuing one of SRAENE’s required activities – a core components impact study (CIS), which seeks to refine and test improvements to one or more components of programs to ultimately improve youth outcomes. Core components can include any part of a program, including curriculum content, supplementary activities, delivery, facilitation, setting, and dosage. Initial work for the CIS began under SRAENE 1.0 and included a CIS proof of concept pilot (approved under the umbrella generic for Formative Data Collections for ACF Program Support OMB #: 0970-0531). This initial proof of concept work identified and pilot tested two facilitation strategies, one of which, a set of co-regulation strategies to support facilitators with building youth’s self-regulation skills (referred to as co-regulation),² was deemed viable for continued exploration. Based on the results of the proof of concept pilot, ACF focused on gathering further information related to the co-regulation strategy (approved under the umbrella generic for Formative Data Collections for ACF Research, OMB#: 0970-0531). The data collection focused on (1) gathering evidence to inform direction of the future large-scale evaluation of the effectiveness of using co-regulation strategies, (2) informing guidance and products to support successful replication of co-regulation strategies by additional programs if rigorously evaluated, and (3) helping further ACF’s co-regulation learning agenda.

Ultimately, we learned that it is feasible to train facilitators to use the co-regulation strategies and implement them within SRAE programs in high schools, but facilitators varied widely in the frequency and quality of their use of the strategies. In response to these findings, we developed explicit guidance for facilitators on how to integrate the co-regulation strategies into an existing SRAE curriculum. To support the next formative evaluation phase, we now propose to collect data on the feasibility and use of this integrated guidance from up to two sites. This proposed data collection will (1) gather evidence to inform the direction of a future large-scale evaluation of the effectiveness of using co-regulation strategies, (2) assess the feasibility of facilitators consistently implementing the co-regulation strategies, and (3) help to further ACF’s co-regulation learning agenda. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The goal of this proposed data collection is to inform the development of ACF research, specifically related to integrating co-regulation strategies into the SRAE grant program improvement efforts. This co-regulation guidance pilot study will contribute new knowledge and will build preliminary evidence to support several key questions outlined by ACF’s co-regulation learning agenda³, such as: What is the feasibility of implementing co-regulation strategies? What practice-based supports are needed? How are co-regulation strategies implemented in different contexts and among different subpopulations?

This pilot study seeks to document the feasibility of using the integration guidance while facilitating SRAE classes and the extent to which the guidance results in more consistent implementation of the strategies. The information collected through this study will be used to inform future research and a potential impact study on the effectiveness of using co-regulation strategies on youth well-being and other program impacts⁴. This specific request is related to the collection of qualitative and survey data that will be used to describe how the facilitators use the integration guidance and implement the co-regulation strategies during their classroom instruction, which ACF could use to inform the planning for a possible future effectiveness evaluation of the co-regulation strategies.

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for research and evaluation (0970-0356):

• inform the development of ACF research

• maintain a research agenda that is rigorous and relevant, and

• inform the provision of technical assistance.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions

This pilot study will address one primary research question, with several secondary questions. As noted above, this question reflects the information needs for ACF’s co-regulation learning agenda and continues to explore whether integrating co-regulation strategies can improve youth learning through improved facilitation.

1. Does use of the co-regulation integration guidance appear to support the consistent inclusion of the co-regulation strategies when implementing a SRAE program?

   1. How easy is it for facilitators to use the co-regulation integration guidance?

   2. How does the guidance support facilitators’ implementation of the co-regulation strategies?

   3. What successes and challenges are associated with use of the guidance and implementation of the co-regulation strategies?

Study Design

To conduct the co-regulation guidance pilot study, the team will recruit up to two sites to receive training on the six SRAE co-regulation strategies and associated integration guidance. For information about the sites to be recruited, see Supporting Statement B, Section B2. Data will be collected from all facilitators in each site (estimated at five facilitators per site). We estimate the data collection period will take up to 12 weeks, accounting for training the facilitators and collecting data.

Program facilitators working in the two selected sites will be asked to complete several brief surveys: a daily facilitator log (Instrument 1. Facilitator log) to occur after facilitating each class over a 6-week period; a brief post-training pulse check (Instrument 2. Facilitator post-training pulse check) immediately following the last training session; and a brief post-coaching pulse check (Instrument 3. Facilitator post-coaching pulse check) that they will complete immediately following the last coaching support call. For the facilitator log, facilitators will be asked to indicate what strategies they used during each class they facilitated and an overall rating of how they felt it went. The post-training and post-coaching surveys will capture facilitators’ satisfaction with training and coaching and identify areas for improvement.

Facilitators will also be asked to participate in six weekly, 15-minute interviews (Instrument 4. Facilitator mini-interview guide) to discuss their perceptions of how the strategies are working, examples of why they thought certain strategies went well or why they went poorly, and their reactions to the usefulness of the integration guidance.

Table A.1 includes data collection by instrument, participant, content, purpose, and mode and duration of the data collection. To understand the feasibility of using the guidance, we are focusing data collection on the facilitators who interact directly with youth.

Table A.1. Study design summary

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Facilitator log	Instrument 1. Facilitator log	Respondents: Program facilitators Content: Use of the strategies, reactions to how implementation went Purpose: To determine the frequency with which facilitators use the co-regulation strategies and their perceptions of how implementing the strategies went	Mode: Paper, web Duration: Daily input during selected weeks, 2 minutes per completed log
Facilitator survey	Instrument 2. Facilitator post-training pulse check	Respondents: Program facilitators Content: Feedback on training; satisfaction; recommendations for improvement Purpose: To capture facilitator satisfaction with the training and recommendations for how this could be improved in the future	Mode: Web Duration: 5 minutes per survey
Facilitator survey	Instrument 3. Facilitator post-coaching pulse check	Respondents: Program facilitators Content: Feedback on coaching sessions; satisfaction; recommendations for improvement Purpose: To capture facilitator satisfaction with coaching sessions and recommendations for how these could be improved in the future	Mode: Web Duration: 5 minutes per survey
Facilitator mini-interview	Instrument 4. Facilitator mini-interview guide	Respondents: Program facilitators Content: Use of strategy; success or challenges in implementation; suggestions for improvement Purpose: To determine how the strategy is being used and facilitators’ perceptions and experiences with using the strategy	Mode: Video conference Duration: 15-minutes weekly during selected weeks

Other Data Sources and Uses of Information

This information collection builds on previous information collections in support of this project (as described previously). For this next formative stage, no other data sources are planned.

A3. Use of Information Technology to Reduce Burden

The study team plans to use information technology wherever possible to reduce burden. The facilitator mini-interviews will be conducted remotely via videoconference. With participant permission, the study team plans to record interviews to help reduce respondent burden by reducing the time needed to take notes during data collection. The facilitator log and pulse check surveys will be available as web-based instruments. We will provide links via email that facilitators can use to access and complete the web-based instruments using a tablet, smartphone, or laptop.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

None of the instruments ask for information that can be reliably obtained through other sources.

A5. Impact on Small Businesses

The programs participating in the study will be small, non-profit organizations. The SRAENE team will request information required only for the intended use. The burden for respondents will be minimized by restricting the interview and survey length to the required minimum, conducting interviews at times convenient for the respondents, and not requiring additional record-keeping on the part of the programs.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published two notices in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This first notice was published on August 11, 2023 (88 FR 54614) and provided a sixty-day period for public comment. The second notice published on December 14, 2023 (88 FR 86656) and provided a thirty-day period for public comment. ACF did not receive any substantive comments.

Consultation with Experts Outside of the Study

The SRAENE 2.0 team consulted with an external expert in co-regulation, Aly Frei, who provided feedback that clarified the purpose and usefulness of this next proposed phase of the study. At the end of the pilot study, the SRAENE 2.0 team plans to consult with an SRAE curriculum developer, the Dibble Institute, to finalize the integrated guidance.

A9. Tokens of Appreciation

No tokens of appreciation are planned for this data collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

This data collection effort will collect personally identifiable information (PII) from facilitators (names, work email addresses, and telephone numbers) to obtain consent to participate in data collection activities and arrange data collection (including scheduling and sending invitations/reminders for data collections).

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

All study participants will be informed of the planned uses of data, that their participation is voluntary, and that the study team will keep their information private to the extent permitted by law. The study team will discuss issues of privacy during training sessions with staff who work on the project. The contractor, Mathematica, requires that staff complete online security awareness training when they are hired and then participate in annual refresher training thereafter. Training topics include the security policies and procedures outlined in the Mathematica Corporate Security Manual. All records containing data will be transferred using a secure file transfer protocol site, in case the files contain PII such as facilitator name or, in the case of the youth focus groups, the youth and parent names. As specified in the contract, Mathematica will protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. In addition, the study leaders at Mathematica will conduct project-specific trainings of all staff who work on the study to communicate the expectations on privacy, informed consent, and data security procedures.

Data Security and Monitoring

As specified in the contract, the contractor shall protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. The contractor has developed a Data Security Plan that assesses all protections of respondents’ PII. The contractor will ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor who perform work under this contract and subcontract receive training on data privacy issues and comply with the above requirements. All Mathematica staff must sign an agreement to (1) maintain the privacy of any information from individuals, businesses, organizations, or families participating in any projects conducted by Mathematica; (2) complete online security awareness training when they are hired; and (3) participate in a refresher training annually.

As specified in the evaluator’s contract, the contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard. The contractor will incorporate this standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing to the extent possible including sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive information to ensure secure storage and limits on access.

No information will be given to anyone outside the SRAENE study team and ACF. All PII, typed notes, and audio recordings of interviews and focus groups will be stored in restricted, encrypted folders on Mathematica’s secure network, which is accessible only to the study team.

A11. Sensitive Information ⁵

There are no sensitive questions proposed in the instruments.

A12. Burden

Explanation of Burden Estimates

Table A.2. summarizes the estimated reporting burden and costs for each instrument. The estimates for Instruments 1-3 include the time for respondents to review the instructions, complete, review, and transmit their responses. The time estimate for Instrument 4, the interview, includes time for participants to review the instructions and participate in a virtual interview. Figures are estimated as follows:

1. Facilitator log. Implementation logs will be web-based. All facilitators in the study (N = 10) will complete the implementation log each time they deliver the SRAE program for 6 weeks in the spring 2025 semester. We estimate that facilitators will teach 3 classes per day 5 days per week over the 6-week period [(5 days per week * 6 weeks = 30 days) * 3 classes per day = 90 completions]. We anticipate that all facilitators will respond to the log and will monitor their completion to obtain logs for at least 80 percent of their completed sessions. The log is estimated to take 2 minutes to complete each time.

2. Facilitator post-training pulse check. The survey will be administered to all program facilitators in the study (N = 10) immediately following the last session of the co-regulation training. Because the survey will be administered during the training session, we anticipate a 100 percent response rate. The facilitator post-training survey is estimated to take 5 minutes to complete via the web.

3. Facilitator post-coaching pulse check. The survey will be administered to all program facilitators in the study (N= 10) immediately following the last coaching session. Because the survey will be administered during the coaching session, we anticipate a 100 percent response rate. The facilitator post-coaching survey is estimated to take 5 minutes to complete via the web.

4. Facilitator mini-interviews. The interviews will be completed with all program facilitators in the study (N= 10). Facilitators will participate in six weekly interviews. We anticipate a 100 percent response rate. The interviews will be completed via a web conference call, with each interview will last approximately 15 minutes.

Estimated Annualized Cost to Respondents

The study team expects the total annual cost to be $1,662.46 for all instruments in the current information collection request. The Occupational Employment Statistics (2023)⁶ from the Bureau of Labor Statistics have been used to estimate the average hourly wage for the participants of this study and derive total annual costs.

Table A.2. Total burden requested under this information collection request

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 1: Facilitator log	10	90	0.033	30	$26.40	$792.00
Instrument 2: Post-training pulse check	10	1	0.083	0.8	$26.40	$21.12
Instrument 3: Post-coaching pulse check	10	1	0.083	0.8	$26.40	$21.12
Instrument 4: Facilitator mini-interview	10	6	0.25	15	$26.40	$396.00
Total	10	98	Avg: 0.048	47		$1,230.24

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The estimated total cost to the federal government for this study is $108,605 (Table A.3). This includes costs for collection and processing the data, conducting analysis, and preparing internal reports.

Table A.3. Estimated total cost by category

Cost Category	Estimated Costs
Field Work	$79,260
Publications/Dissemination	$29,345
Total costs over the request period	$108,605

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for ACF research (0970-0356).

A16. Timeline

Table A.4. contains the timeline for data collection, analysis, and reporting activities. The study team expects to collect data in winter and spring 2025, followed by analysis and reporting in late spring and summer 2025.

Table A.4. Schedule for the co-regulation guidance pilot study data collection and reporting

Activity	Timing
Data collectiona
Facilitator post-training pulse check	Within one month following OMB approval
Facilitator mini-interviews	Within two months following OMB approval
Facilitator logs	Within two months following OMB approval
Facilitator post-coaching pulse check	Within the final week of facilitator coaching (~two months following OMB approval)
Data analysis	One month following the collection of all data
Reporting	Two months following the completion of data analysis

^a After obtaining OMB approval

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix A. Study Notification and Reminder Materials

Instrument 1. Facilitator log

Instrument 2. Facilitator post-training pulse check

Instrument 3. Facilitator post-coaching pulse check

Instrument 4. Facilitator mini-interview guide

1 https://www.acf.hhs.gov/opre/blog/2022/03/co-regulation-connection-human-services-developing-learning-agenda

2 A prior formative evaluation completed as part of the Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Programs for Youth (SARHM, OMB Control Number 0970-0355) guided the development of the co-regulation strategies and suggested they can be integrated into youth-serving programs, such as SRAE.

The other facilitation strategy that was not ready for further exploration teaches facilitators how to assess youth’s attitudes and beliefs on the delay of sexual initiation.

3 https://www.acf.hhs.gov/opre/blog/2022/03/co-regulation-connection-human-services-developing-learning-agenda

4 Related information collections that are subject to the Paperwork Reduction Act will be submitted for review and approval under an appropriate mechanism (generic or full approval) as needed.

5 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

6 U.S. Bureau of Labor Statistics. “May 2023 National Occupational Employment and Wage Estimates.” Available at https://www.bls.gov/oes/current/oes_nat.htm. The mean hourly wage for educational instruction and library workers (Occupational Code 25-9099) of $26.40 was used for the program facilitators who complete the facilitator log, facilitator post-training and post-coaching pulse checks, and the facilitator mini-interviews.

11