SSA - PREP Local Evaluation Support Instruments

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Personal Responsibility Education Program (PREP) Local Evaluation Support Instruments

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

October 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Selma Caal (COR)

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request: This data collection will inform program evaluation support provided by the Administration for Children and Families to State and Competitive Personal Responsibility Education Program (PREP) grantees, as well as to provide support to PREP in Education Innovative Strategies program (PREIS) grantees for their local evaluations. This request includes three types of data collection instruments: 1) a survey that will be used to gather information about State and Competitive PREP grantees’ evaluation plans and related technical assistance (TA) needs to provide them with the needed evaluation TA; 2) a PREIS Analysis Plan Template that will be used to gather information about PREIS grantees’ data analysis plans to provide targeted TA for program evaluation analysis planning and subsequent TA for their program evaluations; and 3) a Sample Enrollment Progress Tool to help grantees assess current sample enrollment and compare actual sample sizes to their target sample sizes to identify areas for TA pertaining to enrollment and follow-up. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: A new cohort of Competitive PREP and State PREP grantees was funded on September 30, 2024. We would like to administer the survey as soon as possible to gather information about grantees’ program evaluation plans at the beginning of the grant period to best support them and inform the types of evaluation support needed. We would like to distribute the Analysis Plan Template and Sample Enrollment Progress Tool to PREIS grantees and their local evaluators by November 2024 – to allow them sufficient time to adequately track sample enrollment and follow-up, set up planned analysis, conduct analyses, and write final reports before their grants end in September 2026.

A1. Necessity for Collection

To improve the life course of adolescents and reduce the risks related to sexual activity, Congress first authorized the Personal Responsibility Education Program (PREP) as part of the 2010 Affordable Care Act. Section 513 of the Social Security Act (42 USC 713) -- as amended by Section 50503 of the Bipartisan Budget Act of 2018 (Pub. L. No. 115-123) extended by Section 3822 of the CARES Act, 2020 (Pub. L. No. 116-136) authorizes funding for PREP grants, including State and Competitive PREP grants, as well as PREIS grants. The legislation also mandates that the Secretary evaluate the programs and activities carried out with funds made available through PREP.

In 2021, the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) contracted with Abt Global (Abt) and their subcontractor Child Trends to provide evaluation support and technical assistance (TA) to PREP grantees and their evaluators. The TA is provided by the PREP Local Evaluation Support Team (PREP LES) and is intended to enhance evaluation rigor and set the stage for the next generation of evidence building specifically for this grant program.

For State and Competitive PREP grantees, a formal evaluation is not a requirement of the grant, but instead, State and Competitive PREP grantees may elect to conduct their own program evaluations.

The survey for State and Competitive PREP grantees proposed in this request is necessary to help understand the types of evaluation activities planned by grantees and their evaluators and their program evaluation TA needs. This will inform the best approach to support grantees while they are conducting their evaluations.

For PREIS grantees, the grant requirements state that grantees must plan and conduct their own rigorous program evaluations. The information garnered from this information collection is necessary to provide targeted TA to grantees to ensure analysis plans are rigorous and meet the Standards of Rigor we have developed for the PREIS grantees. Additionally, the information collected will also help to ensure evaluations enroll and collect data from a large enough sample to be able to detect potential program impacts.

Legal or administrative requirements that necessitate the collection

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The overall purpose of these information collection efforts is to gather information to inform TA support provided by the PREP LES Team. To meet this goal, the team proposes to administer a survey and to gather information through a template and associated tool, as described in the following subsections.

State PREP and Competitive PREP Grantee Evaluation Activities Survey

As part of their evaluation support efforts, Abt is working to identify the program evaluation plans and related needs of two groups of PREP grantees - State and Competitive PREP. This request includes a brief survey to grantee staff to gather information from all State and Competitive PREP grantees about their PREP programs, their evaluation plans, and what TA they need for these program evaluations.

Specifically, the proposed survey would ask State and Competitive PREP grantees about their priority/target populations, program setting, curricula/program models used, and program evaluation plans. Additionally, we would like to ask these grantees about their needs for program evaluation TA and their preferences for how that support is provided (e.g., conversations with other grantees, office hours, guidance documents, online trainings, webinars).

The feedback from this survey will allow us to understand grantees’ current and future evaluation activities, and to improve the evaluation support provided by Abt and their subcontractors. We will also use this survey to help gather contact information for key program staff who should receive ongoing communications from Abt related to program evaluation support. This contact information and descriptive program information will populate a password-protected website maintained by Abt for grantee-specific information related to their evaluation efforts and support.

PREIS Analysis Plan Template and Sample Enrollment Progress Tool

Another main purpose of this information collection is to help PREIS grantees develop data analysis plans that meet Standards of Rigor laid out by the project team, in collaboration with ACF. The analysis planning template provides a systematic, comprehensive way of collecting information pertaining to grantees’ analysis plans, including how they will prepare the data, their analytic approach, and plans for reporting results. Developing a structured analysis plan before examining the data will allow the grantees to foster an efficient and effective approach for analyzing the data and reporting the findings. Additionally, this analysis template is critical for the PREP LES Team to identify grantees’ strengths, determine areas where the plans do not meet Standards of Rigor, develop recommendations for improvement, and inform subsequent TA.

In addition, the Sample Enrollment Progress Tool will help grantees assess current sample enrollment and survey completion to compare actual sample sizes to the target sample sizes on which power calculations were based. This will help to identify potential areas for concern and evaluation TA related to evaluation recruitment and/or retention. This tool will help to ensure grantees have adequate sample size and minimal attrition to meet the Standards of Rigor.

All information provided in the template and tool is for internal grantee and ACF use; there are no plans to share the information publicly.

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Planning for provision of programmatic or evaluation-related T/TA.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Guiding Questions

The information requested is for program and evaluation support only. There is no independent study being conducted using the information collected. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Information Collection Procedures and Processes

Each State and Competitive PREP grantee (85 grantees) will be asked to complete the State PREP and Competitive PREP Grantee Evaluation Activities Survey (Instrument 1). Each grantee will complete the survey once in the Fall 2024 using the instructions as guidance and to determine who is best suited to complete the survey. Then the Abt team will analyze the survey data and use the findings to develop plans to provide program evaluation TA to these State and Competitive PREP grantees.

All 12 PREIS grantees will be asked to complete the following:

• PREIS Analysis Plan Template: Grantees will use the instructions to complete the template (Instrument 2), with direct assistance from their LES Liaison (as needed). Then the project team (ACF and Abt) will review the plans and provide detailed feedback to help grantees strengthen the rigor of their plans. Once finalized, the information collected will serve as the grantees’ analysis plan and will be referred to throughout the rest of the grant period to guide grantees’ analyses and for evaluation TA purposes. Grantee teams will initially submit a first draft of their analysis plan by May 2025. Collectively, the grantee and the LES team will work on revising the plan and submit a final plan for approval by September 2025. Each of the analysis plans will undergo a review process conducted by the PREP LES Team. Using a standardized process, the PREP LES Team will review the grantees’ analysis plans to succinctly document key strengths and weaknesses in the proposed analysis plans and identify opportunities for additional TA.

• Sample Enrollment Progress Tool: Grantees will complete one version of the tool, depending on the evaluation design (e.g., individual assignment – Instrument 3 or cluster assignment – Instrument 4) on a quarterly basis using the instructions (see Appendices B and C) as guidance, with direct assistance from their LES Liaison (as needed). Then the project team (Abt) will review the tool and provide feedback to help grantees troubleshoot any recruitment or retention issues. Grantee teams will complete the Sample Enrollment Progress Tool on a quarterly basis beginning in November 2024. The PREP LES Team will review completed tools and provide feedback/recommendations to help trouble shoot any evaluation recruitment or retention issues identified.

Data Collection Activity	Instrument	Respondent, Content, Purpose of Collection	Mode and Duration
Documentation of State PREP and Competitive PREP grantee evaluation plans and support needs	State PREP and Competitive PREP Grantee Evaluation Activities Survey (Instrument 1)	Respondents: State and Competitive PREP grantee organization evaluator staff Content: Grantee evaluation plans and support needs Purpose: To learn about program evaluation plans to inform future program evaluation TA to PREP grantees	Mode: Online Survey Duration: Approx. 10 minutes
Documentation of PREIS grantee analysis plans	PREIS Analysis Plan Template (Instrument 2)	Respondents: PREIS grantees and program evaluator staff Content: Components of their planned analysis, including data preparation and analytical approach Purpose: To support grantees to develop rigorous data analysis plans and inform TA	Mode: Written responses in a template (Word Document) Duration: 12 hours/grantee including work on all drafts and related revisions
Documentation of PREIS grantee sample progress	Sample Enrollment Progress Tools (Instruments 3 – for individual assignment evaluations; and Instrument 4 – for cluster assignment evaluations)	Respondents: PREIS grantees and local evaluator staff Content: Current sample enrollment and survey completion numbers, target sample sizes Purpose: To support grantees in tracking their progress towards achieving target sample size and inform TA	Mode: Written responses in a reporting tool Duration: .25 hours per response

Other Data Sources and Uses of Information

For all instruments, this data will be used in concert with feedback from grantees and their evaluators attained during small group or 1:1 calls, along with an annual evaluation support feedback survey (Personal Responsibility Education Program (PREP) Local Evaluation Support: PREP LES Annual Satisfaction Survey – OMB Control Number 0970-0401). All data will be used to develop future TA activities. Specific to the PREIS Analysis Plan Template, grantees will also use their completed evaluation plan templates (PREIS Evaluation Plan Template and Guidance - OMB #0970-0531) to support the completion of the Analysis Plan Template.

A3. Use of Information Technology to Reduce Burden

This data collection effort will make use of electronic surveys, template and tools for completion and electronic submission.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

There are no other sources of information that would allow us to determine the necessary details of grantees’ evaluation plans or evaluation support needs for State and Competitive PREP grantees.

No unnecessary information is being requested.

A5. Impact on Small Businesses

The potential exists for data collection activities to affect small entities associated with the grantees. State PREP and Competitive PREP or PREIS evaluation activities may be led by evaluators affiliated with small businesses. Proposed data collection efforts are designed to minimize the burden on all organizations involved, including small businesses and entities, by collecting only critical information consistently across grantees through the use of a brief survey. As the survey, template and tools are electronic, respondents can complete the information at any time that is convenient for them, minimizing overall burden, as well. Specific to the PREIS template and tools, the TA provided by the LES liaison will also minimize the burden on all organizations involved, as the LES liaison will work collaboratively with the grantee to help with the initial draft of the template and tools. Upon receiving feedback from the LES liaison on the initial draft of the template, the LES liaison and grantee will again work collaboratively together to revise the template.

A6. Consequences of Less Frequent Collection

The State and Competitive PREP survey is a one-time data collection. The frequency proposed for the PREIS Analysis Plans and Sample Enrollment Progress Tools is needed to allow the PREP LES Team to effectively and efficiently review grantees’ analysis plans, provide support, and identify additional TA opportunities, which would affect the overall quality and rigor of the PREIS evaluations.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on January 28, 2022, (87 FR 4603), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.

Consultation with Experts

ACF consulted with several current State/Competitive PREP grantee team members and program specialists from the Family and Youth Services Bureau (FYSB) to review and provide feedback on the survey instrument. See below for a list of individuals consulted.

Name	Organization
Kirk Vandersall	Arroyo Research Services
Christopher Butler	The Parenting Center
Jordan Nelon	Centerstone’s Institute
Cheri Thomspon	FYSB
Katherine Godesky	FYSB

Several current PREIS grantees’ evaluators reviewed both the PREIS Analysis Plan Template and Sample Enrollment Progress Tools to provide feedback. See below for a list of individuals consulted.

Name	Organization
Cindy Walker	Consultant to AMTC & Associates
Jordan Nelon	Centerstone
Ashley Raphael Cole	Gang Alternative, Inc.
Catie Henley	The Policy & Research Group
Seow Ling Ong	Education, Training, and Research Associates (ETR)
Andrew Warnke	Evaluation Strategies
Dallas Elgin	RTI

A9. Tokens of Appreciation

No tokens of appreciation for respondents are proposed for this information collection.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

For the State and Competitive PREP survey, the survey asks for information on organizational and individual roles and responsibilities related to the grant and program evaluation-related activities. Specific staff members may be listed by name in order identify who is responsible for what program/evaluation activities. This information will be used by the team to later reach out to the correct individuals to notify them about various TA opportunities and resources. The survey also collects professional contact information, which is likely publicly available.

For the PREIS tool and templates, these data collection effort do not include collecting personally identifiable information.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. The one exception is for project information (e.g., priority/target population, program setting, curricula/program models used) and contact information of key project personnel (name, role on the project, and professional email address) that will be collected as part of the State and Competitive PREP survey that will be used to populate their profile on the project website (note that the website requires login to access this information; only the PLESD project team, OPRE, FYSB, and other PREP grantees have access to this site). As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

At least some of the information collected under this request will likely be retrieved by an individual’s personal identifier in a way that triggers the Privacy Act of 1974, as amended (5 U.S.C. 552a). The system of records notice (SORN) for this collection is OPRE Research and Evaluation Project Records, 09-80-0361. Each individual will be provided with information that complies with 552a€(3) prior to being asked for information that will be placed into that system of records. This means respondents will receive information about the authority, the purposes for use, the routine uses, that the request is voluntary, and any effects of not providing the requested information.

Data Security and Monitoring

As specified in the contract, Abt shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. The Contractor has a documented Data Security Plan (DSP) in place that will be updated as needed throughout the life of the contract. All data, including portable media (e.g., voice/video recordings) and computerized files, are kept in secure areas.

For the State and Competitive PREP survey, grantees will submit their survey responses securely, such as via Confirmit. The resulting database will be saved in a secure project file. We do not intend to disseminate any information collected via this survey.

For the PREIS instruments, grantees will submit their completed Analysis Plan Templates and Sample Enrollment Progress Tools via email, which will be saved in secure project files. We do not intend to disseminate any information collected via these templates.

A11. Sensitive Information ¹

There are no sensitive questions in this data collection.

A12. Burden

Explanation of Burden Estimates

• State PREP and Competitive PREP Grantee Evaluation Activities Survey (Instrument 1): A maximum of 85 State and Competitive PREP grantees will complete the survey. We expect grantees will need approximately 10 minutes to complete the survey. Each grantee will complete the survey once.

PREIS Analysis Plan Template (Instrument 2): All 12 PREIS grantees will complete the Analysis Plan Template. We expect grantees will need about 12 hours in total to complete the Analysis Plan Template. This burden estimate includes the assumption that all PREIS grantees will need to draft the initial analysis plan template (8hrs). Grantees may need to revise this template between two to four times based on feedback received from the project team, with two being the anticipated average number of revisions (2hr). Grantees may also need to update the analysis plan later (2hrs).

• Sample Enrollment Progress Tools (Instrument 3 and 4): A maximum of 6 PREIS grantees will complete the Sample Enrollment Progress Tool for Individual Assignment, and a maximum of 7 PREIS grantees will complete the Sample Enrollment Progress Tool for Cluster Assignment. We expect grantees will need about .25 hours to complete the tool, and we anticipate they will complete the tool on a quarterly basis, up to 8 times, over the course of 2 years.

Estimated Annualized Cost to Respondents

For all cost calculations, we estimate the average hourly wage for program directors and managers to be the average hourly wage for “Social and Community Services Manager” ($40.10), taken from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics, 2023. We estimate the average hourly wage for local evaluators to be the average hourly wage for “Economists” ($55.64), taken from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics, 2023.²

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
State PREP and Competitive PREP Grantee Evaluation Activities Survey	85	1	.17	14	$40.10	$568.22
PREIS Analysis Plan Template	12	1	12	144	$55.64	$8,012.16
Sample Enrollment Progress Tool – Individual Assignment	6	8	.25	12	$55.64	$333.84
Sample Enrollment Progress Tool – Cluster Assignment	7	8	.25	14	$55.64	$389.48
Totals	218			184		$9,303.70

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Activity	Detail	Estimated Cost
Survey administration for the State and Competitive PREP Grantee Evaluation Activities Survey	Survey programming/testing Survey administration (initial email and reminders) Management	$6,111
Analysis for the State and Competitive PREP Grantee Evaluation Activities Survey	Descriptive analysis Analysis of open-ended responses Summary	$9,050
Field work for the PREIS Analysis Plan Template and Sample Enrollment Progress Tools	Reviewing templates and tools Providing feedback on template and tools Attending 1:1 calls with grantees	$156,098
Total costs over the request period		$171,259

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

Following OMB approval, we will email the State and Competitive PREP Grantee Evaluation Activities Survey to State and Competitive PREP grantees (see Appendix A for a copy of the email text. The survey will be open for about one month and data will be analyzed and summarized in winter 2024 with findings anticipated in January 2025. Findings will be discussed with ACF and incorporated into the project’s evaluation TA plan. Ultimately the findings will inform program evaluation TA efforts beginning in Spring 2025.

For the PREIS Analysis Plan Template and Sample Enrollment Progress tools, we will share the template and tools with PREIS grantees (see email text in Appendices D and E) and an informational webinar recording, so they can begin to complete the template and tool and use them as a resource. The goal is to distribute templates in mid-November.

PREIS grantees will submit a first draft of the Analysis Plan Template for review by their LES liaison by spring 2025. The LES liaison and grantee will work together to revise the template to allow PREIS grantees to submit a plan ready for FYSB approval by September 2025.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument 1 - State PREP and Competitive PREP Grantee Evaluation Activities Survey

Instrument 2 – PREIS Analysis Plan Template

Instrument 3 – Sample Enrollment Progress Tool – Individual Assignment

Instrument 4 – Sample Enrollment Progress Tool – Cluster Assignment

Appendix A - Invitation email for SPREP/CPREP grantees

Appendix B – Individual Assignment Tool Instructions

Appendix C – Cluster Assignment Tool Instructions

Appendix D – PREIS Analysis Plan Template email

Appendix E – Sample Enrollment Progress Tool email

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

2 Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Social and Community Service Managers, on the Internet at https://www.bls.gov/ooh/life-physical-and-social-science/economists.htm (visited June, 2024).

12