SSA - Next Gen Formative Generic #2

Next Generation of Enhanced Employment Strategies Project

OMB Information Collection Request

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part A

Approved May 2019

Revised September 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer: Hilary Bruck

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for non-substantive changes to the semi-structured discussion guide previously approved in May 2019 under ACF’s existing formative generic OMB clearance (OMB #0970-0356) to inform the selection of interventions and programs for the Next Generation of Enhanced Employment Services (NextGen) Project. The proposed changes are necessary to gather information on the effects of the COVID-19 pandemic on the interventions and programs being considered for inclusion in the evaluation. The project, which is sponsored by the Office of Planning, Research, and Evaluation (OPRE), aims to identify and evaluate innovative interventions designed to promote employment and economic security among people with multiple, complex barriers to employment. The NextGen Project could include up to 10 rigorous evaluations.

This generic information collection request (Gen IC) includes activities to identify promising interventions, and programs that offer the interventions, to recommend for rigorous evaluation. It is expected to begin upon OMB approval and continue for up to about 12 months. Stages 1 and 2 of the identification and selection of programs for evaluations, for which data collection was already approved under a previous Gen IC (OMB #0970-0356), include gathering information from stakeholders and contacting administrators of programs implementing innovative interventions. This submission seeks OMB approval for non-substantive changes to previously approved data collection activities that will occur during site visits (conducted in person or virtually, by phone or video) to programs that seem promising and evaluable in Stage 2 and therefore are candidates for inclusion in the study. In May 2019, we received clearance for (1) a semi-structured discussion guide to be used during interviews with staff and (2) a guide to facilitate a group brainstorming discussion about the purposes of the intervention and successes and challenges in its implementation. The site visits also include observations of individual and group activities. The observations do not impose any burden. In this non-substantive change request, we request clearance for revisions to the semi-structured discussion guide to include topics related to the COVID-19 pandemic.

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections: (1) inform the development of ACF research; (2) maintain a research agenda that is rigorous and relevant; and (3) inform the provision of technical assistance.

ACF will submit additional information collection requests (ICRs) as part of the NextGen Project. This will include another generic information collection request (Gen ICs) for formative evaluations of interventions; and full ICRs for baseline, follow-up, implementation, and cost data collection in support of evaluations and case studies of selected interventions.

Study Background

Many Americans have trouble fulfilling their goal of gainful employment and economic independence because they face complex challenges. These challenges may be physical and mental health conditions, substance misuse, a criminal history, or limited work skills and experience. Still, with the right work opportunities and supports, people facing these challenges may be able to find and keep good jobs.

OPRE has spent decades studying strategies to help low-income people find and keep jobs. To further build this evidence, OPRE contracted with Mathematica Policy Research to conduct the NextGen Project. This project will identify and test up to 10 innovative, promising employment interventions designed to help people facing complex challenges secure a pathway toward economic independence. Additionally, the project is working closely with the Social Security Administration (SSA) to incorporate a focus on employment-related early interventions for individuals with current or foreseeable disabilities who have limited work history and are potential applicants for Supplemental Security Income (SSI).

Complex problems often require diverse solutions. These may include work opportunities, treatments for health conditions, personal and workplace supports, and opportunities to build hard and soft skills. Interventions of interest may be delivered by public-private partnerships, interagency collaborations, government initiatives, nonprofit agencies, or social enterprises (organizations that hire workers who face complex challenges to produce goods or services for commercial sale).

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of this Gen IC is to help ACF identify promising interventions and programs for rigorous evaluation in the NextGen Project. This submission seeks OMB approval for non-substantive changes to previously approved data collection that will occur during site visits (conducted in person or virtually, by phone or video) to programs that are candidates for inclusion in the study. In May 2019, we received clearance for (1) a semi-structured discussion guide to be used during interviews with staff and (2) a guide to facilitate a group brainstorming discussion about the purposes of the intervention and successes and challenges in its implementation. The site visits will also include observations of individual and group activities. In this non-substantive change request, we request clearance for revisions to the semi-structured discussion guide to include topics related to the COVID-19 pandemic.

Research Questions

This Gen IC aims to address the following research questions:

• What innovative employment interventions are promising for serving people with multiple, complex barriers to employment?

• Which promising programs might meet the conditions for a rigorous random assignment evaluation?

• What types of formative evaluation might be needed for promising programs before participating in a rigorous random assignment evaluation?

• How has the COVID-19 pandemic changed service delivery, and are those changes likely to persist?

Study Design and Universe of Data Collection Efforts

The NextGen Project aims to identify interventions and programs that will advance the field’s knowledge of what employment interventions work for people experiencing multiple, complex barriers to employment. It will include up to 10 rigorous evaluations of interventions of interest. Each evaluation will include research on the implementation of the intervention, including how the services were designed and implemented; what contextual, organizational, and other factors impeded or facilitated implementation; characteristics of study participants and the services they received; and the role of employers. Each evaluation will include a rigorous randomized controlled trial (RCT) to determine the intervention’s effects on participants’ employment and other outcomes, based on administrative data and surveys. The project will also include a benefit-cost analysis. A future full ICR will include details about the full study design.

This Gen IC request focuses on collecting information to inform the selection of interventions for the evaluations conducted for the NextGen Project. The information gathering conducted under this Gen IC will inform the development of a full ICR. Under this Gen IC, we expect to learn about interventions and programs that implement them that are potential candidates for rigorous testing immediately and others that would require some additional developmental work before being rigorously evaluated.

The evaluation team will conduct formative data collection through site visits (conducted in person or virtually, by phone or video) to programs that will include discussions with select informants. Our approach has three stages. Data collection instruments for Stages 1 and 2 were approved in a Gen IC request submitted in February 2019 (OMB #0970-0356).

1. During Stage 1, we cast a broad net, having informal, semi-structured, short conversations with a range of stakeholders. These conversations focus on eliciting stakeholders’ ideas and suggestions for promising interventions to evaluate and promising programs currently implementing those interventions. Key stakeholders include policy makers; program administrators and managers; knowledge brokers and intermediaries; employer and social enterprise representatives; program developers and training and technical assistance providers; advocates for low-income populations and people with disabilities; and researchers.

2. Based on the knowledge gathered in Stage 1, during Stage 2 we conduct phone or in-person discussions with administrators of programs implementing the interventions that were identified in Stage 1. The purpose of these discussions is to learn more about the program’s goals, services provided, and populations served, and to begin to assess the feasibility of an evaluation.

3. For programs that seem promising and evaluable in Stage 2, during Stage 3 we will conduct in-person or virtual visits to further explore the quality of the program’s implementation and whether it is a good candidate for a rigorous evaluation. The study team will use a semi-structured discussion guide (Attachment A) during the site visits to interview program administrators, supervisors, and direct service staff. The study team will also facilitate a group brainstorming meeting with about six program staff (Attachment B). The purposes of this meeting are to (1) ensure that the intervention has a strong theory of change, making it promising for the evaluation and (2) identify what types of formative evaluation, if any, might be needed before the intervention is ready for a rigorous random assignment study. The study team will also conduct observations of program activities. Based on this information, the study team will determine the feasibility and desirability of including the program in the evaluation and any required formative evaluation.

The discussion guides are similar to guides that have been approved and used successfully in previous OPRE studies, including the Job Search Assistance Strategies Evaluation (OMB #0970-0440), Subsidized and Transitional Employment Demonstration (OMB #0970-0413), and Building Evidence on Employment Strategies for Low-Income Families Project (OMB #0970-0356). We are now seeking approval for non-substantive changes to the site assessment discussion guide (reflected in Attachment A. NextGen Site Assessment Discussion Guide_rev). The COVID-19 pandemic has had profound effects on the delivery of human services, the shift from in-person to virtual services being primary among them. It is critical to understand these effects to inform the assessment and selection of interventions for evaluation under the NextGen Project. Therefore, we have made minor revisions to the previously approved semi-structured discussion guide to include topics related to the COVID-19 pandemic.

A3. Improved Information Technology to Reduce Burden

The project team plans to use improved information technology wherever possible. If the site visits are conducted virtually, program staff will be given the option to dial in by phone, or connect to video and computer audio, depending on their technical capability and preference.

A4. Efforts to Identify Duplication

Before conducting each site visit, the team will have conducted a telephone interview with program leadership to determine whether the program seems promising to evaluate and to gauge leadership’s interest in the evaluation. These information-gathering telephone calls were approved under a previous Gen IC (OMB #0970-0356). During the site visit discussions with staff, we will confirm our understanding but not ask respondents to repeat information we have already learned through previous information collection. Further, the site assessment protocol lays out which level of staff are to answer which questions to avoid asking the same question of staff at multiple levels.

A related project sponsored by ACF, called Building Evidence on Employment Strategies for Low-Income Families (BEES), is also conducting program identification and recruitment activities. OPRE is coordinating stakeholder and program outreach across the two projects to leverage information already collected and to avoid duplication. Additionally, the two projects are coordinating to include a set of common instruments and/or questions in their data collection activities. The full ICRs that ACF will submit as part of the NextGen Project will clearly note these areas of overlap.

A5. Involvement of Small Organizations

We expect involvement of small organizations for program identification and selection to be limited. For the most part, the project team will consult with individual stakeholders and program administrators. If the project team reaches out to small organizations, the burden will be minimized for respondents by conducting telephone discussions at times convenient for the respondents, and requiring no record-keeping or written responses on the part of respondents.

A6. Consequences of Less Frequent Data Collection

Information will be collected only once. If formative data collection is necessary to inform additional evaluations in the future, we will submit additional Gen ICs specific to those evaluations.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

Formative Generic

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

The evaluation team has and will continue to consult select experts in federal agencies as we conduct information collection included in this request. These include select agency staff from the Social Security Administration, Department of Health and Human Service, and Department of Labor. We are also consulting the BEES project team.

A9. Incentives for Respondents

No incentives for respondents are proposed for this information collection.

A10. Privacy of Respondents

For the semi-structured discussions that are part of the Gen IC, no personal identifying information beyond name and professional affiliation (e.g., name of the academic/research institution, name of the State) will be sought. Discussants will be told that their conversations will be kept private to the fullest extent of the law and that it is expected that their name and affiliation will only be included in summary information provided to ACF. ACF staff may participate in discussions.

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

The total annual burden requested under this Gen IC is 214 hours (see table below). The project team expects to conduct about 20 two-day site visits, which will include a mix of discussions with program staff and observations of program activities (about two hours, not included in burden table). We anticipate conducting, at each program visited:

• 90-minute interviews with about 4 program administrators or supervisors (using Attachment A)

• 60-minute interviews with about 8 direct service staff (using Attachment A)

• One 3-hour meeting with about 4 program administrators or supervisors and about 2 direct service staff (using Attachment B)

The total burden has been annualized over the three-year approval period for the overarching umbrella formative generic clearance. The observations of program activities will not place any burden on staff.

Total Burden Requested Under this Information Collection

Instrument	Total Number of Respondents	Annual Number of Respondents1	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Attachment A: NextGen Site Assessment Discussion Guide—Program administrators and supervisors	80	27	1	1.5	41	$49.12	$2,013.92
Attachment A: NextGen Site Assessment Discussion Guide—Direct service staff	160	53	1	1	53	$17.05	$903.65
Attachment B: NextGen Brainstorming Meeting Guide—Program administrators and supervisors	80	27	1	3	81	$49.12	$3,978.72
Attachment B: NextGen Brainstorming Meeting Guide —Direct service staff	40	13	1	3	39	$17.05	$664.95
Estimated Annual Burden Total					214		$7,561.24

Total Annual Cost

The annualized cost burden to respondents is based on the estimated burden hours and the assumed hourly wage rate for respondents. The assumed wage rate is based on the May 2017 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm). The rate used for program administrators and supervisors, $49.12, is equivalent to the local government managers under SOC code 11-1021. The rate used for direct service staff, $17.05, is equivalent to social and human services assistants under SOC code 21-1093. The estimated annualized cost is $7,561.24.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $707,277. Annual costs to the Federal government will be $235,759 for the proposed data collection. This has been annualized over the three-year period of approval of the umbrella formative generic clearance.

A15. Change in Burden

This is a non-substantive change request to a previously approved new individual information collection under the umbrella generic clearance. There are no changes in burden associated with this request.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

The information collected will be used to inform the research design and subsequent data collection efforts for up to 10 evaluations included in the NextGen Project. For this request, in-person or virtual site visits to programs are expected to continue for about 18 months. Consistent with the described potential ways we may share information from Gen ICs under 0970-0356, the data collected under this IC may be shared to inform research design documents or reports; research or technical assistance plans; background materials for technical workgroups; concept maps, process maps, or conceptual frameworks; contextualization of research findings from a follow-up data collection that has full PRA approval; or informational reports to TA providers. In sharing findings, we will describe the study methods and limitations with regard to generalizability and as a basis for policy. Plans for use of later data collected during the study will be further explained in a subsequent package.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Annualized over the three-year period of the umbrella generic.

7