SSB - Next Gen Formative Generic #2

Next Generation of Enhanced Employment Strategies Project

OMB Information Collection Request

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

Approved May 2019

Revised September 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer: Hilary Bruck

B1. Respondent Universe and Sampling Methods

This non-substantive change request to a previously approved generic information collection request (Gen IC) includes activities to inform the Next Generation of Enhanced Employment Strategies (NextGen) Project. Stages 1 and 2 of the identification and selection of programs for evaluations, for which data collection was already approved under a previous Gen IC (OMB #0970-0356), include gathering information from stakeholders and contacting administrators of programs implementing innovative interventions. For programs that seem promising and evaluable in Stage 2, during Stage 3 we will conduct in-person or virtual visits to further explore the quality of the program’s implementation, whether it is a good candidate for a rigorous evaluation, and what type of formative evaluation, if any, might be needed before the program is ready for a rigorous random assignment evaluation. We expect all of the programs we contact will agree to participate in on-site or virtual interviews and meetings. We anticipate visiting about 20 potential programs.

B2. Procedures for Collection of Information

A senior member of the project team will lead the site visit with assistance from a junior member of the team. During the site visit, the team will have discussions with program adminstrators, supervisors, and direct service staff. The senior team member will lead the discussions using a semi-structured discussion guide (Attachment A. NextGen Site Assessment Discussion Guide) and answer any questions about the study. The junior team member will take detailed notes. The guide is designed to collect the minimum information necessary to allow us to understand the program structure and operations, and whether a random assignment study would be feasible and desirable. Under this non-substantive change request, we seek approval for minor revisions to the previously approved semi-structured discussion guide in order to collect information regarding the effects of the COVID-19 pandemic on program operations. It is important to understand whether and how the pandemic has changed service delivery, and whether those changes are likely to persist, to inform our assessment of the program. The requested changes are reflected in Attachment A. NextGen Site Assessment Discussion Guide_rev. Also during the visit, the senior and junior team members will co-facilitate a brainstorming meeting with about six program staff (Attachment B). The purposes of this meeting are to (1) ensure that the intervention has a strong theory of change, making it promising for the evaluation and (2) identify what types of formative evaluation, if any, might be needed before the intervention is ready for a rigorous random assignment study. During the site visit, the team will also observe individual and group activities with participants.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

Participation in this information collection is voluntary. We expect all programs we ask will agree to participate in site visits (conducted in person or virtually, by phone or video). We expect that program administrators and staff will be eager to showcase their innovative interventions with the project team. In addition, we will offer to write a short summary for the program of implementation strengths and opportunities for improvement.

Dealing with Nonresponse

We expect little to no nonresponse because we will have already talked to program adminstrators and begun establishing a relationship with them.

Maximizing Response Rates

The project team will be accommodating of programs’ schedules and will be flexible about the site visit dates and times. If the site visits are conducted virtually, program staff will be given the option to dial in by phone, or connect to video and computer audio, depending on their technical capability and preference. The senior member of the project team who has begun establishing a relationship with the program will arrange the visit. These senior members have had significant experience in working closely with program administrators and staff on previous evaluations and have had a high level of success in engaging programs.

B4. Tests of Procedures or Methods to be Undertaken

Neither the discussion guide nor the brainstorming meeting guide will be pre-tested. Previous studies have successfully used similar instruments during information gathering processes.

B5. Individual(s) Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

This data collection effort will be designed and carried out by senior members of the project team, on behalf of the Administration for Children and Families (ACF), with ACF oversight and approval. The analysis of the interview data will be qualitative, and statistical methods will not be employed. The key individuals involved in this effort are:

• Hilary Bruck, Senior Social Science Research Analyst, ACF

• Gabrielle Newell, Social Science Research Analyst, ACF

• Marie Lawrence, Social Science Research Analyst, ACF

• Connor Williams, Contract Research Assistant, ACF

• Sheena McConnell, Senior Vice President, Mathematica

• Michelle Derr, Senior Researcher, Mathematica

• David Stapleton, Partner, Tree House Economics

• Annalisa Mastri, Senior Researcher, Mathematica

• Angela Rachidi, Senior Researcher, Mathematica

• Crystal Blyler, Senior Researcher, Mathematica

4