SSA - BEES Formative Generic

The Building Evidence on Employment Strategies for Low-Income Families Project (BEES)

OMB Information Collection Request

0970-0356

Supporting Statement

Part A

November 2017

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Tiffany McCormack

Carli Wulff

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for formative data collection activities for The Building Evidence on Employment Strategies for Low-Income Families Project (BEES). This request is for a new collection that falls within ACF’s existing formative generic OMB clearance (OMB #0970-0356). This submission seeks OMB approval for three data collection instruments that will be used as part of the field assessment and to inform the intervention and site selection process:

• Discussion Guide for Researchers and Policy Experts

• Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice

• Discussion Guide for Program Staff

This submission also seeks OMB approval for an email designed for contacting potential informants to arrange a discussion. ACF will submit additional information collection requests (ICRs) as part of this project, including a full ICR for evaluations of the selected programs.

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for a series of tailored, semi-structured discussions under ACF’s existing formative generic OMB clearance (OMB #0970-0356) to inform the selection of interventions and sites for the BEES project. BEES is a large-scale project, sponsored by the Office of Planning, Research, and Evaluation (OPRE), that aims to build evidence on interventions designed to promote employment and economic security among TANF recipients and other disadvantaged groups, including those facing substance abuse (including abuse of opioids) and mental health challenges. If all options in the project are exercised, BEES may include up to 21 rigorous evaluations of employment-focused interventions. We are currently focusing on the first phase, which is expected to include 3 evaluations. We will submit additional generic information collections (Gen ICs), if needed, to inform potential additional evaluations in the future.

This Gen IC includes an initial scan to identify high priority interventions and sites to recommend for rigorous evaluation. The Gen IC is expected to begin upon OMB approval and continue for approximately 12 months. The Gen IC will inform the research design, and recruitment and sampling strategies used for the first phase of the project. This submission seeks OMB approval for three data collection instruments that will be used as part of the field assessment to inform the intervention and site selection process:

• Discussion Guide for Researchers and Policy Experts

• Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice

• Discussion Guide for Program Staff

Study Background

Overview of BEES

Note: This section goes beyond a description of this current Gen IC by providing background on and plans for the full BEES project. This Gen IC is an initial step, with subsequent ICRs expected to be submitted in the future to cover data collection for the evaluation once sites have been selected, including baseline data, survey data, implementation data, and follow-up data for estimating impacts.

BEES is a project sponsored by OPRE within ACF under contract to MDRC and its subcontractors, Abt Associates and MEF Associates. BEES aims to add to the knowledge base of employment interventions for TANF recipients and other disadvantaged groups, including those facing substance abuse and mental health challenges. To maximize learning, the BEES project will build on the substantial body of evidence on employment interventions that have been developed over the past 40 years, as well as the results from key ongoing studies that will be emerging in the next year or two (see section A.4).

Rigorous studies have demonstrated that many types of interventions can improve labor market outcomes for disadvantaged groups. Job search and subsidized employment can increase employment and earnings in the short-term, while earnings supplements can increase both employment and income, at least while supplements remain in place. Career pathways models can produce lasting earnings gains for those who meet program entrance criteria.¹ However, despite this extensive body of evidence, there remain many open questions: lengthier programs consistently struggle with attrition, and it is important to understand the cost effectiveness of various supports for participants; relatively little is known about the best structures and pedagogy for delivering occupational training; and there are few training models that can accommodate individuals with low literacy and numeracy skills.

In addition to these programmatic challenges, many disadvantaged groups also experience behavioral health issues – notably substance use and mental health disorders – that further hinder their ability to obtain stable employment. There is strong evidence of the efficacy of specific treatment approaches for substance abuse, depression, anxiety, and other conditions that can act as barriers to steady employment. The challenge – and a promising area of inquiry for BEES – is to learn how best to promote steady participation in treatment, and how to effectively integrate treatment with employment services.

Current phase

BEES is currently in the planning and design phase in which decisions regarding the interventions to be tested and other aspects of the design will be made. In addition, the recruitment and sampling strategies will be formulated during this phase of the study to ensure that we are able to efficiently and successfully recruit employment interventions for TANF recipients and other disadvantaged groups.

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of this current Gen IC is to help ACF identify high priority interventions for rigorous evaluation in the first phase of BEES. After this information-gathering step, we plan to submit a full ICR for future data collection for the evaluation, including baseline data, survey data, implementation data, and follow-up data for estimating impacts.

This Gen IC request includes three discussion guides that will be used in telephone and/or on-site interviews with select categories of informants: Discussion Guide for Researchers and Policy Experts (Attachment A); Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice (Attachment B); and Discussion Guide for Program Staff (Attachment C). The three guides cover similar topics but are differentiated by the level of detail appropriate for each group of informants. We intend to use the information gathered through these discussions to inform the selection of interventions as well as the research design, recruitment, and sampling strategies used for the first phase. This approach will be repeated for potential later phases.

Research Questions

This Gen IC aims to address the following research questions:

• Based on prior and emerging research, and current practice, what types of programs and approaches are most promising in each domain identified as a priority by ACF, states, or other stakeholders?

• Which specific interventions in each domain are ready to move to the next level of evidence?

The full BEES Project aims to address the following additional research questions:

• What is the impact of the specific interventions, both on critical intermediate measures of success and on longer-term measures of employment and economic security?

• Going beyond the averages, what can be learned about the impact of specific program components, or the impact of interventions for important subgroups of the target population?

• What lessons on program implementation and cost can shed light on the impact results and help facilitate the expansion or replication of successful interventions?

• What lessons can be drawn across the project about the characteristics and implementation of successful interventions?

Study Design

BEES aims to identify interventions and sites that ensure that the evaluation will build on past and ongoing research and reflect the priorities of ACF, states, and other stakeholders. The first phase of BEES will include 3 evaluations of employment-focused interventions; the full project (with up to four phases) may include up to 21 rigorous evaluations. Each evaluation will include research on the implementation of the intervention, which will focus on program practices and implementation fidelity, participant engagement, context, and the counterfactual. Each evaluation will also include research on the intervention’s effects on participants’ employment outcomes, based on data sources including administrative data and surveys. Randomized controlled trials (RCTs) are the preferred method for estimating impacts in the BEES evaluations but other rigorous designs will be considered when necessary. A future ICR will include details about the full study design.

This Gen IC request focuses on collecting information to inform the selection of interventions and sites for the first phase of BEES. In this IC, we expect to identify both interventions and sites that are potential candidates for rigorous testing in the first phase and others that would require some additional developmental work (e.g., adapting a well-studied intervention to a TANF population). The first three evaluations in BEES will test interventions that our reconnaissance determines to be most ready for rigorous testing, most likely at the site where they currently operate. The latter category of interventions might be tested in later phases of BEES, along with others identified in potential future scans.

For this Gen IC, the evaluation team will conduct formative data collection through discussions with select informants. Three discussion guides are included in this Gen IC request: Discussion Guide for Researchers and Policy Experts (Attachment A); Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice (Attachment B); and Discussion Guide for Program Staff (Attachment C). The three guides cover similar topics but are differentiated by the level of detail appropriate for each group of informants. The specific subset of questions asked will depend on the informant’s expertise and background. For example, a researcher or policy expert might be asked about promising interventions and models, while a state-level TANF administrator might be asked about priority areas of programming in their state. We intend to use the information gathered through these discussions to inform selection of interventions and sites as well as the research design, recruitment, and sampling strategies used for the first phase. This approach will be repeated for potential later phases.

The evaluation team will identify informants through a purposeful, snowball sampling process that draws from recommendations from the Contracting Officer’s Representatives (CORs) and other staff in OPRE, referrals from other informants, and Internet searches of employment based interventions. Agencies that may be consulted include: ASPE, SAMHSA, National Institute of Drug Abuse, National Institute of Mental Health, DOL (Chief Evaluation Office, Employment and Training Administration, Office of Vocational Rehabilitation and Employment, Office of Disability and Employment Policy), Department of Education (Institute for Educational Sciences and the Office of Vocational and Adult Education), Department of Justice (Office of Justice Programs and National Institute of Justice), HUD (Policy Development and Research), Social Security Administration (Office of Disability), and Department of Agriculture (Food and Nutrition Service).

The evaluation team plans to interview informants through a combination of phone and in-person discussions conducted one-on-one or in small groups. Since discussions with informants will be conducted on an ongoing basis, we expect that the selection of informants will also be refined periodically, based on the information obtained. In identifying and selecting informants, we will contact individuals who will add new information, based on our extant knowledge base and gaps where we hope to gain further insights, in order to minimize potential burden on Gen IC participants. Priority candidates will be identified based on their field of expertise, purpose of engagement, geographic/locality representation, type of organization they represent, and other key information. There are no quantitative components to this formative data collection effort.

Universe of Data Collection Efforts

Informants in this Gen IC will include researchers and policy experts in the fields of workforce development, opioid and substance abuse, and mental health; state and local administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice; and program staff of employment interventions for TANF recipients and other disadvantaged groups, including those facing mental health and addiction challenges. The information will be collected through semi-structured interviews that will cover a range of topics, including the purpose of and content of employment services, participation in/targeting of employment services, and, when appropriate, staffing of the informants’ organization and their interest in possibly participating in the study. The discussion guides for these three groups of respondents vary slightly, but all focus on these topics. The discussion guides are similar to guides that have been approved and used successfully in prior OPRE studies, including the Job Search Assistance Strategies Evaluation (OMB #0970-0440), Subsidized and Transitional Employment Demonstration (OMB #0970-0413) and the Assets for Independence Evaluation (OMB #0970-0414).

A3. Improved Information Technology to Reduce Burden

The evaluation team plans to use improved information technology wherever possible. When information is available from the internet, it will supplement requests for information. Whenever possible, discussions will be done by telephone to reduce burden on the respondents. We will also use technology to reduce burden by initially reaching out to potential informants via email, providing information about the BEES project and the purpose of the call. The email is included in this Gen IC request as a supplemental document (Attachment D).

A4. Efforts to Identify Duplication

The BEES project is designed to fill gaps in the substantial body of evidence on employment interventions that has been developed over the past 40 years. The evaluation team will begin by carefully reviewing the existing literature and talking to federal officials in order to identify research questions that will add to the knowledge base rather than duplicating what is already known. For example, in the career pathways area, the project might focus on populations that were not well served by previous tested interventions. Similarly, the evidence-based Individual Placement and Support (IPS) supported employment model might be tested for new populations. In the area of opioid dependency/substance, we believe there is little systematic information about current employment interventions targeted specifically to this population. By reviewing the literature and talking to federal officials before starting the information collection, we will ensure that we will not spend informants’ time discussing topics that are already well known.

A5. Involvement of Small Organizations

We expect involvement of small organizations for the field assessment and site selection to be limited. For the most part, the evaluation team will be consulting with individual experts, and systems and program administrators. If the researchers reach out to small organizations, the burden will be minimized for respondents by restricting the interview length to the minimum required, by conducting telephone discussions at times convenient for the respondents, and by requiring no record-keeping or written responses on the part of the programs.

A6. Consequences of Less Frequent Data Collection

During this initial step of the project, information will be collected only once. If formative data collection is necessary to inform potential additional evaluations in the future, we will submit additional Gen ICs specific to those evaluations. Respondents for future Gen ICs would not be the same as those included in this Gen IC.

A7. Special Circumstances

There are no special circumstances for this data collection.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the generic clearance for formative information collection. This notice was published on September 15, 2014, Volume 79, Number 178, page 54985, and provided a sixty-day period for public comment. The second notice was published on January 9, 2015, Volume 80, Number 6, page 1420, and provided a thirty-day period for public comment. ACF did not receive any comments

Consultation with Experts Outside of the Study

The evaluation team will be consulting with various experts in federal agencies before we begin the information collection.

A9. Incentives for Respondents

No incentives for respondents are proposed for this information collection.

A10. Privacy of Respondents

For the informal discussions that are part of the Gen IC, no personal identifying information beyond name and professional affiliation (e.g., name of the academic/research institution, name of the State, etc.) will be sought. Discussants will be told that their conversations will be kept private to the fullest extent of the law and that it is expected that their name and affiliation will only be included in summary information provided to ACF. ACF staff may participate in telephone or on-site discussions. Discussants will be told that, to the extent allowable by law, individual identifying information will not be disseminated publicly.

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

As part of the field assessment, up to 100 burden hours will be spent by policy experts/researchers, State and local administrators, and program staff. Respondents will participate in semi-structured interviews of varying lengths. The time per response is estimated at 1 hour for the researchers/policy experts, 2 hours for State and local TANF administrators, and 2.5 hours for program staff. See the table below for estimated burden for each type of instrument.

Total Burden Requested Under this Information Collection

Instrument	Total/Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Discussion Guide for Researchers and Policy Experts	16	1	1	16	$64.03	$1,024.48
Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice	27	1	2	54	$47.76	$2,579.04
Discussion Guide for Program Staff	12	1	2.5	30	$24.36	$730.80
Estimated Annual Burden Total				100		$4,334,32

Total Annual Cost

The annualized cost burden to respondents is based on the estimated burden hours and the assumed hourly wage rate for respondents. The assumed wage rate is based on the May 2016 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm). The rate used for researchers and policy experts, $64.03, is equivalent to management, scientific, and technical consulting services under SOC code 19-3011. The rate used for State and local administrators, $47.76, is equivalent to the local government managers under SOC code 11-1021. The rate used for program staff $24.36 is equivalent to local government workers under SOC code 21-1023. The estimated annualized cost is $4,334.32.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $346,786. The annual cost is the same, since the data collection will occur within one year.

A15. Change in Burden

This is a new Gen IC under the Formative Generic (0970-0356).

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

The information collected will be used to inform the research design, recruitment, and sampling strategies used for the first phase. Discussions with informants will take place starting in January 2018, pending OMB approval, and may continue through December 2018. The data collected under this IC may be published if it is of methodological interest. Plans for use of later data collected during the study will be further explained in a subsequent package.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1See, for example, employmentstrategies.acf.hhs.gov.

10