The Building Evidence on Employment Strategies for Low-Income Families Project (BEES)
OMB Information Collection Request
0970-0356
Supporting Statement
Part A
July 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Tiffany McCormack
Carli Wulff
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for formative data collection activities for The Building Evidence on Employment Strategies for Low-Income Families Project (BEES). This request is for additional burden hours for a previously approved collection that falls within ACF’s existing formative generic OMB clearance (OMB #0970-0356). This submission seeks OMB approval for three data collection instruments that will be used as part of the field assessment and to inform the intervention and site selection process:
Discussion Guide for Researchers and Policy Experts
Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice
Discussion Guide for Program Staff
This submission also seeks OMB approval for an email designed for contacting potential informants to arrange a discussion. ACF will submit additional information collection requests (ICRs) as part of this project, including a full ICR for evaluations of the selected programs.
A1. Necessity for the Data Collection
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for a series of tailored, semi-structured discussions under ACF’s existing formative generic OMB clearance (OMB #0970-0356) to inform the selection of interventions and sites for the BEES project. BEES is a large-scale project, sponsored by the Office of Planning, Research, and Evaluation (OPRE), that aims to build evidence on interventions designed to promote employment and economic security among TANF recipients and other disadvantaged groups, including those with substance/opioid use disorder and mental health challenges. If all options in the project are exercised, BEES may include up to 21 rigorous evaluations of employment-focused interventions. We are currently focusing on the first two phases, which are expected to include 7 evaluations. We will submit additional generic information collections (Gen ICs), if needed, to inform potential additional evaluations in the future. After this information-gathering step, we plan to submit a full ICR for future data collection for the evaluation, including baseline data, survey data, implementation data, and follow-up data for estimating impacts.
This Gen IC includes an initial scan to identify high priority interventions and sites to recommend for rigorous evaluation. The Gen IC is expected to begin upon OMB approval and continue for approximately 11 months. The Gen IC will inform the research design, and recruitment and sampling strategies used for the first two phases of the project. This submission seeks OMB approval for three data collection instruments that will be used as part of the field assessment to inform the intervention and site selection process:
Discussion Guide for Researchers and Policy Experts
Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice
Discussion Guide for Program Staff
Overview of BEES
Note: This section goes beyond a description of this current Gen IC by providing background on and plans for the full BEES project. This Gen IC is an initial step, with subsequent ICRs expected to be submitted in the future to cover data collection for the evaluation once sites have been selected, including baseline data, survey data, implementation data, and follow-up data for estimating impacts.
BEES is a project sponsored by OPRE within ACF under contract to MDRC and its subcontractors, Abt Associates and MEF Associates. BEES aims to add to the knowledge base of employment interventions for TANF recipients and other disadvantaged groups, including those with substance/opioid use disorder and mental health challenges. To maximize learning, the BEES project will build on the substantial body of evidence on employment interventions that have been developed over the past 40 years, as well as the results from key ongoing studies that will be emerging in the next year or two (see section A.4).
Rigorous studies have demonstrated that many types of interventions can improve labor market outcomes for disadvantaged groups. Job search and subsidized employment can increase employment and earnings in the short-term, while earnings supplements can increase both employment and income, at least while supplements remain in place. Career pathways models can produce lasting earnings gains for those who meet program entrance criteria.1 However, despite this extensive body of evidence, there remain many open questions: lengthier programs consistently struggle with attrition, and it is important to understand the cost effectiveness of various supports for participants; relatively little is known about the best structures and pedagogy for delivering occupational training; and there are few training models that can accommodate individuals with low literacy and numeracy skills.
In addition to these programmatic challenges, many disadvantaged groups also experience behavioral health issues – notably substance/opioid use and mental health disorders – that further hinder their ability to obtain stable employment. There is strong evidence of the efficacy of specific treatment approaches for substance/opioid use disorder, depression, anxiety, and other conditions that can act as barriers to steady employment. The challenge – and a promising area of inquiry for BEES – is to learn how best to promote steady participation in treatment, and how to effectively integrate treatment with employment services.
Current phase
BEES is currently in the planning and design phase in which decisions regarding the interventions to be tested and other aspects of the design will be made. In addition, the recruitment and sampling strategies will be formulated during this phase of the study to ensure that we are able to efficiently and successfully recruit employment interventions for TANF recipients and other disadvantaged groups.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose of Survey and Data Collection Procedures
Overview of Purpose and Approach
The purpose of this current Gen IC is to help ACF identify high priority interventions for rigorous evaluation in the first two phases of BEES. After this information-gathering step, we plan to submit a full ICR for future data collection for the evaluation, including baseline data, survey data, implementation data, and follow-up data for estimating impacts.
This Gen IC request includes three discussion guides that will be used in telephone and/or on-site interviews with select categories of informants: Discussion Guide for Researchers and Policy Experts (Attachment A); Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice (Attachment B); and Discussion Guide for Program Staff (Attachment C). The three guides cover similar topics but are differentiated by the level of detail appropriate for each group of informants. We intend to use the information gathered through these discussions to inform the selection of interventions as well as the research design, recruitment, and sampling strategies used for the first two phases. This approach will be repeated for potential later phases.
Research Questions
This Gen IC aims to address the following research questions:
Based on prior and emerging research, and current practice, what types of programs and approaches are most promising in each domain identified as a priority by ACF, states, or other stakeholders?
Which specific interventions in each domain are ready to move to the next level of evidence?
The full BEES Project aims to address the following additional research questions:
What is the impact of the specific interventions, both on critical intermediate measures of success and on longer-term measures of employment and economic security?
Going beyond the averages, what can be learned about the impact of specific program components, or the impact of interventions for important subgroups of the target population?
What lessons on program implementation and cost can shed light on the impact results and help facilitate the expansion or replication of successful interventions?
What lessons can be drawn across the project about the characteristics and implementation of successful interventions?
Study Design
BEES aims to identify interventions and sites that ensure that the evaluation will build on past and ongoing research and reflect the priorities of ACF, states, and other stakeholders. The first two phases of BEES will include 7 evaluations of employment-focused interventions; the full project (with up to four phases) may include up to 21 rigorous evaluations. Each evaluation will include research on the implementation of the intervention, which will focus on program practices and implementation fidelity, participant engagement, context, and the counterfactual. Each evaluation will also include research on the intervention’s effects on participants’ employment outcomes, based on data sources including administrative data and surveys.
To assess the impact of each program on participants’ employment outcomes, we plan to primarily rely on individual-level quarterly earnings data provided directly by states and/or the National Directory of New Hires (NDNH), maintained by the Office of Child Support Enforcement in ACF. The research team has successfully secured this data for many prior evaluations. They have experience obtaining quarterly earnings data from most of the states in the U.S., and they have used NDNH data in several previous ACF-sponsored projects.
While employment-related outcomes are the primary focus of the project, we may supplement employment-related data with additional data about service receipt for some evaluations, depending on the feasibility of obtaining such data. Specifically, we anticipate that some evaluations will assess the impact of the program on service receipt using individual-level administrative data on receipt of TANF cash assistance and SNAP benefits. As with employment-related data, the research team has successfully obtained TANF and SNAP data directly from states in many previous projects. (The data is often maintained by the same agency that maintains employment-related data.) However, to be clear, the use of such data would be supplemental and is not the primary focus of the evaluation. In addition, the team will of course only rely on such data if appropriate informed consent from study subjects is obtained.
Randomized controlled trials (RCTs) are the preferred method for estimating impacts in the BEES evaluations but other rigorous designs will be considered when necessary. A future ICR will include details about the full study design.
This Gen IC request focuses on collecting information to inform the selection of interventions and sites for the first two phases of BEES. In this IC, we expect to identify both interventions and sites that are potential candidates for rigorous testing in the first phase and others that would require some additional developmental work (e.g., adapting a well-studied intervention to a TANF population). The first three evaluations in BEES will test interventions that our reconnaissance determines to be most ready for rigorous testing, most likely at the site where they currently operate. The latter category of interventions might be tested in the second phase of BEES or later phases, along with others identified in potential future scans.
For this Gen IC, the evaluation team will conduct formative data collection through discussions with select informants. Three discussion guides are included in this Gen IC request: Discussion Guide for Researchers and Policy Experts (Attachment A); Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice (Attachment B); and Discussion Guide for Program Staff (Attachment C). The three guides cover similar topics but are differentiated by the level of detail appropriate for each group of informants. The specific subset of questions asked will depend on the informant’s expertise and background. For example, a researcher or policy expert might be asked about promising interventions and models, while a state-level TANF administrator might be asked about priority areas of programming in their state. We intend to use the information gathered through these discussions to inform selection of interventions and sites as well as the research design, recruitment, and sampling strategies used for the two first phases. This approach will be repeated for potential later phases.
The evaluation team will identify informants through a purposeful, snowball sampling process that draws from recommendations from the Contracting Officer’s Representatives (CORs) and other staff in OPRE, referrals from other informants, and Internet searches of employment based interventions. Agencies that may be consulted include: ASPE, SAMHSA, National Institute of Drug Abuse, National Institute of Mental Health, DOL (Chief Evaluation Office, Employment and Training Administration, Office of Vocational Rehabilitation and Employment, Office of Disability and Employment Policy), Department of Education (Institute for Educational Sciences and the Office of Vocational and Adult Education), Department of Justice (Office of Justice Programs and National Institute of Justice), HUD (Policy Development and Research), Social Security Administration (Office of Disability), and Department of Agriculture (Food and Nutrition Service).
The evaluation team plans to interview informants through a combination of phone and in-person discussions conducted one-on-one or in small groups. Since discussions with informants will be conducted on an ongoing basis, we expect that the selection of informants will also be refined periodically, based on the information obtained. In identifying and selecting informants, we will contact individuals who will add new information, based on our extant knowledge base and gaps where we hope to gain further insights, in order to minimize potential burden on Gen IC participants. Priority candidates will be identified based on their field of expertise, purpose of engagement, geographic/locality representation, type of organization they represent, and other key information. There are no quantitative components to this formative data collection effort.
Universe of Data Collection Efforts
Informants in this Gen IC will include researchers and policy experts in the fields of workforce development, substance/opioid use disorder treatment, and mental health; state and local administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice; and program staff of employment interventions for TANF recipients and other disadvantaged groups, including those facing mental health and addiction challenges. The information will be collected through semi-structured interviews that will cover a range of topics, including the purpose of and content of employment services, participation in/targeting of employment services, and, when appropriate, staffing of the informants’ organization and their interest in possibly participating in the study. The discussion guides for these three groups of respondents vary slightly, but all focus on these topics. The discussion guides were approved in January 2018 for use in BEES and are similar to guides that have been approved and used successfully in prior OPRE studies, including the Job Search Assistance Strategies Evaluation (OMB #0970-0440), Subsidized and Transitional Employment Demonstration (OMB #0970-0413) and the Assets for Independence Evaluation (OMB #0970-0414).
A3. Improved Information Technology to Reduce Burden
The evaluation team plans to use improved information technology wherever possible. When information is available from the internet, it will supplement requests for information. Whenever possible, discussions will be done by telephone to reduce burden on the respondents. We will also use technology to reduce burden by initially reaching out to potential informants via email, providing information about the BEES project and the purpose of the call. The email is included in this Gen IC request as a supplemental document (Attachment D).
A4. Efforts to Identify Duplication
The BEES project is designed to fill gaps in the substantial body of evidence on employment interventions that has been developed over the past 40 years. The evaluation team will begin by carefully reviewing the existing literature and talking to federal officials in order to identify research questions that will add to the knowledge base rather than duplicating what is already known. For example, in the career pathways area, the project might focus on populations that were not well served by previous tested interventions. Similarly, the evidence-based Individual Placement and Support (IPS) supported employment model might be tested for new populations. In the area of substance/opioid use disorder, we believe there is little systematic information about current employment interventions targeted specifically to this population. By reviewing the literature and talking to federal officials before starting the information collection, we will ensure that we will not spend informants’ time discussing topics that are already well known.
A5. Involvement of Small Organizations
We expect involvement of small organizations for the field assessment and site selection to be limited. For the most part, the evaluation team will be consulting with individual experts, and systems and program administrators. If the researchers reach out to small organizations, the burden will be minimized for respondents by restricting the interview length to the minimum required, by conducting telephone discussions at times convenient for the respondents, and by requiring no record-keeping or written responses on the part of the programs.
A6. Consequences of Less Frequent Data Collection
During this initial step of the project, information will be collected only once. If formative data collection is necessary to inform potential additional evaluations in the future, we will submit additional Gen ICs specific to those evaluations. Respondents for future Gen ICs would not be the same as those included in this Gen IC.
A7. Special Circumstances
There are no special circumstances for this data collection.
A8.
Federal Register Notice and Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. The second notice was published on March 30, 2018, Volume 83, Number 62, page 13758, and provided a thirty-day period for public comment. ACF did not receive any comments
Consultation with Experts Outside of the Study
The evaluation team will be consulting with various experts in federal agencies before we begin the information collection.
A9. Incentives for Respondents
No incentives for respondents are proposed for this information collection.
A10. Privacy of Respondents
For the informal discussions that are part of the Gen IC, no personal identifying information beyond name and professional affiliation (e.g., name of the academic/research institution, name of the State, etc.) will be sought. Discussants will be told that their conversations will be kept private to the fullest extent of the law and that it is expected that their name and affiliation will only be included in summary information provided to ACF. ACF staff may participate in telephone or on-site discussions. Discussants will be told that, to the extent allowable by law, individual identifying information will not be disseminated publicly.
A11. Sensitive Questions
There are no sensitive questions in this data collection.
A12. Estimation of Information Collection Burden
Previously Approved Information Collection
Total Burden Previously Approved
The originally approved Gen IC for BEES was for 100 hours of burden. The burden hours were to be spent by policy experts/researchers, State and local administrators, and program staff in semi-structured interviews of varying lengths.
Burden Remaining from Previously Approved Information Collection
16 hours of the 100 remain.
Newly Requested Information Collection
As part of the field assessment, up to 407 total additional burden hours will be spent by policy experts/researchers, State and local administrators, and program staff. Respondents will participate in semi-structured interviews of varying lengths. The time per response is estimated at 1 hour for the researchers/policy experts, 2 hours for State and local TANF administrators, and 2.5 hours for program staff. See the table below for estimated burden for each type of instrument. It includes 16 hours that remain from the original request and 407 newly requested hours, 423 total hours. Note that the total burden has been annualized over the three year approval period for the overarching umbrella formative generic.
Total Burden Requested Under this Information Collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
Discussion Guide for Researchers and Policy Experts |
24 |
8 |
1 |
1 |
8 |
$60.47 |
$483.76 |
Discussion Guide for State and Local Administrators of systems such as TANF, SNAP, WIOA, behavioral health, and criminal justice |
72 |
24 |
1 |
2 |
48 |
$49.12 |
$2,357.76 |
Discussion Guide for Program Staff |
102 |
34 |
1 |
2.5 |
85 |
$24.44 |
$2,077.40 |
Estimated Annual Burden Total |
|
|
|
|
141 |
|
$4,918.92 |
Total Annual Cost
The annualized cost burden to respondents is based on the estimated burden hours and the assumed hourly wage rate for respondents. The assumed wage rate is based on the May 2017 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm). The rate used for researchers and policy experts, $60.47, is equivalent to management, scientific, and technical consulting services under SOC code 19-3011. The rate used for State and local administrators, $49.12, is equivalent to the local government managers under SOC code 11-1021. The rate used for program staff $24.44 is equivalent to local government workers under SOC code 21-1023. The estimated annualized cost is $4,918.92. The total cost over the three years is $14,756.76.
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
The total cost for the data collection activities under this current request will be $250,000. The annual cost is the same, since the data collection will occur within one year.
A15. Change in Burden
This is a request for additional hours for a previously approved collection under the Formative Generic (0970-0356).
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
The information collected will be used to inform the research design, recruitment, and sampling strategies used for the first two phases. For this request, discussions with informants will take place starting in August 2018, pending OMB approval, and may continue through June 2019. (Discussions under the original generic approval began in January 2018.) The data collected under this IC may be published if it is of methodological interest. Plans for use of later data collected during the study will be further explained in a subsequent package.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
1See, for example, employmentstrategies.acf.hhs.gov.
| File Type | application/msword |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |