part a: Reentry Employment Opportunities (REO) Evaluation
OMB No. 1290-0NEW
april 2019
The U.S. Department of Labor’s (DOL) Chief Evaluation Office (CEO) is undertaking the Reentry Employment Opportunities (REO) Evaluation. The overall aim of the evaluation is to determine whether the REO programs improve employment outcomes and workforce readiness for young adults and adults with previous criminal justice system contact. CEO contracted with Mathematica Policy Research and its subcontractor, Social Policy Research Associates (SPR), to conduct this evaluation. The evaluation will include an implementation study and an impact study. This package requests clearance for two data collection instruments, which are included as supporting documents:
Grantee survey
Baseline information form (BIF)
These data collection instruments have been reviewed and approved by Health Media Lab Institutional Review Board. A future information collection request will include protocols for in-person site visit data collection through interviews with grantees, subgrantees, and partners, as well as focus groups with participants and employers, and a participant follow-up survey.
This evaluation is authorized by section 169 of the Workforce Innovation and Opportunity Act (WIOA), which authorizes research and evaluations to improve the management and effectiveness of workforce programs and activities such as the Reentry Employment Opportunity (REO) programs. CEO undertakes a learning agenda process each year to identify Departmental priorities for program evaluations. This evaluation was prioritized as part of that process in FY 2017. Division H, Title I, section 107 of Public Law 115-31, the “Consolidated Appropriations Act, 2017” authorizes the Secretary of Labor to reserve not more than 0.75 percent from special budget accounts for transfer to and use by the Department’s CEO for departmental program evaluation. Further, 29 USC 3224a (1), authorizes the Secretary of Labor to conduct ongoing evaluation of programs and activities to improve the management and effectiveness of these programs.
Understanding the effectiveness of the DOL-funded programs serving justice-involved young adults and adults requires a rigorous evaluation. CEO has contracted with Mathematica Policy Research and its subcontractor, SPR, to conduct (1) an implementation study to provide an overall assessment of the recent REO initiative investments and provide context for the impact study and (2) an impact study to build the evidence base about the effectiveness of employment services for young adults and adults with current and former justice involvement. The REO program evaluation’s data collection efforts that are part of this clearance request will involve grantees and subgrantees1 that received grants during 2016, 2017, and 2018.2
The REO implementation study will address the following key research questions:
How were programs implemented, and what factors influenced implementation?
What are the variations in the model, structure, partnerships, and services of the REO grants?
How did implementation vary by organization type (such as an intermediary organization that operates in more than one state or a community-based organization) and target population?
4. What key program elements appear to be promising?
The REO impact study will address the following key research questions:
What impact do grantees or strategies implemented by grantees have on participants’ outcomes?
Does program effectiveness vary by grantee characteristics, such as population served and services offered?
To what extent do impacts vary across selected subpopulations, such as age group and type of offense?
2. Overview of the data collection
Understanding the implementation and effectiveness of the REO grants requires data collection from multiple sources. For the implementation study, data collection will include key informant interviews with grantees, subgrantees, and partners; individual-level program data, a grantee survey, and focus groups with employers and program participants. For the impact study, the study team will collect baseline information from study participants at program application. Outcome data will come from a follow-up survey of a sample of study participants and administrative earnings data and criminal justice system data for all impact study participants. The data collection activities covered by this clearance include the grantee survey and the BIF.
The data collection activities that are part of this request are:
Grantee survey. As part of the implementation study, an electronic survey will be fielded to approximately 97 grantees to obtain information about the REO grantees’ approach to project management, recruitment and outreach, and service delivery. This survey will include a set of common questions to lead to insights about variations across grantees and grant programs and to contextualize the impact and implementation study data.
Baseline information form (BIF). As part of the impact study, at study intake at impact study sites, participants eligible to receive services will provide demographic characteristics and other information on a BIF. In addition to sample member contact information, the BIF will also collect information on the baseline characteristics of sample members, such as sex, race/ethnicity, education level, and prior justice system involvement. In addition, the BIF will capture social network use and alternative contact information for up to three relatives or friends who might know how to contact the sample member. Whenever possible, BIFs will be collected electronically through the study’s web-based random assignment system. Consent to participate in the research study will be obtained from all impact study participants. To fully ensure informed consent, the study team will collect written consent from all REO evaluation participants (see Supplementary Documents for a copy of the consent form).
DOL will use the data collected through the activities summarized in this request to systematically describe the REO grants, including their organization, administration and management, services, delivery structures and processes; participants served; and common implementation challenges. These data and the study team’s qualitative and descriptive analyses will provide DOL and other policymakers with important information to guide management decisions and future planning for such grant programs. Without conducting the two data collection activities described in this request, a comprehensive evaluation of the REO grant programs cannot occur.
Instruments used to collect data for the evaluation and their proposed uses are summarized in Table A.1.
Table A.1. Data collection instruments included in the request
Data collection instrument |
How study team will use the data |
1. Grantee survey |
This survey will be used to gather common information about organizational settings and intervention characteristics for all REO grantees. |
2. Baseline information form (BIF) |
Descriptive analyses of the BIFs will be used to (1) ensure the program and control groups are well balanced on their characteristics at the time of random assignment, (2) form subgroups for the impact estimation, and (3) adjust for nonresponse if necessary. Data will also be used to locate participants for follow-up data collection. |
The final implementation study and impact study reports will address all the evaluation’s research questions and synthesize data across sources.
Whenever possible, the data collection efforts will use advanced technology to reduce burden on study participants and on staff at participating agencies. For the implementation study, the grantee survey will be available electronically to allow respondents to complete the questionnaire on their own schedule, in multiple sittings, and without having to return any forms by mail.
For the impact study, all program staff involved in sample member intake will have access to a study-specific web-based system, to collect baseline sample member information and conduct random assignment. The web-based system includes checks for duplicates and for eligibility, as well as assignment to the treatment or control group. Two advantages of using a cloud-based infrastructure are that (1) there is nothing for grantees or subgrantees to install and (2) there are many options for accessing the system, reducing the technological burden on program staff.
The data being collected for the REO evaluation are not otherwise available from existing sources. Before administering the grantee survey, the study team will gather pertinent data from the grantee applications and from American Job Center and Local Workforce Development Board websites and other publicly available sources. . Similarly, the BIF has been reduced to items necessary for the evaluation and that cannot be systematically obtained from any other source.
The data collection effort does not involve small businesses or other small entities.
The evaluation represents an important opportunity for CEO to add to the growing body of knowledge about what works to serve participants with prior justice involvement. Not collecting data on grantees through the grantee survey will limit the evaluation team’s ability to fully understand the context of the REO programs and place the evaluation’s findings into that context. Without the implementation study data, the study team would be less able to identify strategies that grantees use to provide services to individuals with justice system contact. In addition, the study team would be less able to identify potential reasons for differences across grantees in impact estimates, if such differences are found.
The BIFs also are necessary to conduct a rigorous evaluation of DOL-funded REO programs. Without collecting baseline information on study participants via the BIFs, the study team will not be able to ensure that random assignment was conducted appropriately. The lack of baseline information would also limit the ability to estimate subgroup impacts and conduct a nonresponse analysis if one is needed.
No special circumstances apply to this data collection.
The 60-day notice FR 83 FR 16128 to solicit public comments was published in the Federal Register on April 13, 2018. One comment was received by an individual seeking information on programs offered to reentering citizens. The comment was not responsive to the ICR and was forwarded to the Reentry Employment Opportunities Office to respond.
Consultation on the research design and data needs is being coordinated by the study team and involves discussions with experts. Table A.2 lists the people who were consulted in preparing this submission to OMB:
Table A.2. Individuals who were consulted for the REO evaluation
Mathematica
Policy Research |
Project director |
Dr.
Karen Needels |
|
Survey director |
Dr.
Jillian Stein |
|
Site selection task lead |
Dr.
Johanna Lacoe |
|
Quality assurance reviewer |
Dr.
Jillian Berk |
Social
Policy Research Associates |
Principal investigator |
Dr.
Andrew Wiegand |
|
Implementation study task leader |
Ms.
Hannah Betesh |
|
Administrative data collection task lead |
Mr.
Christian Geckeler |
No payments will be made to individuals completing the grantee survey or the BIF. The $25 gift card referenced in the Participant Consent Form is an incentive to complete the participant follow-up survey that will be submitted through a future information collection request as mentioned above.
Personally identifying information collected will be kept private to the extent permitted by law. The study team complies with DOL data security requirements by implementing security controls for processes that it routinely uses in projects that involve sensitive data. Further, the study is being conducted in accordance with all relevant regulations and requirements. The study team secures personally identifiable information and other sensitive project information and strictly controls access on a need-to-know basis.
During data collection, the study team will take efforts to protect data and ensure that respondents understand the extent to which information can be kept private.
The BIF will collect baseline information on individuals who have consented to participate in the impact study. Date of birth, address, and telephone numbers are needed to identify and contact sample members. The BIF will also collect information on characteristics of sample members, such as sex, race/ethnicity, and education level, to ensure that random assignment was conducted correctly and/or to enhance the impact estimates. Finally, the BIF will capture social network use and alternative contact information for up to three relatives or friends who might know how to contact the sample member. This type of information is routinely collected as part of enrollment in some programs and is necessary to be able to examine outcomes and impact by subgroups (for example by age or sex) and to locate study participants.
The BIF does include questions that some respondents might find sensitive. These questions ask about delinquent and criminal activities, including arrests and juvenile detentions, and physical and mental health. Because participants’ prior justice contact is a precondition of receipt of REO services, asking about justice involvement as part of the study application process will not ask participants to reveal anything more than they would likely have to reveal to apply for services. Furthermore, collection of this information, although it is sensitive, is critical for the evaluation, and this information cannot be obtained through other sources. The extent of prior involvement with the juvenile and/or adult criminal justice system will be an important characteristic for describing study sample members and will enable subgroup analyses for the impact study. Similar questions have been included in past studies such as in the Evaluation of the Linking to Employment Activities Pre-release (OMB control number # 1291-0008) and the National Guard Youth ChalleNGe Job ChalleNGe Evaluation (OMB control number # 1291-0009) without any evidence of significant harm. Similarly, when asked, pretest respondents all stated that they were comfortable answering the questions in the BIF and that reporting things that one might find sensitive, such as criminal history, were events that they were used to reporting.
As described earlier, all sample members will be provided with assurances of privacy before the completion of study enrollment forms and random assignment.
Table A.3 includes assumptions about the annual number of respondents expected, the average number of responses per respondent, the average hours of burden per response, the annual burden hours estimated, the time value assumed for respondents, and the total annualized monetary burden hours for the grantee survey and the BIFs. All of the activities covered by this request are annualized over three years. Here, we summarize the burden estimates rounded to the nearest whole number for each of the two data collection activities:
Grantee survey. The study team will administer the grantee survey to approximately 97 grantees. Each grantee respondent is estimated to spend approximately 20 minutes to complete the survey, for a total burden of 32 hours (or 11 per year).
BIF. The study team expects to collect a total of approximately 3,780 BIFs across the impact study sites. Each BIF is estimated to take 10 minutes to complete for a total of 642 burden hours (or 214 burden hours per year when rounding to the nearest whole number).
Table A.3. Annualized estimates of burden hours
Type of Instrument |
Annual number of respondents |
Number of responses per respondent |
Average burden time per response (hours) |
Annual estimated burden hours |
Time valuea |
Annual monetized burden hours |
Grantee surveyb |
32 |
1 |
0.34 |
11 |
$33.91 |
$373 |
Baseline information formc |
1,260 |
1 |
0.17 |
214 |
$7.25 |
$1,552 |
Total |
1,292 |
|
|
225 |
|
$1,925 |
Table note: Numbers are rounded to the nearest whole number for all columns other than the “time value” column.
a The average hourly wages were obtained from the U.S. Bureau of Labor Statistics, National, State, Metropolitan, and Nonmetropolitan Area Occupational Employment and Wage Estimates, 2017 (accessed at https://www.bls.gov/oes/current/oessrcma.htm on April 18, 2018). Estimates of administrators’ and managers’ wages are based on the average wages for “social and community service managers” ($33.91). Monetized estimates for participants were assumed to be the federal minimum wage of $7.25.
b The grantee survey data will be collected from 97 grantees. It is expected that it will take an average of 20 minutes for grantees to complete the survey.
c The baseline information forms will be collected from the full impact study sample of 3,780 study participants. It is expected that it will take an average of 10 minutes for participants to complete the form.
There will be no direct costs to respondents for the REO evaluation other than their time.
The total annualized cost to the federal government is $1,649,965. Costs result from the following categories:
The estimated cost to the federal government for the contractor to carry out this study is $4,724,978 for all data collection, administrative records collection, and analysis and reporting. Annualized over three years of data collection, this comes to $1,574,993 per year.
The annual cost borne by DOL for federal technical staff to oversee the contract is estimated to be $74,972. We expect the annual level of effort to perform these duties will require 200 hours for one Washington, DC–based Federal GS 14 step 2 employee earning $56.74 per hour, and 200 hours for one Washington, DC-based Federal GS 15 step 2 employee earning $66.74 per hour. (See Office of Personnel Management 2018 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/DCB_h.pdf.) To account for fringe benefits and other overhead costs, the agency has applied a multiplication factor of 1.6. Thus [(200 hours x $56.74) + (200 hours x $66.74)] x 1.6 = $39,514.
This is a new information collection.
The evaluation plan includes an impact study report and an implementation study report in the final year of the project.
Implementation study report. In September 2022, the evaluation team will complete a report describing the findings from the implementation study. This report will document how sites were selected for the evaluation, as well as the characteristics of sites that participated. The report will also discuss the characteristics of the program participants, the flow of participants through the programs, the delivery of services, participation rates, and any challenges to serving participants.
Final report. In September 2022, the evaluation team also will complete the final report documenting impacts of access to REO grant services on participants’ outcomes. Likely outcomes will include employment and earnings, participation in education and training, the attainment of educational credentials, criminal justice involvement, family status and living arrangements, positive behaviors and activities, risky behaviors, health status and other measures of well-being. If feasible, this report will also examine effects for key subgroups and present an analysis of the effectiveness of certain program components.
The OMB approval number and expiration date will be displayed or cited on all forms completed as part of the data collection.
No exceptions are necessary for this information collection.
1 A subset of the REO grants were awarded to intermediary organizations that competitively selected subgrantees. Parts of the evaluation will include direct grantees and the subgrantees of the intermediaries.
2 This clearance request does not include year 2015 grantees because it is expected that their grants will expire prior to receipt of approval for the request. The funding opportunity announcement for year 2018 grants can be found at https://www.grants.gov/web/grants/view-opportunity.html?oppId=302573.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |