STED-ETJD Supporting Statement Part A May 2013 Revision Final



SUBSIDIZED AND TRANSITIONAL EMPLOYMENT DEMONSTRATION (STED) AND ENHANCED TRANSITIONAL JOBS DEMONSTRATION (ETJD)

SUPPORTING STATEMENT A

REQUEST FOR OMB CLEARANCE

OMB No.: 0970-0413

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

October 2012

Revised May 2013¹

TABLE OF CONTENTS

A. Justification 5

A1. Circumstances Necessitating Data Collection 5

A1.1 Background and Policy Context 5

A1.2 Overview of the STED and ETJD Evaluations 11

A2. How, by Whom, and for What Purpose Are Data to be Used 12

A3. Use of Information Technology for Data Collection to Reduce Respondent Burden………………………………………………………………………………………….. 24

A4. Efforts to Identify Duplication 25

A5. Burden on Small Business 26

A6. Consequences if Data Collection is Not Conducted 26

A7. Special Data Collection Circumstances 26

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission 26

A9. Justification for Respondent Payments 27

A10. Confidentiality 28

A11. Questions of a Sensitive Nature 29

A12. Estimates of the Hour Burden of Data Collection to Respondents 29

A13. Estimates of Capital, Operating, and Start-Up Costs to Respondents 31

A14. Estimates of Costs to Federal Government 32

A15. Changes in Burden 32

A16. Tabulation, Analysis, and Publication Plans and Schedule………………… 32

A17. Reasons for Not Displaying the OMB Approval Expiration Date 33

A18. Exceptions to Certification Statement 33

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Sampling B-2

B2. Procedures for Collection of Information B-2

B3. Maximizing Response Rates B-5

B4. Pilot Testing B-8

B5. Consultants on Statistical Aspects of the Design B-9

LIST OF EXHIBITS

1.1 Conceptual Framework for the STED and ETJD Evaluations…………………………..10

1.2 Annual Burden Estimates………………………………………………………………..30

2.1 Follow-Up Survey Sample Sizes………………………………………………………B-2

2.2 Minimum Detectable Effects, Per Site, Per Survey……………………………………B-4

LIST OF APPENDICES

1. Background on Subsidized and Transitional Employment Programs and Evaluations

2. Informed Consent Form

3. Contact Information Form

4. Baseline Information Form

5. Tracking Letters

6. Program Staff Implementation Questionnaire

7. Participant Implementation Questionnaire

8. Worksite Supervisor Implementation Questionnaire

9. Employer Implementation Questionnaire

10. Interview Guides

11. Participant Focus Group Discussion Guide

12. Worksite Observation Guide

13. Program Staff Time-Use Worksheet

14. Cost Data Collection Protocol

15. Six-month Follow-Up Survey

16. 12-month Follow-Up Survey

17. List of Technical Working Group (TWG) members

18. 60-Day Federal Register Notice

19. Implementation Questionnaire Screenshots

SUPPORTING STATEMENT

Subsidized and Transitional Employment Demonstration (STED)

And

Enhanced Transitional Jobs Demonstration (ETJD)

PART A. Justification

Revised to include alternate survey instruments, which were developed for the youth/young adult STED sites. All changes from the original approved package are highlighted.

This Supporting Statement provides information for a proposed combined data collection effort to determine the effectiveness of subsidized and transitional employment programs being evaluated by the Administration for Children and Families (ACF) Subsidized and Transitional Employment Demonstration (STED) and the U.S. Department of Labor Enhanced Transitional Jobs Demonstration (ETJD). Both of these evaluations are being conducted by MDRC and its subcontractors MEF Associates, Branch Associates, D.I.R. Inc., and Abt Associates on behalf of ACF and ETJD. Both evaluations will ultimately consist of seven sites; however, since two of the programs funded through the ETJD project will be evaluated through the STED project, we anticipate a total of twelve sites will participate in the two projects. All seven of the sites that are part of the ETJD project (including two which are being evaluated as a part of STED) have already begun enrolling participants into the evaluation.

The primary goal of both projects is to increase our knowledge about the efficacy of programs aimed at helping target populations secure and maintain unsubsidized employment. ACF and ETA have agreed to collaborate on the design of data collection instruments to promote consistency across the project.

A.1.1 Background and Policy Context

Subsidized employment programs aim to use public funds to create or support work opportunities for people who would not otherwise be employed. A variety of different subsidized employment models have been implemented or tested in the U.S. since the 1930s. Most subsidized employment programs are designed, at least in part, to provide work-based income support to people who cannot find regular jobs. During some economic downturns, subsidized employment programs have operated on a large scale and served a fairly broad range of people, such as the New Deal’s Works Progress Administration, the 1970s Public Service Employment (PSE) program, and some of the state programs operated using the Temporary Assistance for Needy Families Emergency Contingency Fund in 2009-2010. More commonly, subsidized employment programs have been relatively small and narrowly targeted to people who have difficulty finding work even when the labor market is strong. Some of these programs aim to achieve important indirect goals that are seen as linked to higher employment, such as reduced recidivism for former prisoners, while others have focused on improving public facilities or infrastructure, or providing community services.

A subset of subsidized employment programs have explicitly aimed to build the employability of economically disadvantaged populations. In these programs, the goal is not only to provide work and income support in the short-term, but also to improve participants’ performance in the regular labor market over the longer-term. Past evaluations of subsidized and transitional employment approaches that share this goal have produced mixed results. Yet, the prevalence, popularity, and potential opportunity of these approaches warrant further study to address questions that remain unanswered. Appendix A discusses the variations in subsidized employment approaches in more detail and highlights the major findings from a variety of studies examining these interventions. It is within this context of increased interest and unanswered research questions that the Administration for Children and Families and the Employment and Training Administration are conducting these current studies (i.e., STED and ETJD).

Both studies will examine the impact of variations in subsidized and transitional employment programs on the outcomes of the targeted populations. The basic research questions for both studies are:

• What effects do subsidized and transitional jobs programs targeted at particularly vulnerable clients have on their populations of interest, such as non-custodial parents, TANF recipients, low-income youth transitioning to the labor force, prisoners re-entering the community, low-income individuals with disabilities, American Indians and Alaska natives, and English Language Learners?

• Do different subsidized employment approaches, i.e., length of employment or content of program, lead to different impacts for participants? Further, what are the effects of programs that place clients directly into subsidized jobs in the regular labor force?

• How can existing subsidized and transitional employment models be adjusted to promote longer-term outcomes for participants?

• How might subsidized employment programs adjust their engagement techniques to promote continued participation and improved employment outcomes for populations facing different challenges to non-subsidized work?

• What effect might a “tiered” subsidized employment model that serves a wide range of people and allocates slots to those who need them most have on clients?

• What are the outcomes for participants in a model that includes occupational skills training or sectoral targeted training?

A Conceptual Framework for the ETJD and STED Projects

The ETJD and STED projects should build from, and aim to improve on, the logic model that underlies subsidized employment programs and on the evaluation results described in Appendix A. Because many of the results have been mixed or discouraging, it is important to look critically at the logic model to identify assumptions that should be reexamined. This process can lead to hypotheses about what might work better, and these hypotheses, in turn, may help to identify the types of models that should be tested.

Most of the subsidized employment programs that have been rigorously tested have assumed that, for certain types of hard-to-employ individuals, structured work experience can:

• Build a work history, increasing the odds that participants can get regular jobs. The models assume that employers will value participants’ experience in the subsidized jobs when evaluating job candidates (in On the Job Training-like models, the participants’ subsidized work experience occurs in the firm that is considering hiring him or her).

• Teach soft skills so that participants are better prepared to hold jobs. The models assume that people are best able to learn to work by working. (Few of the subsidized employment models tested to date have focused on hard skill development).

• Give program staff an opportunity to identify and address workplace problems. The models assume that staff can do this best by observing participants on the job rather than by relying on paper-and-pencil assessments.

• Give people an incentive to stay active in programs. The models assume that participants will be more likely to stay engaged when they are being paid.

The fact that most of these programs have been unable to boost unsubsidized employment suggests that at least some of these assumptions may be flawed. Possible explanations for the generally disappointing results – derived from a mixture of evaluation evidence and practitioner hunches – include the following:

• The programs have not been properly targeted. It is possible that there are groups who could benefit from subsidized employment models, but they have not been targeted in the programs that have been studied – or have been served in small numbers so that subgroup impacts could not be detected. There are hints from past transitional jobs evaluations that those programs may achieve stronger impacts for less employable participants.

• Employers do not value subsidized work experience in program settings. It is possible that, when employers evaluate job candidates, a few weeks or months of experience in a “program” job may not be sufficient to overcome skills deficits or negative perceptions of welfare recipients, former prisoners, or other disadvantaged groups. Perhaps employers would feel differently if the work experience took place in their own worksite.

• Learning soft skills is not enough. Perhaps programs would do better if they also taught occupational skills to help people qualify for higher paying jobs. Some argue that these skills are also taught better in a hands-on work environment. Alternatively, perhaps earnings supplements or other incentives are needed to encourage people to keep low paying unsubsidized jobs. A number of other studies have found positive results for earnings supplements² and MDRC is testing employment retention bonuses as part of the Transitional Jobs Reentry Demonstration project.³

• Work experience is not enough. As noted earlier, most subsidized employment programs provide some ancillary supports to address personal or situational barriers that participants face (for example, substance abuse, heavy child support obligations, child care needs, unstable housing, etc.). However, in most cases, these services are not closely integrated with the subsidized employment program and must be accessed by referrals. It is possible that, for some participants, these issues overwhelm the benefits of work experience and hinder long-term employment outcomes.

• The programs are too short. Perhaps participants would learn more and benefit more if they were able to stay in subsidized jobs for longer periods, or perhaps the programs should not be thought of as sequential. In other words, rather than assuming that participants will move in lockstep from subsidized jobs to unsubsidized jobs, programs could encourage participants to come back to subsidized jobs, at least temporarily, if they lose an unsubsidized job.

• The programs don’t focus enough on the “transition.” Perhaps clients learn valuable soft skills, but the programs do not capitalize on this fact in their job placement and post-placement services. Implementation studies have found that there is tendency for program staff to concentrate on the complex task of creating and/or managing worksites, which potentially diverts attention away from the “back end” of the program.

The recent transitional jobs evaluations have provided hints as to which of these hypotheses seem most relevant, but there is much more to learn. Building on these recent studies, a key goal of the ETJD and STED projects is to integrate process and impact data to gain more detailed insight into why programs do or do not achieve their long term goals.

Framework for the STED and ETJD Evaluations

Exhibit 1.1 illustrates the general framework for the ETJD and STED evaluations in any site. Box A represents the potential target population for the program being tested. The characteristics of the evaluation’s research sample (Box B) are shaped by the program’s recruitment or referral and screening procedures. It will be important for the study to understand how the research sample fits into the broader population from which it is drawn, and how the programs sought to identify who is appropriate for a subsidized job.

Sample members will be randomly assigned to a program group that is offered (or, in some contexts, required to participate in) a program that includes subsidized employment, or to a control group that does not participate in the program being tested. Box C shows the key dimensions of the ETJD/STED programs that will be measured in the implementation study: the model (including both characteristics of the work experience component and other services that are provided), its implementation, and the dosage of services that sample members receive. Control group members will be provided with a resource list informing them of other community services they may access. They will have access to all services except those directly provided through the ETJD or STED project grants. Box D depicts the services that the control group will receive in the community. The “I” indicates that the study will compare the type and amount of services received by the program and control groups in order to understand the treatment difference.

Initially, the evaluation will measure short-term outcomes on employment and earnings, which will likely be driven by the subsidized jobs (see Boxes E and F). The study will also measure key indirect outcomes such as welfare receipt, child support, recidivism, or father-child relationships, with the specifics depending on the target population. Over the longer-term, the study will assess whether the program leads to gains in unsubsidized employment.

Finally, the long box at the far right represents the context in which the program operates. Elements of the context – the labor market, characteristics of the system in which the program operates, the service environment – will shape the target population, the services both groups receive, and the outcomes. For example, if a program operates within the TANF system, then the counterfactual (i.e., control group experience) will be profoundly shaped by the jurisdiction’s overall approach to TANF employment services. Thus, the implementation study will examine the local context in each site and we will also collect data on local labor market conditions.

A.1.2. Overview of the STED and ETJD Evaluations

As noted, the STED and ETJD projects have complementary goals and are focusing on related program models and target populations. The ETJD project targets low-income noncustodial parents and ex-offenders. At this point, seven ETJD grantees have been selected and funded by ETA and these sites began to enroll participants in late 2011. Two of these seven sites are being evaluated as part of the STED evaluation. In addition, ACF has recruited an additional five sites for the STED evaluation, resulting in a total of 12 sites between the two evaluations. ACF received clearance from OMB to have discussions with states as part of the Subsidizing Employment Opportunities for Low-Income Families: A Review of State Employment Programs created through the TANF Emergency Fund (OMB Control Number 0970-0384). Three of the new STED sites target current, former, or potential TANF recipients; two of the new STED sites target disadvantaged youth and young adults. Each of the sites in both studies is expected to enroll approximately 1,000 study participants, for a total of 12,000 participants in the STED and ETJD studies combined.

The STED and ETJD evaluations are both being led by MDRC, a nonprofit organization which has contracts with both ACF and ETA. The evaluations will include the following components:

• Implementation and process study: The goal of this component is to understand how the programs operate, who they serve, the context in which they operate, and the dosage of services that sample members receive.

• Impact study: The goal of this component is to assess how the STED and ETJD programs affect employment, earnings, and other key outcomes such as recidivism rates, child support payments, well-being, and family relationships.

• Cost effectiveness study: This component will estimate the net cost per person of the STED and ETJD programs and compare those figures to the financial benefits that the program generate (for example, reduced incarceration or public assistance payments).

The two studies propose to collect four categories of data: 1) baseline data, 2) implementation and process data (some of which will support the cost effectiveness study), 3) surveys of research sample members, and 4) administrative records. This package focuses on the first three of those categories of data:

1. Baseline data for the two evaluations will be collected through the Contact Form (Appendix C) and the Baseline Information Form (Appendix D). In addition, sample members will be asked to provide updated contact information at several points following random assignment. Tracking letters requesting this updated information are included in Appendix E.

2. Implementation and process study materials include four short questionnaires to be completed by program staff, participants, worksite supervisors, and employers; interview guides for program staff members and administrators, employers, and referral partners; a discussion guide for participant focus groups; and time-use worksheet for program staff (which will also inform the cost study). These materials are included in Appendices F through I and are described in detail below.

3. Follow-up data will be collected through two surveys to be administered to study participants 6⁴ and 12 months following random assignment. The projects will attempt to survey all STED participants at both points in time and all ETJD participants at 12 months. The survey instruments are included in Appendices O and P. Each project will include an additional survey wave, at 30 months following random assignment. Further information about this additional information collection will be provided with the instrument in a future submission.

A.2. How, by Whom, and for What Purpose the Information is to be Used

How the Information will be used?

The findings from both studies will be used to inform the Federal government, states operating subsidized and transitional employment programs, program administrators, employers, and other stakeholders about the effectiveness of subsidized and transitional employment programs in helping vulnerable populations secure unsubsidized jobs in the labor market and achieve self-sufficiency.

Who Will Collect the Information

MDRC, the contractor for both studies, will collect data through face-to-face interviews with key informants, short questionnaires, site visits (including observations of program activities and case file reviews), and surveys. Program data and other existing data will be analyzed.

1. Baseline Information

Informed consent (Appendix B), contact information (Appendix C), and baseline information (Appendix D) will be collected from all study participants following determination of program eligibility, but prior to random assignment. These data will provide general information about participants that will ensure the comparability of the program and control groups, obtain data needed for subgroup analyses, and facilitate contact for subsequent follow-up surveys.

Procedures for baseline data collection will differ slightly between the ETJD and STED projects. Notably, the seven ETJD sites (including the two that have joined the STED project) have already begun to enroll participants and obtain informed consent, and baseline and contact data. Collection of baseline data for the ETJD sites received OMB clearance on September 9, 2011 (OMB Control Number 1205-0485). Baseline data is being collected via a Management Information System (MIS) developed and managed by ETA. The baseline data that are being collected for the five STED sites are similar to those being collected in ETJD sites. Where there are differences, we have described them below.

Informed Consent Form: Each of the five new STED sites has individualized informed consent forms (Appendix B); the two new STED sites that target youths and young adults have a separate informed consent form for participants that are minors at the time of study enrollment (Appendix B). These will ensure that participants: 1) understand the STED evaluation, as well as their role and rights within the study; and 2) provide their consent to participate.

To ensure that all study participants receive a clear, consistent explanation of the project, the evaluation team will train program staff on how to introduce and discuss the goals and design of the project, the random assignment process and data collection efforts. All staff will emphasize: 1) that participation in the study is voluntary; and 2) that strict rules are in place to protect sample members’ privacy.

Contact Information Form: Contact information in the five new STED sites will also be collected prior to random assignment. In addition to their own name, address, and phone numbers, participants will be asked to provide information about three additional individuals who are likely to be in future contact with them and can assist the research team to locate them for follow-up surveys. This form is attached in Appendix C.

Baseline Information Form: The baseline form (Appendix D) will collect basic identifying information about sample members in the five new STED sites, including social security number and date of birth. It also includes demographic items such as race and ethnicity, primary language spoken, and housing and marital status. Finally, the form also includes questions about participants’ education and employment history and any involvement with public assistance. These data will be used by the evaluation team to ensure the comparability of program and control groups, to facilitate contact for follow-up surveys, and to inform subgroup analyses in the impact study. Baseline forms will be administered to individual sample members by program staff members in one-on-one or group settings, with program staff members available in all cases to answer any questions.

As noted above, in the seven ETJD sites (including the two sites that have joined STED), baseline data that have already received OMB clearance (see above) are being collected via a Management Information System. In addition to the data that have already received OMB clearance, the STED baseline information form contains a few questions about public assistance receipt which do not appear on the forms administered in ETJD sites. We have also removed some questions about criminal history and child support status which appear on the baseline information forms being administered at the seven ETJD sites. These changes were made because we expect different populations of participants in the two projects, with more TANF and public assistance recipients in STED programs and more ex-offenders and non-custodial parents in ETJD programs.

Tracking Letters.  Because sample members are likely to be highly mobile and difficult to locate, the evaluation team will request updated contact information by mail from all sample members at various points following random assignment. Appendix E includes samples of the letters that will be sent to sample members. Each of these letters requests that participants call a toll-free number or return a form providing updated contact information for the respondent and—in some cases—up to three other contacts.

Sample members in the ETJD evaluation will be sent tracking letters at six points following random assignment (at 3, 9, 16, 20, 24, and 28 months).  These letters will remind participants of their participation in the ETJD study and will request that they notify the survey team if any personal contact information has changed since their intake into the study. The first of these letters will contain a $2 bill and all subsequent letters promise participants $5 for responding to the letter with updated contact information.⁵ ETJD participants will also be sent letters prior to the administration of the 12 and 30-month surveys notifying them that they will soon be contacted by phone to take part in a survey.  

STED participants will be sent two different kinds of letters following random assignment.  Each STED participant will be sent a letter of greeting on each birthday that occurs between random assignment and the administration of the 30-month survey. This letter requests that the participant update his/her contact information if it has changed. A few days prior to the administration of the 6, 12, and 30-month surveys, STED participants will also receive a letter reminding them of their participation in the study and notifying them that they will be contacted to participate in a survey shortly. Each of these notification letters will be nearly identical in content, but will be updated according to which survey is to be administered.

2. Implementation and Process Data

Implementation and process data will be collected during visits to all selected ETJD and STED sites approximately one year and two years after the beginning of program operations. While on site, evaluation staff will conduct interviews with program staff members and administrators, interviews with employers and referral partners, review participant cases, and observe program activities. Attached with this package in Appendix J are the protocols that will be used for interviews with staff members, employers, and referral partners. Discussion guides for focus groups with participants are attached in Appendix K. The form that evaluation staff will use to guide their observations of worksite activities is also attached with this package as Appendix L.

In addition, the evaluation teams intend to administer four questionnaires at different points during the life of the project: a program staff questionnaire; a worksite supervisor questionnaire; a participant questionnaire (to be administered to participants actively engaged in subsidized employment); and an employer questionnaire. These questionaires will obtain quantifiable data on dimensions of program implementation that will facilitate cross-site comparisons and help to explain any differences in program impacts across sites. All of these questionnaires are described below and attached in Appendix F through I.

Finally, the evaluation team will ask program staff members to record the time they spend on a range of program activities. This information will inform the implementation research by providing information on what percentage of time staff members are spending on different activities and in direct engagement with program participants (through such activities as case management, referrals, job counseling, job development, etc.). This information will supplement interviews and on-site observations by providing information on time-use that is directly comparable across sites. It will also inform the analysis of program costs by allowing the team to allocate the percentage of program budgets that is devoted to the activities measured. This worksheet is attached in Appendix M.

All questions have also been extensively reviewed by the evaluation staff, other senior MDRC staff, staff members at ACF and ETA, and members of the STED Technical Working Group (TWG). A list of TWG members is attached as Appendix Q.

Data collection activities are described below.

Participant Implementation Questionnaire. The participant questionnaire (Appendix G) will supplement responses to the 6- and 12-month surveys by providing information about participants’ experience on the job while they are still engaged in subsidized employment. The questionnaire will be administered three times over the course of the evaluation to all participants who are engaged in subsidized employment at the selected point in time. We expect to administer these questionnaires at approximately 6-9 month intervals so that no individual participant will be asked to complete the form more than once. These questionnaires should take approximately 10 minutes to complete; they will be administered with the help of program or research staff. In order to maximize response rates, and out of consideration for the time participants spend completing the questionnaire, we are considering offering a nominal monetary incentive for completing the participant implementation questionnaire. If offered, this will be provided in the form of a gift card and will not exceed $10.

The participant questionnaire includes approximately 25 questions on the following topics:

1. Skill development. Transitional and subsidized jobs programs aim to increase participants’ employability, partly through the acquisition of “soft skills” – that is, general employability skills such as showing up on time and working well with others. Some programs teach these skills directly in pre-employment training, others hope that experiences on the job either teach or advance these skills. Hard skills – such as computer or construction skills -- are provided in some programs. The Participant Questionnaire will attempt to ascertain whether participants believe they are acquiring soft and/or hard skills through their subsidized jobs.

2. Supervision. Supervision and on-the-job coaching are important mechanisms for correcting problems and improving participants’ employability. The Participant Questionnaire contains questions about interactions with supervisors to understand how much time supervisors spend with participants and the content of their interactions.

3. Peer support. Interactions with peers are also thought to be important in changing work behaviors and increasing contacts to help participants get and retain subsequent jobs. The Participant Questionnaire contains questions about support from peers.

The questions for the participant questionnaire were drawn primarily from surveys that were administered to participants in the JOBSTART evaluation and participants in the CEO transitional jobs program for ex-offenders. Those questions that were not drawn from these sources were developed internally by MDRC and its partners. They were developed to provide greater information about subsidized employment participants’ own perceptions of the benefit of their experience and the specific skills and lessons they are taking from it. Many members of the evaluation team have deep knowledge and experience related to transitional jobs and employment programs for the populations in the STED and ETJD evaluations, but there is not a rich source of information providing self-reported experiences of participants while they are going through programs. This questionnaire will fill some of these gaps in existing knowledge and also allow us to compare the experience of participants across sites

Participant focus groups. Focus groups with participants will be conducted twice in each site. These group interviews will be conducted with about 10 individuals currently participating in the program. These participants will be selected based on who is available to participate in the focus group at the time we are conducting it. In our experience, fewer participants show up than we expect. Group interviews will collect important information on how prepared participants felt when starting their subsidized job, views about their subsidized jobs such as what skills they think they are learning, frequency and nature of supervision, and the support they receive from program staff and/or peers. A protocol for these discussions is included in Appendix K.

Program Staff Implementation Questionnaire The program staff questionnaire (Appendix F) will supplement semi-structured interviews with program staff. The questionnaire will be administered to key program staff who are involved in program administration or who work directly with participants (such as job developers and case managers). Each staff person will complete the questionnaire once during implementation visits to STED and ETJD site locations; it will require approximately 10 minutes to complete.

The staff questionnaire is not intended to duplicate the information that will be collected via the semi-structured interviews discussed below. Instead the survey will focus on two aspects of implementation: 1) job development and 2) general organizational issues and work environment.

1. Job Development/Placement. Questions focus on job development and placement in the transitional or subsidized job as well as program staff communications with the transitional or subsidized job employers. In addition, since a critical component of these programs is the connection into unsubsidized employment, there are questions about what methods program staff use to place participants into unsubsidized employment as well as how they talk to participants about making the transition.

2. Work Environment/Organizational Capacity. Questions cover issues such as whether staff get clear direction, regular feedback from supervisors, and staff morale levels. These “generic” organizational factors may be critical in explaining differences in implementation and impacts across sites.

The questions for the Program Staff Questionnaire were drawn primarily from two sources: 1) the Chicago Neighborhood JobStart Staff Survey evaluated by the Social IMPACT Research Center and 2) the Minnesota Family Investment Program Job Counselor Survey, which was administered by MDRC and received OMB clearance. Those questions that were not drawn from these sources were developed internally by MDRC and its partners/subcontractors—MEF Associates, Branch Associates, Abt Associates, and DIR, Inc.

This questionnaire was developed to provide information about the nature of job development and whether job placement for the subsidized jobs emphasizes placements that match participants’ skills and interests or meets employers’ needs and how program staff balance these considerations while achieving placement goals. In addition, it is hypothesized that work environment is positively correlated with staff members’ abilities to perform job duties and these questions will ensure uniform data on organizational capacity and the work environment across the sites. This questionnaire will allow the team to more systematically compare the nature of job development and the job developer/case managers’ work environments across sites and provide contextual information for understanding any variation that may occur in outcomes and impacts.

Worksite Supervisor Implementation Questionnaire. The worksite supervisor questionnaire (Appendix H) will provide information about how supervisors view their role in working with subsidized employees and the experience that supervisors have working with participants in the evaluation. This questionnaire will be administered to the direct supervisors of program participants while evaluation staff are on-site for implementation visits. It will be administered twice over the course of the evaluation, although we only expect each supervisor to complete it once. MDRC will keep track of supervisor names to ensure that each supervisor completes it only once; however, supervisors will not be required to write her or his name on the questionnaire, if s/he wants to remain anonymous. The questionnaire will be administered on paper and will take approximately 10 minutes for supervisors to complete. In order to maximize response rates, and out of consideration for the time worksite supervisors spend completing the questionnaire, we are considering offering a nominal monetary incentive for completing the worksite supervisor implementation questionnaire. If offered, this will be provided in the form of a gift card and will not exceed $10.  

The worksite supervisor questionnaire will capture some of the same topics as the participant questionnaire – but from the supervisors’ perspective. In some cases, these supervisors will be employees of the program site. In other cases, they will be staff of private employers. The goal of the worksite supervisor questionnaire will be to obtain data on the nature of supervision and how program participants compare with other, unsubsidized workers in terms of productivity and supervision. Key topics include:

1. Participant Performance. Work site supervisors have the most direct contact with participants on the job, and thus are well-suited to provide information about participants’ performance on the job (relative to others in the same position) and the disciplinary actions they take.

2. Skill development. Prior evaluations have demonstrated that it can be challenging to ensure that work site experiences provide opportunities for participants to build soft skills. Work site supervisors – especially in private businesses – may be more concerned with getting the work done than coaching participants. This survey is intended to capture the extent to which supervisors spend time coaching participants.

3. Communication with Program Staff. Also important in program implementation is the communication between work sites and case managers and this survey asks about communication and feedback given to program staff.

Many of the questions for worksite supervisors were developed internally by MDRC and its partners based on prior experience evaluating transitional jobs and employment programs. Some of the questions have been drawn from surveys that were administered in the JOBSTART evaluation and in the evaluation of the CEO transitional jobs program for ex-offenders. Like the participant questionnaire, this questionnaire will allow the team to systematically compare the experience of supervisors across sites and to better understand what kinds of approaches may have the most value for subsidized employees and the jobs in which they are working. It will provide more contextual information for understanding any variation that may occur in outcomes and impacts across the sites.

Employer Implementation Questionnaire. The employer questionnaire (Appendix I) will collect basic information about employers who take on subsidized workers. It will be administered twice over the course of the evaluation, although we only expect each employer to complete it once. MDRC will keep track of employer names to ensure that each employer completes it only once; however, the employer will not be required to write her or his name on the questionnaire, if s/he wants to remain anonymous. It will be administered on paper with the help of program staff.

Questionnaires will be sent to the contact person identified by program staff and will be administered in person as available. The individual who made the decision to take on subsidized workers and/or who handled the hiring will be asked to complete the questionnaire. It will collect basic information about the respondent’s role to ensure that the correct person is completing the questionnaire. It will take approximately 10 minutes to complete. In order to maximize response rates, and out of consideration for the time employers spend completing the questionnaire, we are considering offering a nominal monetary incentive for completing the employer implementation questionnaire. If offered, this will be provided in the form of a gift card and will not exceed $10.

Since work experience is typically the heart of subsidized employment programs, it is critical to understand, in some detail, the characteristics of participating employers and their experiences in the program. The questionnaire will collect information on employers’ characteristics such as the number of employees, type of business, and financial health. The questionnaire will also capture experiences in the program such workers’ productivity and changes in the firm’s financial health.

Questions for the employer questionnaire have been drawn from a number of sources. Certain questions were developed internally by MDRC and its partners but the majority of questions have been drawn from surveys that have been administered to employers as a part of previous evaluations and employment projects. These sources include the Chicago Neighborhood JOBSTART Employer survey; Bridging the Opportunity Gap: The Massachusetts Department of Youth Services Career Readiness Vocational and Employability Re-entry Initiative; the On-the-Job Training Survey conducted by the Center for Workforce Information and Analysis at the Pennsylvania Department of Labor and Industry; Subsidizing On-the-Job Training: An Analysis of a National Survey of Employers conducted by John Bishop for ETA; Entry-Level Employers in Colorado: Results from a Survey of 25 Employers by Burt Barnow; and Work Opportunity Tax Credit, a 2001 GAO report presented to the House Committee on Ways and Means.

Like the participant questionnaire, the employer questionnaire will allow the team to systematically compare across sites and to better understand what kinds of approaches may have the most value for subsidized employees and the jobs in which they are working. It will provide more contextual information for understanding any variation that may occur in outcomes and impacts across the sites. Specifically, the data will be used to determine whether firms tend to place subsidized workers in vacant job slots or create new positions. The data will also be used to explore whether certain features of participating employers are correlated with variations in outcomes across the sites.

Case file reviews. Case reviews are an effective way to obtain detailed, quantifiable data on certain implementation practices. The reviews provide a systematic vehicle to probe beyond the data that are typically included in a management information system. While on site, evaluation staff members will document the extent of interaction between the job developer and the client, the extent to which job development was tailored to the individual client, the number of specific job referrals that the client received, the extent of contact between job developers and employers, and so on. Using job development/placement as an example, we intend to use program MIS data to select a random sample of 10-20 cases that were placed in jobs and another sample of 10-20 cases that received job development services but were not placed. If case notes and other data are maintained electronically, we will review the cases before our field visit to supplement the on-site review of documents. We intend to gather information for approximately 20-40 participants in each site during each implementation site visit. We do not expect our case file reviews to place a burden on program staff.

Interviews with program staff, employers and referral partners. Semi-structured interviews with program staff (project directors, case managers, job developers, and job coaches), employers, and referral partners will be conducted twice, during visits to the participating sites. The interviews will be used to obtain basic information about the program model and its implementation as well as to understand the organizational capacity of the primary agency and any partners and how they staff the program. These interviews will cover basic aspects of the program including the recruitment and enrollment process, assessing participants’ skills and needs, the subsidized jobs (type of work, supervision, length), pre-employment training, skills training, case management, job development, and post-placement services. These interviews will also be the source for contextual information such as other initiatives and features of the local economy.

The first site visit will be conducted about one year after the start of random assignment and will include questions about the background of the program model, recruitment and screening, program goals, target population, subsidy details and mechanics. Interviews will take about one hour to complete, and will be conducted either one-on-one or in a group.

The second site visit will be conducted about 12 months later. The purpose of the second interview will be to elicit perceptions of how well plans have been implemented and reasons for departures from what was planned. Again, interviews will take about one hour to complete, and will be conducted either one-on-one or in a group.

Appendix J includes interview guides for program staff and directors employers, and referral partners (including parole officers). Since each site will have different a staffing structure, not all the questions listed in these protocols will be asked of each respondent we interview; rather, the protocols will serve as guides to evaluation staff conducting interviews.

Worksite observations. It is critical to understand, in some detail, what goes on at subsidized worksites. To address this issue, the research team intends to supplement interview and focus group data with direct worksite observations.

Our exact procedures for observing worksite activities will vary depending on whether subsidized workers are spread among many worksites or consolidated in a few. In the former case, we intend to visit and observe a sample of worksites while in the latter we will visit the single worksite. In either case, a member of the research team will spend approximately one to two hours observing the subsidized worksite and completing an observation check list (Appendix L) covering cover issues such as: 1) what kind of work STED and ETJD participants are doing; 2) the extent and nature of interactions between the participants and co-workers or customers; and 3) the extent and nature of interaction between participants and supervisors. We do not expect our observation of worksite activities to place any burden on site staff, supervisors, or participants.

Program Staff Time-use Worksheet. The staff time-use worksheet (Appendix M) will serve both the implementation and cost studies. It will be administered to all program staff from state and local agencies and program provider organizations that are participating in the STED and ETJD evaluations. The worksheet will capture all of the relevant activities that any of the programs might provide, although each program will likely record time in a limited number of categories based on the program intervention. Each program staff person will be asked to keep track of their time for a two-week period, to be determined by the program senior staff and the research team.

The staff time worksheet will be used for the implementation study to describe how staff members spend their time. It will allow the study team to describe the emphasis placed by the program on each program activity and allow for comparisons to be made across sites. For example, it will allow the team to compare program case management and job development practices across sites.

The time-use worksheet will be also used for the cost study to allocate the staff salaries and overhead costs incurred by the program to each program activity based on the percent of time that staff spent on each activity. Using the cost information, time-use worksheet, and participation data, the research team will be able to estimate the average cost of providing each STED/ETJD service to participants.

Cost Data Collection Protocol: Appendix N includes a protocol to be administered with a program fiscal or accounting staff member while on site for implementation visits. Before we administer it to program staff, we will have already received complete financial expenditure reports from the program. For example, these reports will include data on staff salaries, rent, overhead, supplies, etc. As such, the protocol is intended to be used as a guide to supplement these reports. In addition, we will use the program staff time-use worksheet in conducting our cost analysis. The burden for this protocol is under program staff interviews, as it will be used during the site visits. We included one additional program staff from each site to the burden estimate.

3. Survey Data Collection

Currently, we intend to survey ETJD and STED sample members at different points following random assignment. Sample members in the five STED sites will be contacted and asked to complete follow-up interviews approximately 6, 12, and 30 months after they are randomly assigned. Sample members in the five non-STED ETJD sites will be contacted 12, and 30 months after they are randomly assigned. This information collection request includes only the 6 and 12 month surveys; subsequent surveys will be described in a later information request. The survey data will contribute substantially to the ability of the STED/ETJD evaluations to measure the effectiveness of different strategies to increase employment among hard-to-employ populations with the long-term goal of making individuals and families better off. Specifically, data collected will enable us to determine whether or not the resources allocated to the various subsidized and transitional employment strategies did, in fact, lead to increased employment, higher incomes, and improved well being.

Although administrative records data will play a critical role in the evaluation, they leave some important gaps in knowledge about a range of outcomes that are very relevant to the study. The surveys will yield important data not available through administrative records, such as information on the characteristics of jobs held during the follow-up period (e.g., wage rates, hours worked, and fringe benefits), participation in employment-related services, parental involvement, and material and financial hardships. The surveys will also provide information on the sources of program and control members’ income—including disability payments, Earned Income Tax Credit (EITC), and alimony payments—that are unavailable from the administrative data. Furthermore, the survey is the only source of information on earnings and other income received by other members of respondents’ households.

The surveys will also provide important information for the study’s cost analysis, by detailing the types of activities and work supports the individual has participated in or received six months and one year prior to the survey interview. This information will be helpful for establishing the net cost of the program interventions. While program records may be a good source of cost data for the program group, there is no way to collect similar information on the control group, since in most cases little information is available on them.

There are two versions of each survey: one for the sites targeting young adults and youth and one for sites target adult populations (former/current/potential TANF recipients, non-custodial parents, and ex-offenders). The young adult version of the surveys includes more items related to education/training and items related to expectations and planning; as these are areas emphasized in the interventions at the two STED youth/young adult sites. Both surveys are discussed in more detail below, including further details of the differences between the young adult and adult versions of the instruments.

6-Month Survey. The 6-month survey (Appendix O) is a 30 minute survey which is mostly focused on measuring the in-program effects of STED programs during a time period that has been understudied in past evaluations of these programs. In particular, the survey will be administered while many program group members are participating in subsidized employment, a period when there is likely to be a very large impact on employment (i.e., the program group will be much more likely to be employed owing to the subsidized jobs).⁶

This timing provides an opportunity to go beyond past surveys for evaluations of subsidized and transitional job programs, which have mostly focused on the economic benefits of working. It may be that employment confers a range of nonfinancial benefits. Evidence suggests that unemployment increases stress, anxiety and depression. These can lead to additional incidence of disease, reduced job search activities, damage to social and familial relationships, and social withdrawal. Work provides income and potentially income stability. These can reduce stress and anxiety over basic survival issues like paying bills and providing food to one’s family and thus remove the triggers for a host of problems including poor physical health and strain on relationships. However, the experience of work seems to matter as well. That is, work may do more than simply relieve the hardship of unemployment.

Work ties into conceptions of self-worth and self-efficacy. Success at work demonstrates competence to oneself and to one’s family and friends. Furthermore, individuals may benefit from simply “getting out of the house,” and interacting with others. Research on depression has repeatedly emphasized the importance of regular social contact in promoting psychological well-being. Work may also strengthen the social support available to an individual. Interaction with co-workers might foster positive peer effects or generate connections that widen the social networks that participants often use to find work. For ex-offender populations, for example, exposure to and integration into the larger population may provide positive peer “pressure” and possible mentoring opportunities that could reduce recidivism to prison.

The 6-month survey will provide a very rare opportunity to examine some of these issues in the context of a large experimentally-induced impact on employment. Typically, research on the spillover effects of employment relies on correlational data and is subject to selection bias.

What follows is a brief description of each section of the 6-month survey:

• Employment and Service Participation. The proposed survey begins by trying to measure the current employment and service receipt status of study participants. These findings will help the research team to better understand the pattern of findings on other outcomes. For example, if there are no effects on employment, this would help explain a lack of effects on the nonfinancial benefits that are expected to accrue from employment. For the young adult version of the survey, the employment section was revised to include participation in internships, paid and un-paid, as this form of employment is more typical for the youth/young adult population. The sequence of the questions in this section was also revised to capture engagement in education and training, as this type of productive activity is more likely for the youth/young adult population.

• Physical and Mental Health. The survey then proceeds to ask a series of questions (derived from national surveys) about physical and mental health. Job loss and job experiences can be associated with a variety of mental health effects including stress, anxiety, and depression. Poor mental health may manifest somatically in a variety of physical health problems ranging from aches/pains to serious heart conditions so some measures of general physical well-being are included. The young adult version of the surveys uses the same measures as the adult version.

• Efficacy and Locus of Control. A series of psychological measures are collected in the areas of depression, locus of control, and efficacy. General Self-Efficacy is related to self-confidence and motivation/effort and thus a possible intermediate outcome of interest. This could presumably be associated with better work performance (and thus retention) or more rigorous job search. Locus of control captures respondent beliefs about how much of their lives are directly under their control as opposed to the result of luck, fate or manipulation by other individuals. While related, perceptions of self-efficacy and locus of control are usually considered distinct and thus measured separately. The young adult version of the survey includes additional items related to these topics as these domains are of particular interest for this population. The adult survey includes items related to job search and work self efficacy: two domains that are especially relevant to this study. The young adult version includes items related to career orientation, as this population is not expected to have sufficient labor market experience to have formed meaningful perceptions of their job-search and work self efficacy.

• Social Isolation and Social Support. Social isolation and poor social support limit individuals in several ways. Most individuals find their jobs through social networks and may do so passively (stumbling upon job leads/information/offers). Poorly connected individuals have fewer opportunities for this to occur, less access to novel information, and fewer friends/family/acquaintances to query. Social networks also provide needed financial and non-financial support (e.g. baby-sitting, cooking, transportation) that can help an individual weather difficult situations and facilitate job participation. For the young adults, this survey also assesses the presence of mentors, roles models and other influential advisors.

• Social Network Roster and Relationship Origin. Rosters allow more direct measurement of social support. The questions here identify the members of the social networks of respondents and how the network members are known to the respondent. This will allow assessment of whether program group members (1) have larger networks (i.e., list more relationships) and (2) have different relational compositions (i.e., list more work or program related relationships). The adult and young adult versions of the surveys do not differ for this topic area.

12-month Survey. The 12-month survey (Appendix P) will collect data on study participants’ receipt of services and attainment of education credentials, labor market status, material hardship, household income, criminal justice, self-sufficiency and family engagement, including, child support payments and parent-child contact. We expect the 12-month survey to take 45 minutes to administer.

The 12-month survey is comprised of several modules. Most of these modules have two purposes: (1) to provide a systematic description of key outcomes and; and (2) to measure the differences in employment, wage progression, income, and other outcomes between the program groups and a similar group of respondents who were not eligible for the programs. What follows is a summary of the proposed modules in the STED and ETJD 12-month survey:

• Participation in Employment-Related and Education Activities: This modules measures the extent of participation in a range of activities (including job search and education and training), for both the program and control groups. It is the key source of data for measuring the treatment difference that will drive the later impacts on longer-term outcomes. The young adult version of the survey emphasizes engagement in education and training as this type of participation is more relevant to the population.

1. Employment: This module includes an employment history since random assignment and measures a variety of characteristics of each job held. Employment outcomes will be analyzed separately depending on whether employment is in subsidized or unsubsidized jobs. Both versions of the survey use the same employment module, except that the young adult version includes internships as a form of employment for the reasons noted above.

• Marriage, Household Composition, and Material Hardship: This module measures housing and marital status and family composition. It also measures income, the primary income sources (such as child support, Supplemental Security Income, and EITC) during a one-month period at the time of the interview. Finally, this section will measure levels of material and financial hardship. The young adult version of the survey includes additional items related to housing, particularly changes in housing status, as stability in living circumstances is more critical for this population. In addition, the young adult version of the survey focuses on individual income and material hardship while the adult version includes both individual and household income/hardship; this is due to the expected greater prevalence of communal living situations for the young adult population (i.e., they are more likely to be living with room-mates and/or still in the family home).

• Health Coverage: This module measures the extent to which respondents have health coverage, funded by employers or other private sources, or funded by government programs like Medicaid. For the young adult version of the survey, this also includes being covered by their parent or guardian’s health insurance.

• Health and well being: This module measures the extent to which respondents or their family members have any key health problems. This section will include some of the psychological scales from the 6-month survey. The young adult version of the survey includes greater focus on psychological well-being, including substance abuse and mental health treatment. For the young adults, most of the items from the earlier survey will be replicated at 12 months as they are key outcomes for these populations.

• Criminal justice module: This module will measure outcomes that are not measured, or not measured well by administrative records including parole, probation violations and recidivism. This section will be administered to all participants in the sites targeting ex-offenders, and only those with a criminal record in the noncustodial parent (NCP), and TANF sites. Sample members at both of the young adult sites will be administered this module as criminal justice involvement is considered of high importance for this population.

• Custodial parents module: This section is for the sites targeting TANF recipients. The focus is on the cost of child care and child care barriers that participants might confront which might affect their program participation or employment. The young adult survey includes some items on these topics but is less extensive as fewer sample members are expected to be custodial parents.

• Noncustodial parenting module: These questions will be asked of all participants in the NCP sites and NCPs in the Ex-Offender sites. The focus is on child support payments and parental involvement. The young adult survey also includes some items related to this topic but the coverage is less extensive as fewer sample members are expected to be non-custodial parents.

Staff Training

All members of the research teams have extensive experience conducting qualitative and quantitative data collection and analyses. Specific orientation and training on the particular data collection instruments for both studies will be conducted to ensure fidelity to the protocol and consistency of coding.

Purpose of the Data Collection

Findings from this study will be used to inform the Federal government, states, policy makers, program operators and other stakeholders about the efficacy of programs aimed at helping target populations secure and maintain unsubsidized employment.

The exhibit on the following page shows the different timeframes for the 6- and 12-month survey fielding.

Revised Table: 6- & 12-month survey fielding

A.3. Use of Information Technology for Data Collection to Reduce Respondent Burden

Wherever possible, advanced technology will be used in data collection efforts to reduce burden on study participants and on site staff. The following methods will be used:

1) Computer-assisted survey interviews (CATI/CAPI). This helps to reduce respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length and the need for subsequent call backs. Computer programs enable respondents to avoid inappropriate or non-applicable questions. Key data will be pre-loaded on the surveys based on response to previous survey questions (in the same survey) and earlier surveys (including items collected in the six month survey, where applicable, and at baseline). CATI/CAPI also improves data quality through more uniform administration of the survey questions, more accurate implementation of the skip patterns, and immediate application of range checks, edit checks, and consistency checks of item-by-item responses.⁷

2) Social media. Another use of information technology relates to the planned use of social media as a means of communicating with respondents. Social media has had a large impact on how people communicate and provides a flexible means for interaction with respondents. For this reason, we have added questions on the contact information form requesting permission to contact participants or alternative contacts through Facebook. We will use Facebook as means of contact with sample members only if they cannot be contacted via any other means. All contact via Facebook will be done via private messages from the survey contractor and will only be viewable by the sample member. The content of these messages will be a request for the sample member to contact the survey firm via a toll-free number. The private messages will not contain any other information about the sample member, including their status as a member of the STED/ETJD evaluation sample. As with all other personally identifiable information collected as part of the STED/ETJD evaluation, Facebook account information is stored securely with limited access. All project staff with access to such data have received training on the use and handling of personal information and have signed statements affirming their understanding and compliance with the project’s data security procedures.

3) Survey tracking systems. The survey firms have databases that track the location of the participants throughout the project.  Database changes come from mailings to the participants and passively track respondents through the U.S. Postal Service Change of Address database. In addition to being an inexpensive method for being able to contact a respondent, this passive collection approach reduces the need for the respondent to continually provide their most recent address information. It also reduces the probability of seeking alternate contacts in order to find the respondent.

4) Use of Key Survey to administer implementation questionnaires. The research team will work with program sites to determine the most efficient way of administering implementation questionnaires. In some cases, based on preliminary conversations with ETJD program site staff, we expect to administer the questionnaires online using Key Survey. This will minimize burden on the respondent by allowing them to respond to questions at their convenience, without having to schedule time to complete the questionnaires on paper with research staff. Appendix S is a sample screenshot of the program staff questionnaire. The format of the other online questionnaires will be the same as the one provided. We do not plan to administer other data collection instruments online.

5) Integration of other data sources. Finally, when relevant person-level data has been identified as available through an accessible centralized, computerized source, the information has generally been excluded from the proposed data collection package. For example, historical cash assistance (TANF), criminal justice, Food Stamps (SNAP), and UI data will be obtained through administrative records. In addition, all the ETJD sites will use a centralized Management Information System (MIS) maintained by DOL. Information regarding participant eligibility and demographics as well as program participation will be maintained in a single location that is designed for easy access to this information by the research team.

While implementation data collection relies on evaluation staff efforts on-site, we have sought wherever possible to minimize overlap between questions we include in implementation questionnaires and protocols and questions that will be asked through computer-assisted surveys.

A.4. Efforts to Identify Duplication

The information collection will not duplicate information that is already available. Where possible, the evaluations will use available data sources, such as program tracking data, UI wage records, TANF and SNAP data systems, child support systems, and criminal justice records. The surveys will collect data on various other outcomes (such as emotional well being or detailed job characteristics) that is not available routinely or systematically in program records.

In this particular study, it is quite important that the survey efforts be well coordinated across the two survey firms. MDRC has created a joint data management team on these two projects and has standardized data agreements with both firms. The survey firms will work on separate studies which will ensure that they are not trying to contact the same person. Also, both firms will use identical survey questionnaires. Standardizing the content and format of the data deliverables will streamline work and make it easier to combine the data files at the end of the project.

Finally, a central topic of these evaluations involves the tracking of employment, service participation, and well being of participants over time. Thus, it is critical to have a core set of consistently worded questions in order to track how the impacts of the program change over time on a common set of measures. However, whenever possible, measures that do not require tracking over time will be assigned to one and only one follow up survey.

A.5 Burden on Small Business

It is anticipated that the data collection activities proposed here will impact small businesses, though all attempts will be made to minimize the burden. Some employers who will be asked to complete the Employer Questionnaire will be small businesses. However, this brief questionnaire will be pre-tested with 9 or fewer people to ensure that the questions are clearly written and understood by respondents. Site visits to employers, which may include small businesses, will be scheduled at the respondents’ convenience, and information about the purpose and scope of the visits will be provided in advance. Interviews will be scheduled with staff who are familiar with the subsidized job program and intend to seek information only about which key respondents have direct knowledge.

A.6 Consequence If Data Collection is Not Conducted

These complementary evaluations represent an important opportunity for the Federal Government to add to the body of knowledge about the impacts of employment-oriented programs for TANF recipients, noncustodial parents, and ex-offenders. This is consistent with the Administration’s strong focus on evidence-based policymaking. If the information is not collected, analyzed, reported, and disseminated, Federal program or policy activities will not be informed by high quality information upon which to base critical decisions regarding future investments.

A.7 Special Data Collection Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320 (Controlling Paperwork Burdens on the Public). There are no special circumstances that require deviation from these guidelines.

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission

a. Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment on the additional materials through the 60-day Federal Register Notice, published on Friday, March 15, 2013 (FR, Vol. 78, No. 51, pp. 16509-16510). A copy of this notice is attached. During the notice and comment period, ACF received one request for the proposed data collection instruments. Draft instruments were provided.

b. Consultations Outside of the Agency

All data collection instruments included in this package have gone through extensive review by members of the STED technical work group, ACF and ETA staff, and members of the research team (i.e., MDRC, Abt Associates, DIR, Inc., MEF Associates, and Branch Associates). A list of technical working group members is included in Appendix Q.

A.9 Justification for Respondent Payments

The respondent payments and justification for each proposed instrument are outlined below.

• All ETJD participants will receive a small token of appreciation ($2 each) to encourage them to provide updated contact information in response to the first letter that will be sent to them following random assignment. Sample members will be promised $5 for responding to subsequent requests for updated contact information. Tokens of appreciation for updating contact information are not planned for participants in the five STED-only sites because the ETJD mailings specifically request updated contact information. The STED mailings are a) a birthday greeting, and b) an advance letter a few days in advance of data collection. The STED mailings mention staying in touch and calling in if they’ve moved, but are not specifically requesting the sample member to fill out a form and send it back in.

• Follow-Up Surveys –Participants who agree to take part in the 6-month survey will receive $25 for completion of the survey. Participants will receive $40 for completion of the 12-month survey. The amounts for the 12-month survey are based on what was previously approved by OMB in past studies such as the Hard to Employ 15-month follow-up survey, and MDRC’s and the survey firms’ prior experience interviewing similar populations. The payment amount for completion of later follow-up surveys (30 months for both STED and ETJD) will be addressed in future information collection requests.

The purpose of the tokens of appreciation is to improve response rates by decreasing the number of refusals, enhancing respondent retention, and providing a gesture of goodwill to acknowledge respondent burdens. This technique is proposed in addition to many of the techniques suggested by OMB to improve response rates that have been incorporated into our data collection effort and are described in Section B3, because our experience has shown that small monetary amounts are useful when fielding data collection instruments with hard-to-employ populations as part of a complex study design. It is essential to include a token of appreciation in order to maximize the response rate, and it is particularly important with a challenging population and a demanding data collection strategy. In a seminal meta-analysis, Singer, et al. (1999) found that incentives in face-to-face and telephone surveys were effective at increasing response rates, with a one dollar increase in incentive resulting in approximately a one-third of a percentage point increase in response rate, on average. They found some evidence that incentives were useful in boosting response rates among underrepresented demographic groups, such as low-income and non-white individuals.⁸ This is a significant consideration for this study of subsidized and transitional employment programs. Another important consideration is the burden posed by this data collection, which will take on average 30 minutes of the participant’s time for each 6-month survey and 45 minutes for each 12-month follow-up survey.

The instrument that will be used to collect follow-up data from sample members has unique aspects that make administration difficult and threaten response rates. We are therefore requesting clearance to offer a small monetary amount to all sample members who complete each follow-up survey. Aspects of the data collection effort that also make it more difficult to obtain high completion rates are:

• The surveys include questions that could be perceived as intrusive and therefore could make respondents uncomfortable (i.e., questions about their mental health).

• The subject matter of the interview is not intrinsically interesting to respondents. Moreover, many participants may have negative feelings about the other services received that are of interest, such as welfare, Medicaid, job training, etc.

• Other difficulties in administering the surveys come from the population itself. Educationally and economically disadvantaged groups have been found to be more difficult than the general population to convince to participate in surveys.

The amounts offered increase with each survey wave, i.e., 6-month to 12-month, for two reasons: 1) the length of the survey increases with each survey wave; therefore, the survey will likely take the respondent longer to complete than the previous survey; 2) the length of time from the point of random assignment increases with each survey wave; therefore, the survey firm will likely have greater difficulty in tracking down respondents with each survey wave and will need a greater incentive to produce completion rates that will meet or exceed the 80 percent response rate target.

We are aiming to achieve an 80 percent survey completion rate for the follow-up survey. Even with the best data collection practices, it would be very difficult, if not impossible, to obtain such a high completion rate without providing a token of appreciation to participants.

A.10. Assurances of Privacy

Every effort will be made to maintain the privacy of respondents, to the extent permitted by law. All respondents included in the study will be informed that information they provide will be used only for the purpose of this research. Individuals will not be cited as sources of information in prepared reports. All research staff working on the project have been trained to protect private information and have signed a pledge stating that they will keep all information gathered private to the extent permissible by law. All papers that contain participant names or other identifying information will be kept in locked areas and any computer documents containing identifying information will be protected with a password.

A. 11. Questions of a Sensitive Nature

Questions in all components of the 6-month and 12-month surveys are potentially sensitive for respondents. Respondents are asked about highly personal topics. The questions we have included were selected, in part, because they have been widely used in previous research and are respected among experts. Moreover, since all questions will be pre-tested prior to the survey’s full implementation, if problems arise in regard to any specific items, their inclusion will be reconsidered. Finally, respondents will be informed by program staff prior to the start of the interview that their answers are private, that they may refuse to answer any question, that results will only be reported in the aggregate, and that their responses will not have any effect on any services or benefits they or their family members receive.

A.12. Estimates of the Hour Burden of Data Collection to Respondents

The hour burden for this data collection for STED and ETJD sites and participants is outlined below in Exhibit 1.2. We have assumed the maximum possible number of sites and study participants.  The estimates included below are based on experience with previous random assignment studies involving similar populations and data collection instruments. For the surveys, the burden is the same whether the participants receives the adult or young adult version of the instrument.

	Exhibit 1.2: Annual Burden Estimates
Instrument		Total Number of Respondents (life of project)	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hour per Response	Annual Estimated Burden Hours1	Average Hourly Wage of Respondents	Annual Cost
Participant Contact Information Form (5 STED sites)		5,000 (all sample members at five STED-only sites)	1,667	1	.08	133	$7.25	$964.25
Participant Baseline Information Form (5 STED sites)		5,000 (all sample members at five STED-only sites)	1,667	1	.17	283	$7.25	$2,051.75
Participant STED tracking letters		2,310 (33% of STED sample members, including STED-ETJD shared sites)	770	5	.05	193	$7.25	$1,399.25
Participant ETJD tracking letters		1,650 (33% of seven ETJD site sample members)	550	6	.05	165	$7.25	$1,196.25
Participant 6-month survey (Adult sites, instrument previously approved)		4,000 (80% of adult STED sites, including 2 STED-ETJD shared sites)	1,334	1	.5	667	$7.25	$4,835.75
Participant 6-month survey (Young Adult sites)		1,600 (80% of young adult STED sites)	533	1	.5	267	$7.25	$1935.75
Participant 12-month survey (Adult sites, instrument previously approved)		8,000 (80% of adult STED and ETJD sample members)	2,667	1	.75	2,000	$7.25	$14,500
Participant 12-month survey (Young Adult sites)		1,600 (80% of young adult STED sites)	533	1	.75	400	$7.25	$2,900
Participant Implementation Questionnaire		600 (50 participants /site at all STED and ETJD sites)	200	1	.17	34	$7.25	$246.50
Participant Focus Group Discussion Guide		240 (20 participants/site at all STED and ETJD sites)	80	1	.75	60	$7.25	$435
Program Staff Implementation Questionnaire		120 (10 staff members/site at all STED and ETJD sites)	40	1	.17	7	$19.83	$138.81
Worksite Supervisor Implementation Questionnaire		240 (20 supervisors/site at all STED and ETJD sites)	80	1	.17	14	$23.10	$323.40
Employer Implementation Questionnaire		240 (20 employers/site at all STED and ETJD sites)	80	1	.17	14	$52.21	$730.94
Program Staff Interview Guides		120 (10 staff members/site at all STED and ETJD sites)	40	2	1	80	$19.83	$1,586.40
Program Staff Cost Data Collection Protocol		12 (1 staff member/site at all STED and ETJD sites)	4	1	1	4	$19.83	$79.32
Employer Interview Guides		24 (2 employers/site at all STED and ETJD sites)	8	2	1	16	$52.21	$835.36
Referral Partner Interview Guides		24 (2 referral partners/site at all STED and ETJD sites)	8	2	1	16	$24.64	$394.24
Program Staff Time-Use Worksheet		120 (10 staff members/site at all STED and ETJD sites)	40	1	1	40	$19.83	$793.20
Estimated Totals						4,393		$35,346.17

In addition to the instruments listed in the above chart, we plan to submit a future ICR to conduct a 30-month follow-up survey for all STED and ETJD participants.

We have estimated the hourly wage of all participants at the federal minimum wage: $7.25 per hour. This is based on our experience with other transitional jobs projects (most of which pay minimum wage) and our expectation that participants will be re-entering the labor market through STED and ETJD programs. Hourly program staff wages ($19.83) were determined using the national mean wage for community and social service specialists from the Bureau of Labor Statistics. We have estimated the hourly wage of worksite supervisors by using the mean hourly wages of first-line supervisors in two industries from the Bureau of Labor Statistics: retail sales supervisors ($19.18) and production industry supervisors ($27.01). Averaging these wages, we have estimated an hourly wage of $23.10 for worksite supervisors. We expect the wages for supervisors in these industries will be comparable to the wages in jobs which will be employing STED and ETJD participants. The hourly wage estimate for employers is based on the mean hourly wage for Human Resource Managers in the Bureau of Labor Statistics: $52.21 per hour. Finally, since many referrals to STED and ETJD programs will be coming from child support, parole, and/or public assistance agencies, we have estimated the hourly wage of referral partners using the national mean wage for Parole Officers from the Bureau of Labor Statistics: $24.64.

Based on the estimated wages, the total annual cost burden to respondents for this information is $35,346.17 as shown in Exhibit 1.2: Annual Burden Estimates. This figure is based on the mean wages for relevant occupations (community and social service specialists, human resource managers, retail and production industry supervisors, and parole officers) reported in the 2010 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates.

A.13. Other Cost Burden or Respondents and Record Keepers

The proposed data collection will not require the respondents or program staff to purchase equipment or services or to establish new data retrieval mechanisms. Therefore, there are no additional costs to respondents.

A.14. Annualized Cost to the Government.

The estimated annual cost to the federal government for the proposed data collection and analysis is $7,968,517.33. The total cost for all materials included in this package through the end of the project is $23,905,552. These figures include labor hours, other direct costs (photocopying, mailing, travel, etc.) and burden rates.

A.15. Reasons for Any Program Changes or Adjustments

There are no changes or adjustments.

A.16. Tabulation, Analysis, and Publication Plans and Schedule

To determine the effectiveness of the targeted programs, MDRC will collect four categories of data: 1) baseline data, 2) implementation and process data (some of which will support the cost effectiveness study), 3) surveys of research sample members, and 4) administrative records.

The evaluations will include the following components:

• Implementation and process study: The goal of this component is to understand how the programs operate, who they serve, the context in which they operate, and the dosage of services that sample members receive.

• Impact study: The goal of this component is to assess how the STED and ETJD programs affect employment, earnings, and other key outcomes such as recidivism rates, child support payments, well-being, and family relationships.

• Cost effectiveness study: This component will estimate the net cost per person of the STED and ETJD programs and compare those figures to the financial benefits that the program generate (for example, reduced incarceration or public assistance payments).

Major deliverables for both projects are described below.

ETJD Implementation report (2013). This report will serve as an introduction to the project, describing the participating sites, the program models, the research sample members, the early implementation experiences, and participation patterns for the program group.

ETJD Interim impact report (2015). This report will summarize the implementation data from the early report, but will focus mostly on one-year impacts, drawing from both administrative records and the 12-month survey. At this early follow-up point, the report will focus heavily on differences between groups in service receipt and the initial transition from subsidized to unsubsidized employment.

ETJD Final impact report (2017). This report will present results of the data collection for all participants from the second follow-up survey and administrative records as well as the cost-effectiveness analysis. The report will communicate the evaluation findings to a broad audience.

ETJD Public Use File documentation. This report will serve as documentation for the final data sets used by the evaluation, including instructions for data retrieval, code books presenting means and frequencies for all variables, and important decisions made in construction of outcome variables.

STED Profile Report (2013). This report will provide detailed profiles of sites selected to participate in the STED evaluation.

STED Baseline Report (2015). This report will describe the project sites, the random assignment process, and characteristics of the sample.

STED 12-Month Report (2017). This report will describe findings from the implementation study and report impacts on all key outcomes from administrative records and the 12-month survey. We propose to produce a relatively brief main report, with technical appendices providing more detailed tables and explanations

STED 24-Month Report (2018). This report will describe findings from the implementation study and report impacts on all key outcomes from administrative records and the 24-month survey as well as provide a summary of the benefit-cost findings.

The following exhibit outlines the data collection timeline for the STED and ETJD projects.

Revised ETJD and STED Project Timeline

Because data collection will occur for more than 3 years, we will submit a future ICR to update and extend this information collection.

A.17. Reasons for Not Displaying OMB Approval Expiration Date

The expiration date for OMB approval will be displayed on all forms completed as part of the data collection.

A.18. Exceptions to Certification Statement

No exceptions are necessary for this information collection.

1 Revised to include alternate survey instruments which were developed for the youth/young adult STED sites. All changes from the original approved package are highlighted.

2 Berlin (2000).

3 This and other transitional jobs projects are discussed in more detail in Appendix A.

4 The first survey will be administered while research group members are engaged in the study program or soon thereafter and so the exact timing of administration will vary from three to nine months depending on the length of the intervention in each site.

5 The tracking letter that is included in Appendix E is an example of the first letter that will be sent to participants. Subsequent letters will be nearly identical in content, but will be updated to reflect the number of months since random assignment and will promise $5 for responding.

6Unlike the Participant Questionaire discussed earlier, the 6-month survey will be administered to both the program and control groups. Thus, it focuses on measuring differences between the groups.

7 Another benefit of CATI/CAPI is that interviewers can focus on the respondent rather than management of the survey instrument, creating a more pleasant experience for the respondent. The technology ensures that scheduled appointments are honored with respondents. CATI/CAPI also ensures adherence to dialing protocols, maintaining the integrity of the study without unduly burdening sample members and eliminates many human errors, such as accidental calling of resolved sample records. Finally, CATI provides translated scripts for crisp script delivery to non-English speakers.

8 Berlin, M., L. Mohadjer and J. Waksberg (1992). An experiment in monetary incentives. Proceedings of the Survey Research Section of the American Statistical Association, 393-398; de Heer, W. and E. de Leeuw. “Trends in household survey non-response: A longitudinal and international comparison.” In Survey Non-response, edited by R. M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little. New York: John Wiley, 2002, pp.41-54; Singer, E. and Kulka, R. Studies of Welfare Populations: Data Collection and Research Issues, Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs. Ploeg, Robert A.Moffitt, and Constance F.Citro, Editors. National Academies Press, Washington, DC, 2000, pp. 105-128.

10