Jobs Plus OMB_Part A FINAL 6-10-16

Jobs Plus Pilot Evaluation Study

Supporting Statement for Paperwork Reduction Act Submission, Part A

Submitted to:

Office of Management & Budget (OMB)

Submitted by:

U.S. Department of Housing & Urban Development (HUD)

June 13, 2016

Supporting Statement for Paperwork Reduction Act Submission, Part A

Jobs Plus Evaluation Study

SUMMARY

In 2015, the U.S. Department of Housing and Urban Development (HUD) awarded $24 million in funds to nine Public Housing Authorities (PHAs) to launch the Jobs Plus Pilot Program. The purpose of the Jobs Plus Pilot Program is to develop locally-based, job-driven approaches to increase earning and advance employment outcomes for residents of public housing. The place-based Jobs Plus Pilot Program addresses poverty among public housing residents by incentivizing and enabling employment through income disregards for working families, and a set of services designed to support work, including employer linkages, job placement and counseling, educational advancement, and financial counseling. Ideally, these incentives will saturate the target developments, building a “culture” of work and making working families the norm. The Jobs Plus model was designed in the late-1990s to address some of the challenges that public housing residents face when searching for work. The original demonstration showed that when implemented properly, Jobs Plus increased annual earnings by 16% and these effects endured for 7 years without abating. Jobs Plus is an ambitious model, however, that requires planning and delivering the intervention through new interagency and resident partnerships, bringing employment assistance directly into public housing developments, crafting new rent rules to make work pay, fostering neighbor-to-neighbor support for work, and offering assistance on a “saturation” basis.

HUD is partnering with an external research team to evaluate the Jobs Plus Pilot program. A logic model that helps guide the researcher’s approach to Jobs Plus assessment is attached as Exhibit 1. The evaluation includes a process study in nine sites where the program is implemented to understand how sites approach early implementation of the program, in-depth case studies to gain insights as to practice challenges and breakthroughs in these settings, a cost study to understand the costs associated with the program, and administrative data analyses of HUD’s PIH Information Center (PIC) data to understand the background characteristics and reported employment and earnings of development residents over time. (While not the direct subject of PRA materials, administrative data will be used primarily to describe background and program context, and will not suggest early program impacts in the absence of a rigorous counterfactual.)

As described in Exhibit 1, while the process study will assess program inputs, activities, and other contextual factors that may influence longer-term program impacts – primarily through qualitative means associated with the enclosed instruments – another goal of the research is to understand early program outputs, as appropriate for the start-up of a new federal initiative. Although not the subject of the attached instruments, these outputs include, for example, grantee reports of service take up, job placements, and use of Jobs Plus rent incentives.

This Paperwork Reduction Act (PRA) submission includes the instrumentation required for use under the Jobs Plus Evaluation Study.

A. JUSTIFICATION

1. Circumstances that make the collection of information necessary

In 2015, HUD awarded $24 million in funds to nine public housing authorities to replicate the Jobs Plus model. HUD’s replication of the Jobs Plus Pilot Program raises important questions related to the scale up of the model and offers a unique opportunity to compare how varying local conditions and institutional contexts influence the ways the Jobs Plus components are put into practice.

This process study is important for HUD to understand how the Jobs Plus Pilot Program is implemented in the first cohort of nine grantees. The original demonstration of Jobs-Plus, conducted in the mid-1990s, found that the program had significant impacts on residents’ employment and earnings when it was fully implemented. However, the research also indicated that Jobs-Plus is a very difficult program to implement. HUD has modified the program in several significant ways to make the program easier to implement, such as creating a simplified Earned Income Disregard (the Jobs Plus Earned Income Disregard) and providing support and guidance to grantees. In this iteration of the Jobs Plus Pilot program, it is essential that we learn whether and how developments are able to implement the program, and what hurdles challenge implementation. Aside from providing insight into the program at this stage, this process study will help establish a background to help interpret results from a future outcomes evaluation.

The conduct of the process study and the outcomes study will overlap, and the outcomes study is being structured specifically to take advantage of the background knowledge collected through the process study. The process study covers the first cohort of 9 grantees who will operate their Jobs Plus programs from April, 2015 through March, 2019. The outcomes study will use income and employment data from the National Directory of New Hires. This dataset will allow the research team to observe individual’s income and employment prior to the beginning of the Jobs Plus programs and extending beyond the end of the program. By studying the same sites in this process study and a future outcomes study, HUD will form an in depth understanding of how the Jobs Plus program impacted these developments.

HUD has entered into a cooperative agreement with an independent research team to document the start-up of the Jobs Plus Pilot Program, its early outcomes and its costs through a comprehensive process study that compares experiences across nine housing developments where the program is implemented. The evaluation will use a mixed-method comparative approach to understand the implementation of each program component in nine sites where the Jobs Plus Pilot is implemented, costs associated with implementation, and early program outcomes. The study will also lay the groundwork for a rigorous design of a future impact study.

The evaluation of the Jobs Plus Pilot includes two rounds of site visits to all sites (a total of 18 site visits altogether) to conduct interviews with key staff and stakeholders and focus groups with residents. Additionally, the study includes analyses of program participation data, as reported by sites to HUD, to help understand early outcomes and reviews of cost data to help determine costs associated with implementation of the program.

This Information Collection Request (ICR) includes four data collection instruments: (a) Site Visit Interview Guide Round 1 (included as Appendix A) to be administered to program staff and stakeholders in nine target developments, including program directors, housing authority senior managers, property management staff, selected frontline staff, resident leaders, community stakeholders, and service partners (b) Focus Group Discussion Guide Round 1 (included as Appendix B) to be administered to public housing residents in nine target developments, (c) Site Visit Interview Guide Round 2 (included as Appendix C) to be administered to program staff and stakeholders in nine sites, including program directors, housing authority senior managers, property management staff, selected frontline staff, resident leaders, community stakeholders, and service partners, and (d) Focus Group Discussion Guide Round 2 (included as Appendix D) to be administered to public housing residents in nine target developments.

This research is conducted under the authority of the Secretary of the U.S. Department of Housing and Urban Development to undertake programs of research, studies, testing and demonstration related to the mission and programs of HUD (12 USC 1701z-1 et seq.).

2. Purpose of the Information Collection

Information collected through this study will be used to document the establishment and early implementation of the Jobs Plus Pilot program, and to lay the groundwork for a future outcomes evaluation that will seek to understand the impact of the program on employment outcomes for development residents. In addition, this evaluation will assess how well developments were able to implement the Jobs Plus Pilot program and will inform HUD’s management of future grantees. By identifying challenges faced by the first cohort of grantees, HUD can use this information to provide additional guidance to future grantees on areas which have proved to be challenging. In addition to better understanding what hampers implementation, the study will also analyze which strategies improve program management. By comparing implementation strategies along with early outcome measures across grantees, this study will suggest what contributes to successful implementation of the Jobs Plus program.

More specifically, the information collection will help document (a) how the three components of the Jobs Plus model are implemented in the early stages of the program, (b) the costs associated with implementing the program, (c) the extent to which grantees achieve saturation and variation across sties, as well as (d) early outcomes achieved by different subgroups within the target development. Additionally, the information collected will serve to provide a framework for an impact study in the future.

HUD is seeking onetime approval for the administration of the four instruments within this ICR: (a) Site Visit Interview Guide Round 1, (b) Focus Group Discussion Guide Round 1, (c) Site Visit Interview Guide Round 2, and (d) Focus Group Discussion Guide Round 2.

The Site Visit Interview Guide Round 1 will include themes for staff and key stakeholders that will help us understand early implementation of the three components of the Jobs Plus model. In particular, this instrument will help answer questions related to the development of the institutional infrastructure and partnerships to implement the program, how the program is advertised to residents, the overall operation of the program, employment services offered, execution of financial incentives, activities related to community support for work, as well as challenges and accomplishments related to startup of the program. The Focus Group Discussion Guide Round 1 will ask residents how they view the program’s advertisement in the development, early services received by residents, and overall views and understanding of the program. The Site Visit Interview Guide Round 2 will address questions related to evolution of employment services, take up of the Jobs Plus Earned Income Disregard (JPEID), promising strategies to engage residents of the target development, any corrective actions taken, overall quality of services provided, and progress towards “saturation,” or the offer of services to all work-able residents of the development. The Focus Group Discussion Guide Round 2 will ask questions related to messaging, interactions and experiences with the program, and resident assessments of the program.

The findings resulting from the first and second round of site visits of the evaluation will be presented in an interim and final report, respectively, prepared by the independent research team for the Office of Policy Development and Research (PD&R). The independent research team that has been selected by HUD to conduct the evaluation of the Jobs Plus Pilot program will administer all data collection instruments described in this Information Collection Request.

3. Use of information technology

As a process study, the use of automated and electronic data is not appropriate. We will make every effort to reduce the burden on the grantees. The collection of information includes (a) Site Visit Interview Guide Round 1, (b) Focus Group Discussion Guide Round 1, (c) Site Visit Interview Guide Round 2, and (d) Focus Group Discussion Guide Round 2. The interviews will be conducted by an experienced researcher who, whenever possible, will be accompanied by a junior researcher. The interviews will be carried out in person and audio recorded, if the respondent agrees to be recorded. The recordings will be shared only with lead researchers and team members on a “need to know” basis. The focus groups will be carried out in person and audio recorded, granted prior consent of participants. We anticipate using a random sample of program participants to select focus group participants. The sample will be stratified by groups of interest, including but not limited to residents working or not working at program enrollment, with or without young children, and with different levels of engagement within the program.

All information will be protected and held confidentially.

4. Describe efforts to identify duplication

There is no existing data source that can readily be analyzed to document early implementation of the Jobs Plus Pilot Program. Available data from HUD administrative sources will be incorporated as part of the evaluation of the program, but these cannot capture the experiences of program implementation, such as the strategies employed to reach early outcomes. Similarly, there is no existing data source that can be analyzed to document residents’ views and experiences in the program since the program is being implemented anew in the target developments.

5. Impact on small businesses or other small entities

Small businesses are not part of this information collection effort and thus we do not anticipate that this study will burden small businesses.

6. Consequences if the collection is not conducted or is conducted less frequently

The evaluation of the Jobs Plus Pilot program represents an important opportunity for HUD to learn about effective replication and scale up of a program that has shown significant and sustained improvements in annual earnings of public housing residents. Subsequent cohorts of Jobs Plus grantees are expected to launch the Jobs Plus program over the next year (for example, nine new PHAs were awarded grants in December 2015), and the lessons learned through this evaluation will enable HUD to strengthen the program based on the experiences of the initial cohort of grantees. Furthermore, the study will provide assistance to HUD as it identifies appropriate comparison development for future impact study that would assess the effectiveness of the replication and shed light on the relative contributions of different aspects of the model. If this data collection were not to take place, the evaluation of this pilot program would be dramatically reduced, with no way to observe and document the implementation challenges faced by the grantees or the feedback of the residents regarding the program. As a program that HUD is continuing to fund in future years, the Department seeks to make every effort to ensure the successful implementation of the program. Without this type of data collection, the Department’s ability to provide guidance to future grantees on successful program implementation is limited.

7. Special circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320 (Controlling Paperwork Burdens on the Public). There are no special circumstances that require deviation from these guidelines. Under this ICR, HUD will not conduct any data collection:

• requiring respondents to report information to the agency more often than quarterly;

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• requiring respondents to submit more than an original and two copies of any document;

• requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;

• in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

8. Federal Register Notice/Consultation Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, HUD published a 60-Day Notice of Proposed Information Collection in the Federal Register on December 28, 2015. The Docket No. FR-5837-N-06 and the notice appeared on pages 80790-80791. The notice provided a 60-day period for public comments, and comments were due February 26, 2016. No public comments on the proposed information collection were received. A copy of the notice is included as Appendix E.

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

No payments or gifts will be provided to staff or key stakeholders for completing an in-person interview during Round 1 or 2 of site visits. Residents who participate in focus groups will receive compensation of $25 for their time. This amount is comparable to compensation HUD has offered in similar studies. Three factors helped to determine the incentive amounts for the focus group: 1) participant burden; 2) costs associated with the amount of time that the focus group participant will commit to focus group participation; and 3) other studies of comparable populations and burden.

10. Assurance of confidentiality

HUD has entered into a cooperative agreement with an independent research team to conduct this research effort. HUD and the research team will make every effort to maintain the privacy of respondents, to the extent permitted by law. The independent research team maintains an Institutional Review Board (IRB) to ensure that research practices and procedures effectively protect the rights and welfare of human subjects. All respondents included in the study will be informed that information they provide will be used only for the purpose of this research. Individuals will not be cited as sources of information in prepared reports. Respondents included in the study will be asked for verbal consent to participate in interviews or focus groups (See Appendix F for Written Statement to Obtain Oral Consent). All research staff working on the project have been trained to protect private information and have signed a pledge stating that they will keep all information gathered private to the extent permissible by law. All papers that contain participant names or other identifying information will be kept in locked areas and any computer documents containing identifying information will be protected with a password.

11. Questions of a sensitive nature

Some questions to be asked through this information collection, specifically the data collection proposed from residents through focus groups, may potentially be sensitive for some respondents, in that topics related to challenges in the lives of residents (such as struggles to make ends meet) may arise. The data collection proposed in the site interview guides with program staff and key informants focuses on the implementation of the program and therefore does not contain questions of a sensitive nature. Nonetheless, all respondents will be informed that their answers are confidential, that they may refuse to answer any questions, and that results will only be reported in the aggregate. Additional assurances will be offered to focus groups participants, who will be clearly informed that they do not need to respond to questions in reference to their own lives or experiences, that their answers are confidential, that they may refuse to answer any questions, that results will only be reported in the aggregate, and that their responses will not have any effect on any services or benefits they or their family members receive.

12. Provide an estimate of the burden, in hours and costs, of the collection of information

The hour burden estimates for data collection for the process study, which includes interviews with staff and key stakeholders, as well focus groups with participants, is outlined in Table 1. We have assumed the maximum possible number of study participants. The estimates included in Table 1 are based on experience with previous implementation studies involving similar populations and data collection instruments.

Table 1: Data Collection Activities and Anticipated Burden

Instrument	Respondent Sample	Number of Respondents	Average time to complete( hours)	Frequency	Total burden (hours)
Site Visit Interview Guide Round 1	12 staff and stakeholders from all 9 Jobs Plus Pilot sites	108	1.5	1	162
Focus Group Discussion Guide Round 1	15 residents at each of the 9 Jobs Plus Pilot sites	135	1	1	135
Site Visit Interview Guide Round 2	12 staff and stakeholders from all 9 Jobs Plus Pilot sites	108	1.5	1	162
Focus Group Discussion Guide Round 2	15 residents at each of the 9 Jobs Plus Pilot sites	135	1	1	135
Total Burden Hours					594

13. Other Cost Burden to Respondents and Record Keepers

This data collection effort involves no recordkeeping or reporting costs for respondents other than the time burden to respond to questions on the data collection instruments as described in item 12 above. There is no known cost burden to the respondents.

14. Provide estimates of annualized costs to the federal government

The estimated cost to the federal government for the data collection, including instrument development and deployment with nine sites over the course of two visits per site, and other expenses directly related to instrument use, totals $400,000 over a thirty-six month period, for an annualized cost of $133,000/year.

“Data Collection” is comprised of three separate and distinct activities: 1) Data collection, including in-depth case study observations, 2) Analysis, including coding and analysis of the qualitative and quantitative data, and 3) Interim information dissemination, including feedback to HUD and the grantees to promote stronger learning and performance loops.

Data Collection:

1. Data Collection through Round I, Round II and In-Depth Case Study Site Visits – $400,000

1. Training, review and monitoring of staff participating in site visits

2. Review of program documents

3. Conducting two rounds of site visits at nine sites, involving:

• Preparation and scheduling

• Focus group recruitment and follow-up to ensure attendance

• Conducting interviews, writing notes to complement transcripts

• Observing practice, writing notes and synthesizing observations

4. Multiple case study observations at three sites

5. Annual IRB reviews

6. Additional costs including transcription and travel costs

Analysis:

2. Preparation of Codebook and Coding itself – $70,000

   1. Codebook development and training

   2. Interview Transcription & Uploading

   3. Coding

3. Analysis of Process Study Data – $200,000

   1. Analysis of the coded process study data for the interim and final reports

   2. Generation of thematic queries and qualitative comparison

4. Analysis of participation data and program costs (separate from site visits) – $100,000

   1. Quantitative analysis of program participation

   2. Cost study across nine sites, including in-depth study of in-kind costs

Interim Information Dissemination:

5. Information Dissemination (Meetings and Communication)  – $200,000

   1. Developing and conducting webinars for HA’s to review round I findings and the interim and final report

   2. Ongoing Communication with PHAs in the form of monthly calls

   3. Briefings with HUD to review round I & II field visit findings

15. Explain the reasons for any program changes or adjustments

This submission is a new request for approval; there is no change in burden.

16. Tabulation, Analysis, and Publication Plans and Schedule

Table 2 shows the report schedule for this project.

Table 2: Report Schedule for the Jobs Plus Pilot Program Evaluation

Deliverable	Focus	Schedule*
Round 1 Site Visits	First wave of data collection from grantees and residents	June – August 2016
Data Analysis	Analysis will focus on synthesizing data collected from round 1 site visits, focus group feedback, programmatic data, and any available administrative data	September 2016 – March 2017
Interim report	Report focusing on early start-up of Jobs Plus.	06/15/17
Round 2 Site Visits	Second wave of data collection from grantees and residents	January 2017 – March 2017
Data Analysis	Analysis will focus on synthesizing data collected from round 2 site visits, focus group feedback, programmatic data, cost data, and any available administrative data	April 2017 – December 2017
Final Report	Report focusing on overall accomplishments of Jobs Plus through 2018, the costs of the initiative, and lessons for HUD and future implementers of the program.	08/15/18

* All dates are subject to change based on the timing of OMB approval

17. Reasons for not displaying OMB approval expiration date

The expiration date for OMB approval will be displayed on any forms completed as part of the data collection.

18. Exceptions to Certification Statement

No exceptions are necessary for this information collection.

Page 9 – Part A Supporting Statement