50447 AP_OMB Part A Data Collection_R8-clean to DOL ver2X

PART A: JUSTIFICATION for Evaluation of America’s Promise Job-Driven Grant Program

OMB No. 1290-0NEW

September 2018

Part A: Justification

The Chief Evaluation Office of the U.S. Department of Labor (DOL) has commissioned an evaluation of the America’s Promise Job-Driven Grant program (America’s Promise). This program aims to create or expand regional partnerships that will identify the needs of specific industry sectors relying on the H-1B visa program to hire skilled foreign workers and prepare the domestic workforce for middle- and high-skilled, high-growth jobs in those sectors. The America’s Promise evaluation offers a unique opportunity to build knowledge about the implementation and effectiveness of these regional partnerships. Mathematica Policy Research and its subcontractor Social Policy Research Associates have been contracted to conduct an implementation and impact evaluation. This package requests clearance for two data collection activities as part of the implementation evaluation:

1. Grantee survey

2. Partner network survey

A future information collection clearance request will be submitted with additional implementation evaluation instruments, including protocols for semi-structured telephone interviews and protocols for program stakeholder interviews and focus groups for use during site visits. The impact evaluation will rely on administrative data (no primary data will be collected).

A.1. Circumstances making the collection of information necessary

A skill gap between the qualifications of American workers and the needs of many American businesses has persisted. U.S. firms annually sponsor hundreds of thousands of nonimmigrant H-1B visas to fill skilled positions.¹ To reclaim some of these jobs for the American workforce, in January 2017, DOL awarded more than $110 million to 23 grantees for America’s Promise. The purpose of these four-year grants is to support local partnerships between workforce agencies, employers, industry representatives, training providers, community-based organizations, and economic development agencies to identify the needs of specific industry sectors relying on the H-1B visa program for workers and implement career pathway programs that build the skills of the domestic workforce for middle- and high-skilled jobs in those sectors.

The America’s Promise grant program and subsequent evaluation are authorized by Title 29 of the American Competitiveness and Workforce Improvement Act, which states that “the Secretary of Labor shall . . . award grants to eligible entities to provide job training and related activities for workers to assist them in obtaining or upgrading employment in industries and economic sectors . . . projected to experience significant growth and ensure that job training and related activities funded by such grants are coordinated with the public workforce investment system (29USC3224(a)).”

A.2. Purposes and use of the information

The data collected through the activities summarized in this request will be used by DOL to comprehensively describe implementation of the America’s Promise grant program, including its partnerships, training and support services provided, target population, and common implementation successes and challenges. The evaluation will also assess the impacts of America’s Promise on participant outcomes. These data and the study team’s descriptive and impact analyses will provide DOL and other policymakers with important information to guide management decisions and future planning efforts regarding such grant programs.

1. Overview of the evaluation

The evaluation of America’s Promise includes two components: (1) an implementation evaluation to understand program implementation and partnership development and (2) an impact evaluation to measure the effects of America’s Promise on participant outcomes. It will take place over five years (2017 to 2022), will address the following research questions:

1. How were regional partnerships developed and maintained? What factors influenced the partnership development and employer engagement?

2. What types and combinations of services and approaches were provided? How were they implemented?

3. What were the characteristics of enrolled participants?

4. What was the community context of the America’s Promise grantees?

5. What impact did America’s Promise have on participants’ labor market outcomes?

6. How did the impact of America’s Promise vary by participant characteristics or program components?

The implementation evaluation component will answer research questions 1-4. This component includes a grantee survey involving all 23 grantees, review of grant documents from all 23 grantees, a partner network survey involving approximately six grantees, program stakeholder interviews and focus groups during site visits to approximately 12 grantees, and telephone interviews with program stakeholders at approximately 11 grantees. The current clearance request includes the grantee and partner network surveys. Protocols related to the site visits and telephone interviews will be covered in a future clearance request. The impact evaluation component will use administrative data to address research questions 5 and 6. This means that no primary data will be collected to estimate impacts.

The first step in planning the evaluation was to review information from the 23 America’s Promise applications to identify sites that appear suitable for the various components of the implementation evaluation. In selecting sites, DOL will prioritize grantees that are implementing promising or innovative strategies. In addition, DOL will aim to select a diverse set of grantees that vary in such factors as the structure and maturity of partnerships, number and strength of employer partnerships, type of sector, population served, type of training, urbanicity, and region. As part of the implementation evaluation, the partner network survey data collection will prioritize grantees that have developed promising partnerships or partnership activities, such as grantees with active advisory councils, engaged employer partners, partners dispersed throughout the region, or engaged new types of partnerships.

2. Overview of the data collection

Understanding the implementation and effectiveness of America’s Promise requires data collection from multiple sources. The implementation evaluation data collection covered by this clearance request includes a grantee survey and a partner network survey. These data collection activities are as follows:

1. Grantee survey. The purpose of this survey, to be administered to all 23 America’s Promise grantees in fall 2018, is to collect information about America’s Promise grantees’ characteristics, program features, program services, partner participation, and early implementation challenges and successes. The survey is expected to take approximately 30 minutes to complete with an anticipated 100 percent response rate.

2. Partner network survey. In a subset of approximately six implementation study sites, the partner survey will be administered twice to the grantees and approximately 24 of each grantee’s partners in early 2019 and summer 2020. The survey will collect information about partner characteristics, role in the partnership, perspectives on partnership goals, frequency and type of collaboration with other partners, and assessment of partnership quality. The survey is expected to take approximately 20 minutes to complete, and we anticipate that 80 percent of partners will respond.

Proposed uses for each data collection activity are described in Table A.1. A full version of each instrument is attached to this Justification Statement.

Table A.1. How data will be used, by data collection activity

Data collection activity	How the data will be used
1. Grantee survey	We will conduct descriptive analyses to describe the America’s Promise grantees’ characteristics, program features, program services, partner participation, and early implementation challenges and successes.
2. Partner network survey	We will conduct descriptive analyses for a subset of implementation study grantees to describe changes over time in partner characteristics, role in the partnership, perspectives on partnership goals, frequency and type of collaboration with other partners, and assessment of partnership quality.

A.3. Use of technology to reduce burden

Data collection will be conducted using advanced technology to reduce burden on program participants and grantee and partner staff whenever possible.

The grantee and partner network surveys will be administered via web survey, so that they can be accessed from any computer, allowing for the greatest ease of access. For each survey, respondents can partially complete it and return to it later if desired. Each survey will use logical skips, check boxes, and drop-down menus to reduce burden.

A.4. Efforts to avoid duplication

The America’s Promise evaluation will not require collection of information that is available through alternate sources. For example, the study will use available information from grantee applications and existing administrative data sets to ensure that data collected through grantee and partner surveys is not available elsewhere.

A.5. Methods to minimize burden on small entities

The data collection effort is not expected to involve small businesses or other small entities. However, some respondents to the partner survey could be small entities, such as community based organizations. We will only request information required for the intended use and minimize burden by restricting the length of surveys to the minimum required.

A.6. Consequences of not collecting the data

If the grantee and partner surveys are not collected, DOL and other stakeholders will not have the information necessary to understand how America’s Promise was implemented across the grantees or to learn about implementation challenges and promising strategies. Without this information, DOL and future grantees would not be able to replicate the program if it is found to be effective. Furthermore, without these data, federal policymakers will not be able to describe the full range of America’s Promise grantee characteristics, program features, types of partners and the extent of partner collaboration, and implementation experiences.

A.7. Special circumstances

No special circumstances apply to this data collection.

A.8. Federal Register announcement and consultation

1. Federal Register announcement

A 60-day notice to solicit public comments was published in the Federal Register, 82 FR 32204 on July 12, 2017. One comment was received. It suggested that the program was ineffective and costly, but no evidence was provided in support of the statement. DOL acknowledged receipt of the comment. The purpose of this study is to provide evidence as to the effectiveness of the program and understand strategies to support on-going program improvements.

2. Consultation outside the agency

Consultation on the research design and data needs is being coordinated by the study team and involves discussions with experts and program staff. The purpose of consultation with outside experts is to ensure the technical soundness of the study and the relevance of study findings and to verify the importance, relevance, and accessibility of the information sought in the study. These experts participating in the evaluation technical working group are listed in Table A.2. The purpose of the consultation with program staff is to better understand the feasibility of the research design within the regional context of grantees.

Table A.2. Individuals providing consultation on America’s Promise evaluation design

Peter Mueser, PhD

Professor, Department of Economics and Truman School of Public Health

University of Missouri

Columbia, MO 65211

Mary Alice McCarthy

Director, Center on Education and Skills

New America

740 15th Street NW, Suite 900

Washington, DC 20005

Margaret Hargreaves

Principal Associate

Community Science

438 N. Frederick Avenue, Suite 315

Gaithersburg, MD 20877

A.9. Payment or gifts

There are no payments or gifts to respondents. Tasks and activities conducted by program and partner staff are expected to be carried out in the course of their employment, and no additional compensation will be provided outside of their normal pay.

A.10. Assurances of privacy

Information collected will be kept private to the extent permitted by law. The study team complies with DOL data security requirements by implementing security controls for processes that it routinely uses in projects that involve sensitive data. Further, the study is being conducted in accordance with all relevant regulations and requirements.

A.11. Justification for sensitive questions

There are no sensitive questions included in the grantee survey or the partner network survey.

A.12. Estimates of burden hours

Table A.3 provides annual burden estimates for each of the data collection activities for which this package requests clearance. All of the activities covered by this request will take place within about a three-year period. To calculate the estimated cost burden for respondents, average hourly wages from the U.S. Bureau of Labor Statistics, National, State, Metropolitan, and Nonmetropolitan Area Occupational Employment and Wage Estimates for May 2017 were multiplied by the number of hours per respondent type. The following summarizes the annual burden estimates for each of the two data collection activities:

1. Grantee survey. We will administer the grantee survey with 23 grantee directors using a web-based format. Each survey will take 30 minutes hours to complete. The total burden for the grantee survey is 11.5 hours; the annualized burden is 3.83 hours.

2. Partner network survey. We will administer the partner network survey in a web-based format. Grantees and approximately 24 of their partners in six sites will be asked to participate in the survey, for a total of 150 respondents ((1 grantee + 24 partners) × 6 sites). Each survey will take 20 minutes to complete, and each respondent will complete the survey two times. As a response rate of 80 percent is expected, total burden for the network survey is 80 hours (120 respondents × 20/60 hours × 2); the annualized burden is 27 hours.

Table A.3. Estimated Annualized Respondent Hour and Cost Burden

Data Collection Activity	Annual number of respondents	Number of responses per respondent	Total Number of Responses	Average burden per response (in Hours)	Annual estimated burden hours	Average hourly a	Annual monetized burden hours
Grantee survey	8	1	8	30/60	4	$44.41	$178
Partner network survey	40	2	80	20/60	27	$44.41	$1,199
Unduplicated Total	48	--	88		31		$1,377

^a The hourly wage of $44.41 is the May 2017 median wage across Education Administrators, Postsecondary (see http://www.bls.gov/oes/current/oes_nat.htm)

A.13. Estimate of cost burden to respondents

There are no additional costs to respondents other than their time.

A.14. Annualized costs to the federal government

The total annualized cost to the federal government is $52,723. Costs result from the following categories:

The estimated cost to the federal government for the contractor to carry out this study is $29,400 for the grantee survey and $83,552 for the partner network survey. Annualized, this comes to $37,651:

= $37,651

The annual cost borne by DOL for federal technical staff to oversee the contract is estimated to be $15,072. We expect the annual level of effort to perform these duties will require 200 hours for one federal GS 14 step 4 employee based in Washington, D.C., earning $47.10 per hour. (See Office of Personnel Management 2018 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/GS.pdf). To account for fringe benefits and other overhead costs, the agency has applied multiplication factor of 1.6:

200 hours × $47.10 × 1.6 = $15,072.

Thus the total annualized federal cost is $37,651+ $15,072 = $52,723.

A.15. Reasons for program changes or adjustments

This is a new information collection.

A.16. Plans for tabulation and publication of results

1. Analysis plan

The study team will summarize quantitative data collected through the grantee and partner surveys using basic descriptive methods. Analysis of data from each source will follow a common set of steps involving data cleaning, variable construction, and computing descriptive statistics. To facilitate analysis of each data source, the team will create variables to address the study’s research questions. The study team will use network analysis to study grantee partnerships. The partner network survey will include a set of network questions in which respondents report on their relationships as measured by the frequency, type, and focus of communication with all other key partner respondents. These data will be used to describe the levels of communication and collaboration between partners as well as changes over time.

2. Publications

In early 2022, we will produce a report on the implementation and impact evaluations as well as other dissemination products such as fact sheets and issue briefs on topics of interest to DOL.

A.17. Approval not to display the expiration date for OMB approval

The OMB approval number and expiration date will be displayed or cited on all forms completed as part of the data collection.

A.18. Exception to the certification statement

No exceptions are necessary for this information collection.

1 U.S. Department of State. “Report of the Visa Office 2016, Table XVI (B) Nonimmigrant Visas Issued by Classification (Including Border Crossing Cards) Fiscal Years 2012–2016.” Washington, DC: U.S. Department of State, 2017. Available at https://travel.state.gov/content/visas/en/law-and-policy/statistics/annual-reports/report-of-the-visa-office-2016.html