Workforce innovation and opportunity act (WIOA) Implementation evaluation,
OMB No. 1290-XXXX
August 2018
In this document, the Department of Labor (DOL) requests clearance from the Office of Management and Budget (OMB) under the Paperwork Reduction Act (PRA) for a new collection: semi-structured interviews with state- and local-level workforce, adult education, and vocational rehabilitation staff, associated with the Workforce Innovation and Opportunity Act (WIOA) Implementation Evaluation.
WIOA (Pub. L. 113-128), signed into law on July 22, 2014, aims to transform and modernize the public workforce system. Building on reforms under its predecessor, the Workforce Investment Act (WIA), WIOA’s overarching goals include improving the quality and outcomes of workforce services; ensuring that workforce services are evidence-based and guided by the needs of business and workers; and increasing access to the public workforce system, especially among individuals with significant barriers to employment. Many of the changes to the workforce system under WIOA went into effect on July 1, 2015, and additional changes have been implemented through June 30, 2018.
The WIOA implementation evaluation was funded by the DOL Chief Evaluation Office (CEO) in partnership with the Employment and Training Administration (ETA) with the goal of better understanding the implementation of WIOA, the variations in implementation across states and localities, and the need for further administrative guidance, regulations, TA, and policy. DOL contracted with Mathematica Policy Research in partnership with Social Policy Research to conduct the evaluation. The implementation evaluation and other evaluations of WIOA are also authorized by Section 169 of WIOA which states that “the Secretary, through grants, contracts, or cooperative agreements, shall provide for the continuing evaluation of the programs and activities under this title, including those programs and activities carried out under this section” (Sec. 169(a) 2014).
DOL is seeking clearance for instruments for two data collection activities through two separate clearance packages:
Semi-structured interviews with state- and local-level workforce, adult education, and vocational rehabilitation staff, which will be administered during site visits to approximately 14 purposively-selected states and approximately 28 localities, over a six-month period (clearance requested in this package); and
A survey of state-level workforce administrators in 50 states and the District of Columbia (clearance requested in a separate package; see 60-day Federal Register notice, 83 FR 8110, February 23, 2018).
In this clearance request, we seek approval for the semi-structured interviews with state- and local-level workforce, adult education, and vocational rehabilitation staff. The interviews will be conducted during site visits to approximately 14 states and approximately 28 localities (this will include both state- and local-level respondents). The site visits and semi-structured interviews are necessary to enable an in-depth exploration of state implementation of the provisions of WIOA across the country. The semi-structured interviews will allow collection of rich qualitative data on the experiences of state workforce agencies and their partners in implementing the law. Understanding the breadth of experiences of diverse states and localities during their implementation of WIOA requires a significant sample of states, with multiple local areas in each state. This information is not available through other sources in a comprehensive and systematic way.
While the U.S. economy has largely recovered from the Great Recession, many employers still struggle to find skilled workers, a problem that hampers the growth and productivity of the overall economy. Workers with low skills, disabilities, or other barriers to employment, the long-term unemployed, and others unattached to the workforce continue to face challenges finding and maintaining meaningful employment. WIOA seeks to address these challenges. To support implementation of the Act’s comprehensive reforms, DOL has provided extensive guidance and TA to state administrators, including draft and final rule documents, numerous Training and Employment Guidance Letters and Unemployment Insurance Program Letters, a “WIOA Implementation” TA website, and ongoing in-person and webinar trainings and sponsored peer-learning groups for state and local workforce stakeholders.
The WIOA implementation evaluation will examine the progress and variations in implementation across states and localities and assess their needs for further administrative guidance, regulations, technical assistance (TA), and policy changes. It will examine whether the implementation of WIOA is changing the public workforce system and core programs authorized under Title I (Adult, Dislocated Worker, and Youth programs) and the Employment Services program authorized under the Wagner-Peyser Act and amended by Title III. It also will assess whether implementation of the Act is contributing to greater integration with stakeholders in core programs authorized under Titles II (Adult Education and Literacy) and IV (Vocational Rehabilitation).
The evaluation will address the following high-level research questions:
What changes under WIOA Titles I and III have been implemented, and what is still planned? How does implementation vary across states? What implementation challenges are states experiencing and how are they being addressed?
To what extent is WIOA’s vision for an integrated workforce system being achieved through state and local level synergies between Titles I and III and Titles II and IV stakeholders?
What additional TA, guidance, or policy changes would help states better administer core programs and provide guidance and oversight at the local level?
To inform the above research questions, DOL seeks approval under this request to conduct semi-structured, in-depth interviews with key stakeholders in WIOA implementation at the state and local levels, including (1) workforce board staff, (2) Title I Adult and Dislocated Worker (A/DW) program staff, (3) Title I Youth program staff, (4) Titles II and IV staff, (5) Title III staff, (6) state Unemployment Insurance (UI) staff (state-level only), (7) partner staff (such as Temporary Assistance for Needy Families [TANF] and UI), (8) local American Job Center (AJC) operators (local-level only), and (9) local AJC managers (local-level only). At the state level, site visitors will meet with lead staff at the state workforce board and relevant state agencies. At the local level, site visitors will meet with local workforce board representatives, and will visit an AJC to meet with the operator and manager, as well as local (or regional, where appropriate) representatives of the core and partner programs/agencies. Responses to the interviews are voluntary.
This package seeks clearance for interview protocols for state-level respondents and the interview protocols for local-level respondents. We will use the information produced by the evaluation to develop additional TA, guidance, and policies that can affect the workforce system and facilitate ongoing implementation of WIOA provisions.
Site visitors for the WIOA Implementation Evaluation will use electronic audio recorders to record the semi-structured interviews. This will allow the visitors to conduct interviews in the shortest amount of time possible, as they will not be required to use interview time to take notes on the content of the conversation. There will be no other information technology used by site visitors. The evaluation team will schedule site visits by telephone and will conduct the site visit interviews in person.
The WIOA Implementation Evaluation aims to collect in-depth qualitative data on all aspects of WIOA implementation in approximately 14 states. Data on some specific topics related to implementation are, or will be, available from other sources, but the sample size, mode of data collection, or research topic of those sources is too focused to be duplicative of this evaluation. Studies conducted by the Government Accountability Office have focused specifically on performance reporting, strategic planning, out-of-school youth, and Federal agency collaboration. The National Association of State Workforce Agencies has administered and plans to administer questionnaires with states on topics including workforce data systems, research capacity, staff capacity, accounting information technology systems, supplemental funding, and unemployment insurance tax. This evaluation’s site visits are designed instead to provide a comprehensive picture of WIOA implementation through rich qualitative data, and the evaluation team’s semi-structured protocols are unlikely to be similar to those used in other data collection efforts.
Some of the site visit respondents may also participate in the survey of Title I and III workforce programs (discussed in a separate clearance package) and may respond to questions on topics similar to topics in the semi-structured interviews. However, the survey questions will generally have closed-ended response options to facilitate a descriptive analysis of WIOA implementation across all states, while site visit questions are open-ended to prompt discussion and obtain rich and nuanced information on implementation experiences in specific contexts.
During site visits to local areas, the evaluation team may request to speak to representatives of small businesses, small not-for-profit organizations, or small government jurisdictions who are involved in WIOA implementation. It is estimated that 1 percent of the site visit respondents may represent small entities, which translates to about 3 respondents annually. The evaluation team will tailor protocols to each respondent to ensure that respondents are only asked a minimum number of relevant questions, and site visitors will not ask questions that have already been well-addressed by previous respondents. Only the minimal amount of data needed for this study will be collected. The average interview length with local respondents will be about 70 minutes, and site visitors will be flexible to rearrange their schedule and interview time frames to accommodate respondents’ other obligations. Local-level respondents will only be asked for information once; they will not participate in the state-level survey.
The semi-structured interviews will only be conducted once with each respondent. Although the Federal investment of resources in WIOA implementation requires the systematic collection of comprehensive implementation data, the semi-structured interview questions have been designed with a focus on prioritizing topics of primary interest to the evaluation and federal stakeholders, in order to minimize the burden on respondents. If these data are not collected, DOL and other stakeholders will not have the information necessary to understand how WIOA legislation is being implemented across the country and what additional TA, guidance, or policy changes will help states better administer WIOA’s provisions and provide guidance and oversight at the local level.
There are no special circumstances for the proposed data collection.
The 60-day notice to solicit public comments was published in the Federal Register (82 FR 56845, November 30, 2017). A copy of this 60-day notice is included as supplemental information with this information collection request. DOL received one public comment. A summary of the comment and response are described below.
A private citizen, Jean Public, questioned the value of conducting site visits to collect data and suggested that a survey would be more efficient.
Response: We agree that a survey of states is a very useful way of gathering information; we plan to conduct a survey with all 50 states and the District of Columbia for this evaluation. This survey is described in a separate Federal register notice.
Given the variation in states’ and local areas’ experiences in implementing such far-reaching and complex legislation as WIOA, it is important to complement the quantitative data from the survey with rich and nuanced qualitative data from site visits so that DOL can understand the full picture of WIOA implementation. Site visits are limited to just 14 states – enough to capture a range of experiences and provide meaningful data. The localities within each state will be selected in part to maximize the efficiency of the site visits.
Consultations on the research design, sample design, and data needs were part of the evaluation design phase of the WIOA Implementation Evaluation. The evaluation team convened a technical working group (TWG) for this purpose, and members of the TWG are listed below. One of the TWG members is a state workforce agency staff member and represents the respondents from whom information will be collected on the survey. The objectives of these consultations were to ensure the technical soundness of the evaluation, to verify the importance, relevance, and accessibility of the information sought in the evaluation, and confirm the availability of data and the appropriateness of the frequency of collection.
Technical Working Group (TWG) members:
Yvette Chocolaad, Policy Director, National Association of State Workforce Agencies
Cynthia Forland, Assistant Commissioner, Workforce Information and Technology Services, Washington State Employment Security Department
Allison Metz, Senior Research Scientist and Director of the National Implementation Research Network (NIRN), University of North Carolina at Chapel Hill
Ron Painter, Chief Executive Officer, National Association of State Workforce Boards
Carl Van Horn, Distinguished Professor of Public Policy and Director, John J. Heldrich Center for Workforce Development at Rutgers University
Respondents will not receive payments.
The WIOA Implementation Evaluation will not collect or report any sensitive personally-identified information (PII). Nonetheless, the evaluation team will adhere to a set of strict policies to ensure that information collected will be kept private to the extent permitted by law. All interview respondents will be notified at the beginning of each interview as is stated in the protocols that the information that they provide will never be linked to their names, that their names will never be shared in any evaluation report, and that their participation is voluntary. Interviewers will read a statement to assure respondents of privacy and to ask for their verbal consent to participate in the interview. Site visit interviews will be conducted in private areas, such as offices or conference rooms.
Interview transcripts and resultant reports from qualitative coding of the data will not identify respondents by name, and the evaluation team will carefully safeguard evaluation data. Interview notes will not be shared by the evaluation team with DOL or anyone outside of the project team, except as otherwise required by law. All evaluation team site visitors and interviewers have received training in privacy and data security procedures.
The WIOA Implementation Evaluation will not ask respondents to answer questions of a sensitive nature.
Table A.1 provides the annualized respondent hour and cost burden estimates for the semi-structured interviews for which this package requests clearance. The evaluation is requesting clearance for a period of three years. Burden estimates are based on the evaluation team’s experience conducting similar data collections. The burden calculations below include time for pre-visit scheduling calls with the main contacts in each state and local area.
Table A.1. Estimated Annualized Respondent Hour and Cost Burden
Type of Instrument |
Number of Respondents |
Number of Responses Per Respondent |
Total Number of Responses |
Average Burden Per Response (in hours) |
Estimated Burden Hours |
Average Hourly Wagea |
Annual Burden Costs |
State-level staff interview protocol |
|||||||
Workforce boardb |
14 |
1 |
14 |
70/60 |
16 |
$34.07 |
$545 |
Title I A/DW programc |
23 |
1 |
23 |
78/60 |
30 |
$34.07 |
$1,022 |
Title I Youth programd |
5 |
1 |
5 |
1 |
5 |
$34.07 |
$170 |
Titles II & IVe |
28 |
1 |
28 |
80/60 |
37 |
$34.07 |
$1,261 |
Title IIIf |
14 |
1 |
14 |
80/60 |
19 |
$34.07 |
$647 |
Unemployment Insuranceg |
5 |
1 |
5 |
90/60 |
8 |
$34.07 |
$273 |
Other state partnerh |
5 |
1 |
5 |
90/60 |
8 |
$34.07 |
$273 |
Local-level staff interview protocol |
|||||||
Workforce boardi |
37 |
1 |
37 |
70/60 |
43 |
$34.07 |
$1,465 |
Title I A/DW programj |
19 |
1 |
19 |
1 |
19 |
$34.07 |
$647 |
Title I Youth programk |
19 |
1 |
19 |
1 |
19 |
$34.07 |
$647 |
Titles II & IVl |
37 |
1 |
37 |
90/60 |
56 |
$34.07 |
$1,908 |
Title IIIm |
19 |
1 |
19 |
1 |
19 |
$34.07 |
$647 |
Other local partnern |
19 |
1 |
19 |
1 |
19 |
$34.07 |
$647 |
American Job Center Operatoro |
9 |
1 |
9 |
75/60 |
11 |
$34.07 |
$375 |
American Job Center Managerp |
9 |
1 |
9 |
75/60 |
11 |
$34.07 |
$375 |
Total |
262 |
- |
262 |
-- |
320 |
-- |
$10,902 |
a Hourly wage reflects the May 2016 mean hourly wage estimate for “social and community service managers”, as reported by the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates, 2016, Table 1 “National employment and wage data from the Occupational Employment Statistics survey by occupation, May 2016 (accessed from the following web site as of July 11, 2018: https://www.bls.gov/news.release/archives/ocwage_03312017.pdf.
b Workforce board. The evaluation team will interview approximately three state workforce board staff in each state.
c Title I A/DW program. The evaluation team will interview five state Title I A/DW staff in each state.
d Title I Youth program. The evaluation team will interview one state Title I Youth staff in each state.
e Titles II and IV. The evaluation team will interview six state Titles II and IV staff in each state.
f Title III. The evaluation team will interview three state Title III staff in each state.
g UI. The evaluation team will interview one state UI staff in each state.
h Other state partner. The evaluation team will interview one other state partner staff in each state.
i Workforce board. The evaluation team will interview approximately four local workforce board staff in each of the two local areas in each state.
j Title I A/DW program. The evaluation team will interview two local Title I A/DW staff in each of the two local areas in each state.
k Title I Youth program. The evaluation team will interview two local Title I Youth staff in each of the two local areas in each state.
l Titles II and IV. The evaluation team will interview four local Titles II and IV staff in each of the two local areas in each state.
m Title III. The evaluation team will interview two local Title III staff in each of the two local areas in each state.
n Other local partner. The evaluation team will interview two other local partner staff in each of the two local areas in each state.
o AJC operator. The evaluation team will interview one local AJC operator staff in each of the two local areas in each state.
p AJC manager. The evaluation team will interview one local AJC manager staff in each of the two local areas in each state.
There are no additional costs to respondents other than their time.
The total annualized cost to the federal government is $674,498. Costs result from the following two categories:
The annualized cost to the federal government for the evaluation contractor, Mathematica Policy Research (Contract Number: DOLQ129633249/DOL-OPS-16-U-00191), to carry out this evaluation is $662,418. The total cost of the implementation evaluation is $1,987,253 for three years. Therefore the annualized cost is $1,987,253 / 3 years = $662,418.
The annualized cost for federal technical staff to oversee the evaluation is $12,080. This is calculated by the following: an annual level of effort of 200 hours for one Washington, D.C.-based Federal GS-14 step 4 employee earning $60.40 per hour. (See Office of Personnel Management 2017 Hourly Salary Table at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2017/DCB_h.pdf.) Therefore the annualized cost is 200 hours X $60.40 = $12,080.
The total annualized cost to the federal government is $662,418 + $12,080 = $674,498.
This is a new data collection effort (OMB No. 1290-XXXX).
The analysis of data collected from the site visits will not require statistical methodology or estimation. The qualitative data collected will be analyzed using qualitative coding software such as NVivo or ATLAS.ti, and some descriptive characteristics of the states and local areas visited may be tabulated to show the variation across sites. The implementation evaluation includes site visits to 14 purposively-selected states and approximately 28 purposively-selected localities. Although the results will not be generalizable to the entire country, the results will reflect a broad range of state experiences with implementation. When the qualitative data from the site visits are synthesized with the quantitative data from the survey of 50 states and the District of Columbia (included in a separate clearance package), the evaluation team will be able to provide a national picture of WIOA implementation.
The evaluation team will develop several products to share the findings from the evaluation with key stakeholders at DOL (internal) as well as the workforce development community (external). Products that will include data collected from the semi-structured interviews include a final report, an issue brief, and a special topic paper, which will be made available on the DOL website. An estimated project schedule is provided in Table A.2, with estimated dates of completion and publication of reports.
Table A.2 WIOA Implementation Evaluation project schedule
Action |
Estimated Date of Completion |
Estimated Date of Publication |
Semi-structured interviews with state- and local-level staff |
November 2018 – March 2019 |
|
Issue brief |
June 2019 |
October 2019 |
Special topic paper |
July 2019 |
November 2019 |
Final report |
August 2019 |
January 2020 |
The OMB approval number and expiration date will be displayed or cited on all forms completed as part of the data collection.
No exceptions are necessary for this information collection.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |