WORKFORCE INNOVATION AND OPPORTUNITY ACT (WIOA) IMPLEMENTATION EVALUATION,
OMB NO. 1290-XXXX
AUGUST 2018
In this document, the Department of Labor (DOL) requests clearance from the Office of Management and Budget (OMB) under the Paperwork Reduction Act (PRA) for a new collection: semi-structured interviews with state- and local-level workforce, adult education, and vocational rehabilitation staff in 14 states and up to 28 local areas, associated with the Workforce Innovation and Opportunity Act (WIOA) Implementation Evaluation.
The WIOA implementation evaluation was funded by the DOL Chief Evaluation Office (CEO) in partnership with the Employment and Training Administration (ETA) with the goal of better understanding the implementation of WIOA, the variations in implementation across states and localities, and the need for further administrative guidance, regulations, TA, and policy. DOL contracted with Mathematica Policy Research in partnership with Social Policy Research to conduct the evaluation.
The universe of potential sites for this evaluation includes all 50 states and DC and all of the local workforce development areas within each state. Four states and eight localities (two in each state) participated in pilot site visits in fall 2017 to test state-level staff interview and local-level staff interview protocols. Fourteen states and up to two localities in each of those states will be included in the implementation study site visits during 2018. These 14 states and up to 28 localities are the subject of this data collection request. This section outlines the respondent universe, proposed methods for respondent selection, and the expected response rates for the respondents participating in the proposed research.
States and localities for pilot visits. No statistical methods were used in selecting pilot visit sites and interviewees. The four states (California, Massachusetts, Mississippi, and Ohio) and two localities within each state were purposively selected to test state-level staff interview and local-level staff interview protocols. Regional DOL administrators recommended a subset of states based on the extent of their progress with WIOA implementation. The study team conducted calls with the recommended states and then selected four states for pilot visits that would allow the team to test protocols in a range of state and local structures. Each pilot visit state recommended localities that had made progress with WIOA implementation and that were logistically feasible for the study team to visit within the pilot site visit timeframe.
States and localities for site visits. No statistical methods will be used in selecting states and local areas for the 2018 implementation site visits. States and localities will be purposively selected for site visits based on information from publicly available documents. The study team will select:
two to three states within each of DOL’s six regions to ensure geographic representation;
a mix of urban and rural areas among the site visit states;
states with a variety of administrative structures across the core program agencies; and
at least some states will be states with a single workforce area.
Localities within each state will be selected to ensure that:
site visitors visit a mix of urban and rural areas; and
the sites are logistically feasible to access.
State- and local-level WIOA administrators and staff. No statistical methods will be used in selecting staff interviewees for the 2018 implementation site visits. Across the 14 states and up to 28 localities, the study team will interview the administrators and staff whose positions are best in line with the information the team is seeking. The study team estimates that the number of staff to be interviewed totals 784, including administrators of Titles I through IV programs, workforce board staff, partners at both the state and local levels, and AJC operators and managers. As demonstrated in Table B.1 and as further described in section B.3, the study team anticipates a 100 percent response rate for the interviews among the individuals the team identifies as respondents. The study team anticipates that this will typically include representatives from each of the following groups of staff at both the state- and local-levels: (1) workforce board staff, (2) Title I Adult and Dislocated Worker (A/DW) program staff, (3) Title I Youth program staff, (4) Titles II and IV staff, (5) Title III staff, (6) state UI staff (state-level only), (7) partner staff (such as Temporary Assistance for Needy Families and Unemployment Insurance) , (8) local One Stop Operator staff (local-level only), and (9) local American Job Center (AJC) managers (local-level only).
Table B.1 summarizes this information, including the sampling method, estimated sample sizes, and assumptions about response rates for each data collection activity and respondent type by site.
Table B.1. Sampling and response rate assumptions, by data collection activity and respondent type
Respondent type |
Sampling methoda |
Estimated universe |
Sample |
Estimated response rate (percent) |
Estimated responses (per state) |
Estimated responses (across all states) |
Site visit states |
Purposive |
14 |
-- |
100 |
-- |
-- |
Site visit localities |
Purposive |
28 |
2 |
100 |
-- |
-- |
State-level staff |
Purposive |
280 |
20 |
100 |
20 |
280 |
Local-level staff |
Purposive |
504 |
36 |
100 |
36 |
504 |
To understand the progress of WIOA implementation, the study team will conduct site visits to states and local areas. Site visitors will spend up to four days in each state, including two days interviewing state-level respondents, and up to two days interviewing local-level respondents in up to two localities (up to one day per locality). The data collection activities covered by this clearance include semi-structured interview data, described next.
During each visit, two members of the study team will conduct semi-structured, in-depth interviews with the staff responsible for WIOA implementation at the state- and local-level (see B.1.3. above.) The data collected using the semi-structured interview protocols will focus on how policies under WIOA are being implemented, and to identify areas where changes or supplemental TA or policy are needed to better support states in implementing WIOA. The interview protocols for state-level respondents and local-level respondents are included in this package. For each visit, protocols will be tailored according to the respondents present in that state or local area and their roles. Site visitors will obtain consent from each respondent before beginning each interview.
The data collected from interview respondents will be qualitative information about staff’s experiences and insights given their role in implementing WIOA. The study team will interview all staff in the roles listed above, and will not need to use sampling methods to create a sample of respondents. Similarly, no statistical methods of estimation will be needed in the analysis of the interview data. The qualitative data collected will be analyzed using a qualitative data analysis software such as NVivo or ATLAS.ti. This coding will facilitate identification of key themes across sites and enable the study team to describe WIOA implementation. Due to the purposive selection of site visit states and localities, the findings from site visit data will not be generalizable to all states and localities.
The study will not use a probability sample for either the selection of states and local areas or the selection of respondents in the states and local areas.
States vary in the types of challenges they face implementing WIOA because of their diverse administrative structures and the different approaches states have taken to implement WIOA. Because the study team will not visit all states and territories implementing WIOA, it is important that the study capture information across the range of structures states use to deliver workforce services and the range of economic environments they operate in. For this reason, the study will purposively select states to vary across different administrative structures, different regions of the country, and different urban and rural geographies. The information needed to purposively select sites comes from publicly available documents. The study team will not visit all local areas in a state but will select with the help of state administrators up to two local areas that are perceived to be generally representative of the types of administrative structures across local areas in the state with a mix of urban and rural geographies. The study will also prioritize local areas that are reasonably close to the state office location in order to conduct the visit in an efficient manner.
The study will identify all administrators and staff whose positions are best in line with the information the team is seeking and include them for interviews in selected states and localities. Without talking to all the key staff, the study team might miss important information regarding the implementation of WIOA.
The study team will collect semi-structured interview data from staff in each state and locality only once. The study team’s experience during the pilot site visits informed site visit schedule and protocol modifications to ensure that data are collected in the least burdensome manner during the upcoming state and local site visits.
To encourage participation by states and localities, either Federal or regional DOL staff will contact the selected states informing them of the upcoming visits and request their participation. As the study team schedules site visits, they will explain the nature of the visits and share a suggested schedule, so that staff know what the team expects of them when they agree to participate. Participating states will in turn suggest potential local areas for site visits based on the study team’s parameters. Site visitors will work with state and local site staff to ensure that the timing of the visit is convenient for all staff involved. Site visitors will also work with state staff to ensure that selected local areas are aware of the purpose of the study and available to participate. State staff will be included on any communication with the local areas to encourage participation.
To ensure full cooperation from WIOA core and partner program staff, the study team will be flexible in scheduling interviews and activities to accommodate the particular needs of respondents. Furthermore, data collectors will meet with in-person interview respondents in a central location that is well-known and accessible, such as the state or local workforce board office or local AJC.
Although the study team will try to arrange interviews that accommodate respondents’ scheduling needs, there might be instances when a respondent is unable to meet while the team is on site; when this happens, a member of the study team will request to meet with the respondent’s designee or schedule a follow-up call at a more convenient time. With these approaches, the study team anticipates a 100 percent response rate for staff interviews, as has been achieved on similar qualitative data collection efforts, such as those for the Workforce Investment Act Adult and Dislocated Worker Programs Gold Standard Evaluation, the Evaluation of the Linking to Employment Activities Pre-Release grants, the Evaluation of the Summer Youth Employment Initiative, and the Impact Evaluation of the Trade Adjustment Assistance Program.
Methods to ensure data reliability. We will use several well-proven strategies to ensure the reliability of the interview data collected during the site visits. First, site visitors, all of whom already have extensive experience with this data collection method in a workforce setting, will be thoroughly trained in this study, its research questions, and goals. They will also be trained on how to probe for additional details to help interpret responses to interview questions. Second, this training and the use of the protocols (included in package) will ensure that the data are collected in a standardized way across sites. Finally, all interview respondents will be assured that their responses will remain private; reports will never identify respondents by name, and any quotes will be devoid of identifying information, including site name.
All procedures and protocols to be used in the WIOA Implementation Study have been reviewed by content and methodological experts to ensure clarity and optimal ordering of the questions.
To ensure that the WIOA Implementation Study data collection protocol could be used effectively as a field guide to yield comprehensive and comparable data across implementation study sites, senior research team members conducted pilot site visits in fall 2017. During the pilot visits the site visitors also assessed the site visit schedule to determine how best to structure data collection activities given the amount of data to be collected and the amount of time allotted for each data collection activity. Adjustments were made to the interview protocols to ensure that the information necessary to answer the research questions would be collected within the available time. In advance of the 2018 implementation site visits, the study team will train all site visitors on the data collection instruments to ensure a common understanding of the key objectives and concepts as well as fidelity to the protocols. The training session will cover topics such as the study purposes and research questions, data collection protocols, procedures for scheduling visits and conducting on-site activities (including a review of interview facilitation techniques and procedures for protecting the privacy of respondents), and post-visit files and summaries.
B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data
Consultations on the methods used in this study were part of the study design phase to ensure technical soundness. As stated previously, no statistical methods will be employed. Consultations included stakeholders within the U.S. Department of Labor and a technical working group (TWG). Members of the study team and the TWG are listed in Table B.2.
Table B.2. Individuals who were consulted for the WIOA Implementation Evaluation
Mathematica Policy Research
|
Grace Roemer Project Director
Pamela Holcomb Co-Principal Investigator
Samina Sattar Deputy Project Director
|
Social Policy Research Associates
|
Kate Dunham Co-Principal Investigator
|
TWG members |
Yvette Chocolaad, Policy Director National Association of State Workforce Agencies
Cynthia Forland, Assistant Commissioner, Workforce Information and Technology Services Washington State Employment Security Department
Allison Metz, Senior Research Scientist and Director of the National Implementation Research Network (NIRN) University of North Carolina at Chapel Hill
Ron Painter, Chief Executive Officer National Association of State Workforce Boards
Carl Van Horn, Distinguished Professor of Public Policy and Director John J. Heldrich Center for Workforce Development at Rutgers University
|
Staff responsible for overseeing the collection and analysis of data are listed in Table B.3.
Table B.3. Individuals who will oversee the collection and analysis of data for the WIOA Implementation Evaluation
Mathematica
Policy Research |
Grace Roemer Project Director
Pamela Holcomb Co-Principal Investigator
Samina Sattar Deputy Project Director
|
Social Policy Research Associates
|
Kate Dunham Co-Principal Investigator
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |