Key Informant Interview Protocol-- State Administrator

OMB Control No.: 1290-XXXX

Expiration Date: XX/XX/2021

Analysis of Employer Performance Measurement Approaches

State WIOA Administrator Interview Guide [DRAFT]

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Chiefevaluationoffice@DOL.gov and reference the OMB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of Labor, Chief Evaluation Office, 200 Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview guide to the email or mailing address.

Introduction/Purpose of the Study

Thank you for agreeing to participate in this interview today.

My name is _______________, and I’m a researcher from [The Urban Institute, a non- profit research organization located in Washington, DC; George Washington University, located in Washington, DC; Capital Research Corporation, located in Arlington, VA]. With me today is [name and affiliation]. We’re here today because the Urban Institute and its partners are under contract with the U.S. Department of Labor (DOL), Chief Evaluation Office (CEO) to conduct an analysis of new performance measurement approaches for measuring the effectiveness of employer services in the public workforce system. As part of the study, we are conducting site visit interviews with state and local WIOA administrators and other workforce development staff in several different states. The information collected during these interviews will inform our reporting to DOL on the use, validity, reliability, and practicality of measuring the effectiveness of services provided to employers in the public workforce system.

We are interested in learning about your experiences related to providing services to employers in your state and measurement approaches, including the National Pilot Measures and other measures of employer services. This interview should last about 60 minutes.

Privacy Statement

Before we begin the interview, we want to let you know that, although we will take notes during this interview, when we write our reports and discuss our findings, information from all the people we speak with will be compiled and summarized without identifying individual respondents. We also want to make it clear that your participation is strictly voluntary, and you may choose to not answer questions at any time.

Finally, to help us accurately capture the information you share, we would like to record this interview. The recording is just a back-up for our notes and will be kept within our small research team. The interview recording will be deleted once we have developed a full set of notes from the interview. Is it okay with you if we record the interview? [GET CONSENT] Do you have any questions before we begin?

[BEGIN RECORDING IF CONSENT GRANTED]

1. BACKGROUND INFORMATION

1. Obtain the following information for each respondent [Fill in as much information in advance of the interview as possible (from the survey, document review, internet search, etc.), as applicable. Confirm the information we have and ask for the information we don’t have. Request a business card from each respondent.]

   1. First and last name

   2. Official job title

   3. Name of department/agency

   4. Number of years with the agency _____; Number of years in position _____

   5. Our understanding is that you are responsible for overseeing [Indicate the WIOA programs we think are correct]. Is this correct?

   6. [If respondent oversees multiple programs] Do the services provided to employers and the approach to data collection and measurement of employer services vary across programs? _______

   7. Please describe in a few sentences the scope of your job responsibilities related to these WIOA Titles [Indicate WIOA programs/Titles] and any other programs you may be responsible for.

In this interview, we will be focused primarily on understanding the goals of the services provided to employers for [indicate the WIOA Title(s) given the interviewees scope of responsibility as answered above]. We will be asking questions about the decisions your state has made about the DOL national performance measures, including the use of any alternative measures, and the extent to which services provided under the Titles you administer contribute to the services counted for those measures.

[NOTE: If the state survey responses are not available to the study team in advance of the site visit, the interviewer will probe for further detail about measurement approaches in Sections C and D].

If the survey responses are available to the study team in advance of the interview:

You or someone from your Department has already answered questions from a survey sent out through this same study about these topics, but in this interview we want to follow up to understand the survey answers in more depth. [Note to interviewer: Hand to the administrator the fact sheet with the National Pilot Measures and definitions, and the list of alternative measures we understand them to be using based on their survey responses and/or reports to DOL.] Here is information for your reference during the interview on the National Pilot Measures and the alternative measures that we understand your state has selected. We will be referencing these measures periodically during the interview.

2. EMPLOYER SERVICES STRATEGIES AND GOALS [If respondent oversees multiple programs and services vary across programs, ask for each program administered.]

2. What employer services does your state provide under the [program name] program?

3. What do you see are the goals of those services?

4. How are those services implemented? What agencies and staff provide those services?

5. Does your agency work with other agencies/entities to provide those services?

6. Are you involved with any cross-state initiatives to provide services to employers?

7. Have you made any changes to the way your program serves employers as part of your implementation of WIOA?

8. What are some of the challenges of providing services to employers? What supports are needed to address these challenges?

9. What strategies have been used to effectively provide services to employers?

3. NATIONAL PILOT MEASURES [If respondent oversees multiple programs and their measurement approach varies across programs, ask for each program administered.]

Now we are going to turn to the DOL National Pilot Measures for a moment. [Note to Interviewer: Refer them back to the sheet you handed them at the start of the interview and reference your summary information about their state’s selection and implementation of measures.]

10. Which National Pilot Measures has your state implemented? [Fill in as much information in advance of the interview as possible (from the survey, document review, internet search, etc.), as applicable. Confirm the information we have and ask for the information we don’t have. Probe for if they have used the same measure(s) or something similar previously.]

11. Was your department/program involved in selecting the DOL National Pilot Measures?

    1. [If they were not involved] How/when did you learn that the state had selected the national measures? Probe on how they have been involved in implementation.

    2. [If they were involved in the decision-making] Why did your state choose those measures for the national pilot? Why was the other measure less appealing?

12. Have you been involved in implementing the new National Pilot Measures? Probe for involvement in tracking and collecting data on service delivery and reporting responsibility.

    1. [If yes] In what ways have you been involved? [Probe on developing guidance, deciding on how to calculate, determining how to collect the data, which entities should pilot, etc.]

    2. [If no] What insights or perspectives do you have on the new National Pilot Measures? [Probe on if they have heard how it is going, and their involvement in collecting and reporting any data on the measures. SKIP to Question 22]

13. When did [the programs under your scope of responsibility] begin collecting data for the pilot measures? Has your team had any difficulties collecting the necessary data for the measures? What kinds of challenges has your team encountered either within the services you administer or in collecting and combining data across the services? [Fill in the information for this question as much as possible from the survey, and use the interview question to confirm and fill in any gaps]

14. Has your team had any difficulties collecting the necessary data for the measures? What kinds of challenges has your team encountered either with tracking and reporting on the services you administer or in combining data across services delivered/programs administered? [Fill in the information for this question as much as possible from the survey, and use the interview question to confirm and fill in any gaps]

15. Do the measures seem like a fair representation of the amount and quality of the services your team provides to employers? Why or why not?

    1. Have you calculated any outcomes yet? What do they indicate?

    2. Do you think the measures appropriately reflect the goals you stated around employer measures (Question 3)?

16. Have the pilot measures contributed to your agency’s or the overall state’s strategic planning or policies in any way? How about particular to your program area(s)?

    1. If so, please explain.

    2. If not, do you anticipate they will in the future? What are the barriers or challenges to the measures contributing to your state’s strategic planning or policies?

4. ALTERNATIVE MEASURES [If respondent oversees multiple programs and their measurement approach varies across programs, ask for each program administered.]

[To the interviewer: In advance of the interview, you will determine if the state is using any alternative measures. If not, then skip to Question 22. You will also review the survey to see how involved this particular administrative unit likely has been in the selection and use of the measures.]

Now we are going to turn to the alternative measures your state has indicated it is collecting to measure the effectiveness of its services to employers.

17. According to the information we have, we understand that the state is using the following alternative measures: [fill in measures]

18. Were you involved in the selection of those measures?

    1. [If they were not involved] How/when did you learn that the state had selected the alternative measure(s)? Probe on how they have been involved in implementation.

    2. [If they were involved in the decision-making] Please describe the process for selecting the alternative measure(s). What were the criteria, if any, for deciding to implement the alternative measure(s)?

19. If the state is using any alternative measures:

    1. Do you know if the measure(s) exist before WIOA or the national pilot? [If yes, they did] When did your team begin collecting data for those measures?

    2. What do these measures capture that the National Pilot Measures do not?

    3. What is the extent to which these measures better reflect the goals and strategies of the programs you oversee?

20. Do the alternative measure(s) seem like a fair representation of the amount and quality of the services your team provides to employers? Why or why not?

    1. Have you calculated any outcomes yet? What do they indicate?

21. Have the alternative measure(s) contributed to your agency’s or the overall state’s workforce strategic planning or policies in any way? If so, please explain. If not, do you anticipate they will in the future?

5. PAST MEASURES AND ANY ADDITIONAL THOUGHTS ABOUT MEASURING EMPLOYER SERVICES

22. Prior to implementation of the National Pilot Measures and the alternative measures we just discussed, did your program(s) collect data on the amount, type, or quality of services provided to employers in your state that we have not discussed? If so, please describe. [Fill in as much information in advance of the interview as possible (from the survey, document review, internet search, etc.), as applicable. Confirm the information we have and ask for the information we don’t have].

For each measure described, probe for the following:

1. When and for how long was the measure used?

   1. [For Title I and III administrators only] Have you ever used the American Customer Satisfaction Survey (ACSI) to measure the quality of services provided to employers? If so, for which programs?

2. Why was the measure implemented?

3. What data was collected to inform the measure?

4. What were the difficulties related to using the measure?

5. What were the benefits of using the measure?

6. Why was the measure discontinued?

   1. If respondent has used the ACSI at any point to measure the effectiveness of employer services, probe for any other impressions of the survey and reasons the state stopped using it, including whether the state received a waiver from the ACSI requirement under WIA.

23. From your perspective, what are the most important things to try to capture when measuring business’s interactions with the services your program(s) administer? What are the most important things to try to capture when measuring business’s interactions with the State public workforce system as a whole? If those things aren’t currently being captured, do you have any ideas for how to best measure them?

24. Have the measures helped you consider potential changes to your program(s)’ strategy for serving employers? Why or why not?

25. Do you feel that measuring services to employers will positively impact the services you administer through the program(s) you oversee? Why or why not? Do you feel that measuring services to employers is beneficial to the State’s public workforce system as a whole? Why or why not?

Closing

Is there anything you would like to mention that you haven’t had a chance to say before we end the interview?

Thank you so much for your participation. Please feel free to reach out to us should you have any questions or comments in the future.

4

Analysis of Employer Performance Measurement Approaches

State WIOA Administrator Interview Guide