Att 3.3 Local Title I Youth Programs

OMB No. 1290-XXXX
Exp. Date XX/XX/2021

3.3 PROTOCOL FOR LOCAL TITLE I YOUTH STAFF
Instructions to site visitors: Bullets below each question represent probes for important details;
depending on how well you are doing on time, try to make sure respondents touch on each of
these issues. Text in brackets [ ] should be tailored based on the state or local area. Sections in
gray are the high priority sections for this respondent.

Introduction
Thank you for taking the time to meet with me today. My name is [NAME] and I work for
[Mathematica Policy Research/Social Policy Research Associates]. I am part of an independent
research team conducting a study of WIOA implementation on behalf of the U.S. Department of
Labor. Thank you very much for agreeing to participate in this conversation.
The purpose of this three-year study is to help DOL understand how states are implementing
changes to the core workforce programs authorized under Titles I and III of WIOA, as well as
how those changes are influenced by integration with other partners such as adult education,
vocational rehabilitation, and TANF. To help us better understand WIOA’s implementation, we
are conducting site visits to 14 states, including two local areas in each state, to learn about their
experiences. The information you share will help us understand the range of experiences that
states have had, including key successes and challenges.
We have about [duration] for our conversation. I want to let you know that all interview data will
be reported in the aggregate and your name will never be mentioned in any report that we write,
though we might use quotes from your interview to illustrate findings, without using your name.
I would also like to record our conversation so that I can listen to it later as I complete my notes,
and I will not share the recording with anyone outside of the research team. Being part of this
discussion is voluntary, and you may choose not to answer a question if you wish, or to pause the
recording at any time. Do you have any objections to being part of this interview or to my
recording our discussion?
Okay, I’m going to turn on the audio recorder now.

Respondent Background
1.

[If not already known for each person participating in the interview]
a. What is your official title and role?
b. What are your main responsibilities related to implementation of WIOA?

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection
displays a valid OMB control number. Public reporting burden for this collection of information is estimated to average 1 hour per response,
including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and
reviewing the collection of information. The obligation to respond to this collection is voluntary. Send comments regarding the burden
estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Chiefevaluationoffice@DOL.gov
and reference the 0MB Control Number 1290-XXXX. Comments can also be mailed to: U.S. Department of Labor, Chief Evaluation Office, 200
Constitution Ave., NW, S-2312, Washington, DC 20210. Note: Please do not return the completed interview guide to the email or mailing
address.

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

A. Governance and Planning
1. Local Plan Development
1.

What input did you have in the state plan, and how did the state plan influence your
local plan?

2.

Please describe the general planning process your local board undertook to produce
your WIOA local plan.
a. Who led the process?
b. Who else was involved?

3.

Describe your general planning process for developing your local plan. What entity
and staff members led the process?
a. How and to what extent were other entities involved?

4.

Did this process differ from previous local plan development efforts?
a. Was the process longer than with previous plans?
b. Were certain partners more involved? Less involved?
c. How did any changes to the process from prior planning efforts improve the final
plan?
d. What were you able to achieve that had not been possible before? What did you
lose, if anything?

5.

Do you have any lessons to share on developing your local plan around what worked
well or has been challenging?

6.

What aspects of your state’s guidance on the local plan were helpful?
a. What could have been more helpful?

7.

What additional guidance is needed from the state or USDOL for you to be able to
carry out the required revision to your local plan?

8.

Did the state provide any TA resources or activities related to developing your local
plan that your area participated in?
a. How useful was it, and what additional TA on the topic could the state provide
that would be helpful for revising your local plan?

2. Regional Designations and Regional Plans
1.

How satisfied are you with your state-designated region under WIOA?
a. Do you think it accurately reflects your regional labor market?

2.

Has being part of a region helped you to better engage with regional employers? With
regional economic development entities?

3.

How did the board develop the regional plan?
a. What role did the local board members play?

3.30

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

b. Who else was involved? Was there anyone who wasn’t involved that you think
should have been at the table?
4.

What challenges did you experience in developing your regional plans? What worked
well?
a. Is there anything you plan to change related to the regional planning process for
the next time?

3. Local Workforce Development Board Role
1.

If the board has a Youth committee, did the board choose to keep its previous Youth
Council?

2.

Have the new membership requirements under WIOA streamlined board operations?

3.

How, if at all, have the role and approaches of the board changed under WIOA?
a. Is it more strategic, more innovative, more focused on continuous improvement,
more focused on improving service integration?
b. Why has the role of the board changed?
i. New board mandates under WIOA?
ii. Required changes in membership?
iii. Some other reason?

a) Sector Strategies
1.

What policies and strategies has your local board adopted to increase the use of sector
strategies in the workforce system? Involving new partners, expanding services and
training options, targeting new populations, etc.?

2.

Has the focus on developing and expanding sector strategies been at the state, local,
or regional levels?

3.

What funding sources are being used to develop and support your sector strategy
programs and initiatives (e.g., local formula funds, WIOA state set-aside funds, Title
II or IV funding)?

B. AJC System
1. AJC Certification
1.

Were you at all involved with the AJC certification process? If so:
a. What challenges did you face and how did you address them?
b. What worked well in the certification process?
c. How useful do you think the certification process is in ensuring AJC
effectiveness, accessibility, and continuous improvement? How could it be
improved?

3.31

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

2. Infrastructure Funding and MOUs
1.

How does your MOU (excluding resource sharing components) differ from the last
one you had under WIA?

2.

What was included in your budget for One-Stop infrastructure and other costs?
a. Why are you contributing for those components?
b. How much was cash versus in-kind? Why?

3.

What was the process for working with the LWDB determining proportionate use and
relative benefit for your program?
a. How well did that process work?
b. How could it have been improved?

4.

How has the WIOA RSA requirement affected your partnership (either positively or
negatively)?

5.

How often is the LWDB reviewing your agreement (for issues like value of non-cash
contributions, proportionate use and relative benefit)?
a. Are there any challenges related to this review process?

6.

What challenges did you face in the process of developing your MOU and budget?

7.

Has your state agency taken any steps to facilitate negotiation of One-Stop operating
budgets (e.g. meetings at the state level, with state partners, to facilitate their
participation)?

8.

Did your area achieve consensus on your IFA?
a. [If so]: Why do you think your area succeeded?
b. [If not]: Which required partner(s) did not agree? What were the sticking points?
Where are you in the process of reaching agreement?

9.

How helpful was the guidance provided by your state and USDOL on resource
sharing and MOUs? What could have been more helpful

10.

What TA have you participated in on MOUs and resource sharing?
a. How helpful was this TA?
b. What could have been more helpful?

11.

What additional guidance or technical assistance do you need from your state or
USDOL on MOUs and resource sharing?

3. One-Stop Operator
1.

Operators have different responsibilities in different areas; for your local area, what is
the operator’s role as it relates to:
a. Partner meetings?

3.32

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

b. Cross-training sessions?
c. Coordinating AJC operations & policies?
d. Managing AJC facilities?
e. Implementation of MOUs/RSAs?
2.

Does the operator have any other significant roles in your local area?

3.

How and why, if at all, has the operator’s role changed with the implementation of
WIOA?

4.

What is the role of the AJC manager?

5.

How and why has the AJC manager’s role changed, if at all, with the implementation
of WIOA?

6.

How, if at all, do you think these changes in what entity serves as the operator will
affect AJC operations?

4. Collocation at AJCs
1.

Across all the AJCs in the local area, how often is your program collocated at
comprehensive centers? At affiliate sites?
a. What do you think are the primary reasons for this?

2.

[If youth program is not collocated at comprehensive AJC(s)]: Are you providing
access to services via training an onsite staff member or via a “direct linkage”?

3.

Why did you opt for that approach and how is it being operationalized?
a. What challenges and benefits have you experienced with each approach?

4.

Has your collocation at AJCs changed since WIOA implementation? If so, how and
why?

5.

What guidance have you received from the state on collocation of partners at AJCs
under WIOA?
a. Did that guidance change from what was provided under WIA? If so how?
b. Is any additional guidance needed?

B. Overall AJC Streamlining & Accessible Services
1. Accessibility
1.

How integrated and streamlined is service delivery in the local area?

2.

Please describe any new state or local policies and actions that promote a more
seamless, accessible and customer-focused workforce service delivery network?

3.

Has the state or LWDB encouraged or required any changes to your AJC customer
intake process? What changes did this involve?

3.33

WIOA IMPLEMENTATION EVALUATION

4.

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

To what extent has the state or LWDB emphasized customer or human-centered
design as a key method for making your state’s One-Stop career center system more
customer-focused?
a. What problems or goals do you plan to address through this approach?
b. What challenges and benefits have you encountered in operationalizing it?

5.

Please describe any state or local efforts to improve the accessibility of One-Stops for
individuals with disabilities (e.g. policies, additional funding, or training).

2. Referral Process
1.

How frequently do you refer your customers to partner programs?
a. Which partners are you most likely to initiate referrals to?
b. Which partners are customers most frequently referred from?

2.

How are referrals made to each agency?
a. Do staff provide contact information for the program to a customer?
b. Do staff connect directly with the program they are referring a customer to?
c. Do staff submit a referral via shared MIS?
d. Or, is there some other referral process?

3.

Are referrals tracked? Is information shared about referred customers with the
program the customer is being referred to?

4.

What do you think is the most successful aspect of your referral process?

5.

Is there anything about the referral process you would like to improve?

3. Co-enrollment
1.

About how frequently are your customers co-enrolled in partner programs? Which
programs and for what reasons?

2.

How, if at all, do co-enrollment processes differ from referral processes?

3.

How, if at all, is information about co-enrolled customers shared with partner
programs?

4.

What is the most successful aspect(s) of your co-enrollment processes?
a. What requirements particularly facilitate co-enrollment?
b. Are there factors that impede co-enrollment?

5.

Is there anything about your co-enrollment processes you would like to improve?

4. AJC Partner Meetings
1.

Do you have regular AJC partner meetings? (If no, skip to cross-training)

2.

Are these meetings for individual AJCs or only across the local area?

3.34

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

3.

Do you attend regularly?

4.

Who from your staff attends these meetings? Program managers? Line staff? Both?

5.

Who organizes and facilitates these meetings?

6.

How often do they occur? For how long?

7.

What topics are covered in these meetings?

8.

How helpful have these meetings been?
a. Is there any way they could have been improved?

5. Cross-training
1.

Have you participated in any cross-training sessions across different programs? (If
not, skip to 9)

2.

How often have these sessions occurred? At what locations?

3.

Are these in-person or virtual?

4.

How long did/do they last?

5.

Who attends regularly? Are these local or state-wide trainings?

6.

Who organizes/facilitates?

7.

What is covered in these sessions?

8.

How helpful have these sessions been?
a. Is there any way they could have been improved?

9.

In what other ways—if any—do you share information with partner programs?
a. How well do these other methods work?

6. Use of shared MIS
1.

Which management information system (MIS) do you use? For what purposes?
i. Case management?
ii. Participant tracking?
iii. Performance reporting?
iv. Fiscal reporting?

2.

Can other partners use this system too? If so:
a. Can all partners view data? Are there specific fields that are restricted?
b. Can all partners enter data into the system? Are there specific fields that only
some partners can enter?
c. Can all partners revise data? Are there specific fields than only some partners can
modify?

3.35

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

3.

How do you ensure that the data being entered into these systems by different
partners is accurate?

4.

How are you protecting the security of the data being entered and stored?

5.

Has the state or LWDB encouraged, required, or led the development of a shared case
management system?

6.

How important is it for seamless services that partner programs use the same MIS?

C. Performance Accountability
1. Common Measures
1.

Have you faced any challenges implementing the WIOA common performance
measures?

2.

Have you faced any challenges in implementing state-specific performance
measures?

3.

Have you faced any challenges in implementing local performance measures?

D. Services for job seekers and employers
1.

What strategies have you implemented to increase the use of work-based and jobdriven training (such as OJT, apprenticeships, transitional training, internships)?
a. What work-based training models are most commonly used in your local area?
b. How is your local area working to improve the use of registered apprenticeship?
c. Are there regulations around apprenticeship that could or should be relaxed to
improve the use of registered apprenticeship in your state/region?

2.

Please describe your efforts to increase the use of registered apprenticeship programs.
a. Have there been any challenges related to including apprenticeships on the ETPL?

3.

Please describe your efforts to improve access and services to more barriered
customers, including disabled customers and those with low basic skills.

4.

Have you received any guidance or TA from the state on improving services to
customers with barriers? How helpful was it?

5.

What, if any, additional guidance should USDOL or your state should provide on
serving customers with barriers?

E. Youth
1.

How successful have you been in meeting the 75% OSY requirement?
a. What strategies or policies have you implemented to meet this requirement?

3.36

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

2.

Please describe any key challenges you have had in meeting this requirement. If
you’ve met the requirement, has anything in particular worked well for accomplishing
that?

3.

What guidance has your state provided regarding the 75% youth OSY expenditure
requirement?

4.

What, if any, additional guidance should USDOL or your state should provide related
to the 75% OSY requirement?

G. Overall Questions
1. Partnerships
1.

How would you assess relationships among core partners? Required partners? Other
partners?
a. Has this changed because of WIOA?
b. If not, why do you think WIOA’s focus on enhanced partnerships has not affected
your partnerships?

2.

Are there sufficient systems in place to ensure regular and effective communication
between partners?

3.

What have been the key challenges and successes related to increasing collaboration
across partners?
a. Are there differences in the extent of collaboration with your core partners versus
other required partners, such as UI, SCSEP, Veterans programs, and TANF?

4.

Overall, how integrated are workforce services now?
a. Has this changed because of WIOA?
b. What changes reflect this more integrated service delivery? For example, are case
managers of different programs meeting more often to discuss co-enrolled clients?

5.

[If respondent feels workforce system is not integrated]: Why do you think WIOA’s
focus on enhanced partnerships has not affected the integration of services?

2. Systems Change
1.

Overall, do you think the workforce system here has changed as a result of WIOA?
a. If so, what are the major changes you can attribute to WIOA? E.g. employer
engagement, partnerships, services more streamlined, data sharing, others?
b. If not, do you feel systems had already begun to change prior to WIOA, or that
efforts under WIOA have not yet been effective in changing the system?

2.

Have there been any negative, unintended consequences of implementing the WIOA
regulations?

3.37

WIOA IMPLEMENTATION EVALUATION

OMB SUPPORTING STATEMENT: PROTOCOL 3.3

a. Are there ways in which you feel the new regulations actually make it more
difficult for you to achieve your goals?
3.

[If respondent doesn’t think WIOA has changed the system]: Do you feel systems had
already begun to change prior to WIOA, or that efforts under WIOA have not yet
been effective in changing the system?

4.

What additional changes are planned or needed to transform your workforce system
to meet the goals of WIOA?
b. What changes would you like to see to make the workforce system better overall?

3. Guidance and Technical Assistance
1. Of the TA provided by DOL national and regional offices on WIOA implementation,
which assistance has been most helpful?
a. Which TA formats and types of materials have been most useful for you?
2. Are you satisfied with the level of TA DOL has provided on WIOA implementation?
3. Have you accessed the ION site to obtain TA materials or guidance on specific issues? If
so, which issues?
4. In what areas would you like to receive additional TA?

Wrap-up
1. How would you describe your overall progress to date with WIOA implementation?
a. Where do you still have plans to make significant changes?
2. Beyond what we’ve discussed today, are there other areas you would like to highlight?
3. Are there any other areas of WIOA implementation our study should explore?

3.38