Process Study Interview Guide
Interview Guide for Process Study of
the H-1B Technical Skills Training Evaluation
Introductory statement for grantee/site staff: The H-1B Technical Skills Training Evaluation (H-1B TST) is being conducted by Abt Associates, under contract to the U.S. Department of Labor. As part of this evaluation, evaluation staff members will visit each of the six H-1B TST evaluation sites. A major aim of the evaluation is to identify lessons learned from your experiences in implementing the projects under this initiative. In conducting site visits to each of the project sites, we are talking to project directors and staff, as well as partner organizations. We are here to learn about your service delivery model and understand how it was implemented under the H-1B TST grant. Our aim is to learn from your experiences, not audit or judge your programs. The views you express will be kept private, and nothing we publish in this evaluation will identify you by name.
Introductory statement for other organizations (Program Partners, Employers): The U.S. Department of Labor is funding a major evaluation H-1B Technical Skills Training grant programs that provide job training in fields that rely on the H-1B visa program. Abt Associates is conducting the study, which is called the H-1B Technical Skills Training Evaluation (H-1B TST). We are members of the H-1B TST Team. [Local site or program name] is participating in this evaluation. We are speaking with service providers in the area to document the other types of employment and training services available in the community. Today we’d like to spend some time learning about your organization and the services you provide. Neither your name nor the names of your colleagues we speak to will appear in any public document. Do you have any questions before we get started?
Confidentiality Statement: Before beginning the interview, I (we) want to thank you for agreeing to participate in the study. I (we) know that you are busy and we will try to be as focused as possible. We have many questions and are going to talk to many different people, so please do not feel as though we expect you to be able to answer every question. And, we understand that your participation in this discussion is voluntary and you may choose to not answer some questions.
In addition, before we start, I want to let you know that though we take notes at these interviews, information is never repeated with the name of the respondent. When we write our reports and discuss our findings, information from all interviews is compiled and presented so that no one person can be identified. We also ask that you refrain from sharing anything we discuss today with others to help us ensure your privacy and the privacy of others we are interviewing. Do you have any questions before we begin?
Topic 1: Local Context - Program Managers, Program Partners, Employers
What is the program’s service area? Possibilities include:
Portion of a city/county
Single county
Multiple counties (within one state)
Other (regional effort)
How large is this area (in terms of geography and population)?
What are the demographics of the service area? (e.g., race/ethnicity, immigrants)
Have the demographics been consistent over time, or have they changed recently?
What has been the economic environment in which your H-1B TST grant-funded project has operated?
Unemployment rate for area served (start of grant and at time of visit; pull from BLS ahead of visit and confirm any trends)
Availability of job openings in area served (generally and for particular population being served)
Wage rates in area served (generally and for particular population being served)
Other local economic conditions that may have affected the project’s ability to recruit and retain participants and training participants’ ability to find employment (e.g., in- or out-migration of major employers, major layoffs, and natural disasters)
What are the community and government supports available to low-income and unemployed individuals, and dislocated workers in the service area? (These can include support from your organization that is not part of this program; support from the government, such as welfare; or support from organizations active in your community, such as scholarships.)
What services do they provide?
Do you collect information on program participants’ enrollment in these programs?
If so, what information do you collect? Can you provide this information to us?
Topic 2: Program Design and Operations
Grantee Organization – Program Managers
Please provide background on your organization [note: obtain brochure/recent annual report on the organization]:
Type of organization
Organization’ budget for most recently completed program year
Organization’s major sources of funding (e.g., WIA, funding from federal/state/city agencies, foundations, private contributions, fee for service, etc.)
Organization’s total # of paid staff: _______ # of people, _______ # of FTEs
Types of clients/customers served or targeted
Please provide background on any major programs/initiatives (besides the initiative that is part of the H-1B TST evaluation) which are designed to assist the grant’s target population in obtaining job training and employment and that are operated by your organization:
Name of program/initiative
Number and types of clients/customers served for most recently completed program year (unduplicated count)
Service area for program
Brief description of services provided
Whether the program is linked in any way to the H-1B TST program
Whether program is being evaluated and type of evaluation
Did you establish a new program with your H-1B TST grant, or expand an existing program?
Were there components to the program that you had in place before you received the grant?
Describe any other organizational features that have affected the H-1B TST program implementation/operations.
Program Goals and Start-Up – Program Managers
What are the overall goals of the program?
Who originally established these goals?
Have these goals changed over time? If so, how and why?
Are the goals modeled on or informed by an existing program model that has been successful elsewhere?
Does a theory of change inform the program’s goals (i.e. are there short-term, intermediate-term, and long term goals that feed into desired program outcomes/or impacts)? If yes, please explain further.
How does the design of the program link to employer or local workforce needs?
Which specific industries does the program target?
Have you modified the SOW (in your contract with the U.S. Department of Labor) during the course of the project?
If so, how and why? (e.g., difficulties recruiting participants, unwillingness/inability of partner to be part of initiative, change to time period or scope of work, etc.)
How did your project start-up and early implementation go (e.g., on-time, slow, etc.)?
What factors facilitated project start-up?
What factors hindered project start-up?
Target Group and Recruitment - Program Managers, Program Staff (those involved in recruitment)
Which types of individuals form the target population of the program?
How have you recruited participants? What methods have you used? Possibilities include:
Distribution of flyers, posters or other educational/informational materials
Informational websites
Toll-free informational hotlines
Outreach campaigns using media (e.g., TV, radio, newspaper, ads on buses/bus shelters, etc.)
Direct mail campaigns
Door-to-door outreach campaigns
In-person outreach presentations in the community (e.g., K-12 schools, neighborhood centers, One-Stop Career Centers, libraries, etc.)
Word-of-mouth
Did you alter your recruitment strategy over the course of the grant or evaluation? If so, in what ways, and did the changes work?
Have you worked with any other organizations to assist with recruitment and/or provide referrals to your program? Possibilities include:
Program partners
Educational institutions
One-Stop Career Centers
Community or faith-based organizations
Courts/correctional system
Employers and businesses
Have there been recruitment challenges? If so, what challenges have been encountered and how have each of these challenges been addressed? Some possible challenges include:
Difficulty finding eligible participants
Many who applied did not meet program eligibility requirements
Some of the outreach strategies did not generate many applicants
Partner organizations did not provide enough referrals
There were not enough resources for recruitment
Changing economic or other conditions in the recruitment areas
Other similar programs competing for the same pool of participants
Potential participants unwilling to consent to participation in the evaluation
What has been the response of the targeted population to the initiative?
Please describe any “best practices” that have emerged out of the recruitment strategies you have employed.
Organizational and Staffing Structure – Program Managers
Please describe the H-1B TST program’s position within the organizational structure:
How is the program situated/managed within the larger organization or institution?
Who has authority over the program?
Is there a role for the local One Stop and/or other employment service network in the delivery or management of the program? If so, please describe the role.
Please describe the program’s staffing structure (number of staff in each position):
Do staff work exclusively on program activities, or do they split their time with other duties/programs?
Do your program staff members ever work off-site?
Is the program is fully staffed or short-staffed?
Have you had any staff turnover since the start of the grant?
Have there been any changes to the staffing structure since the start of the grant?
Institutional and Community Partnerships– Program Managers, Program Partners (Question 20-21 Only), Employers (Question 22 Only)
Under your H-1B TST grant, does your institution/organization operate all grant-funded programs and activities or do you contract, have financial arrangements, or have memoranda of understanding with others for some programs, activities, or other components?
What percentage of your H-1B TST grant activities do you contract out to other organizations?
How many and which organizations do you have a contract or other financial arrangements with to operate H-1B TST grant activities?
Had you partnered with the organization on other initiatives prior to the H-1B TST one? If yes, was the H-1B TST set up in a similar way (such as regarding organizational responsibilities and contractual issues)? If deviations were made, how and why?
If you had not partnered before, how/why were the partners brought into the grant effort?
For each partner (whether contracted or not), what services do you provide to the grant-funded program?
Who coordinates your interactions with the H-1B TST program?
Have you encountered any difficulties in this partnership? If so, please describe.
Has your role changed over time? Please explain.
For each contracted organization, what is total amount of the subcontract?
What kinds of linkages does the program have with employers?
Did employers have any role in developing the program?
Do employers assist with course/curricula development and oversight?
Does the training program have a hiring pipeline or hiring quotas with any employers?
What other services does the training program offer employers (screening, drug testing, etc.)?
Topic 3: Program & Service Delivery
Intake and Assessment – Program Managers and Staff
Who determines eligibility of individuals to participate in your training programs, and what, if any, criteria are used to select among candidates recruited? (ask for copies of any assessment materials)
How are the service needs of participants determined? Please take us briefly through the assessment process, noting any formal assessment tests that you use under this grant (e.g., TABE, interest inventories, substance abuse screening, etc.).
Is an individual service strategy or employment development plan created for each participant? (note: request a blank copy of the form used)
Training Strategy – Program Managers, Instructors, OJT Supervisors
Does the program use classroom training, on-the-job training, or both, please describe the core program components.
Obtain a copy of the curriculum used in the training program.
For classroom training, please describe the training program:
How many students participate in the program at once?
What is the average class size?
How many instructors are there?
What is the sequence of the curriculum?
How long is each of the program components (e.g., total hours; hours per day; days per week)?
How long does it take a typical student to complete the curriculum/program?
Is there open program entry and exit?
Does the program assist individuals in moving from one step on a career ladder to the next?
Are certain courses or units required? If so what is required?
What kinds of credits do participants get for each course they complete?
Is basic/remedial skills training required or available? Explain.
Does the program use contextualized instruction that integrates basic skills and training content?
Are there any work-based components (internships, etc.)? Please explain.
For OJT, please describe this service:
Who provides the on-the-job training?
How are placements made?
What is the length and hours of the program?
What is the level of the wage subsidy?
What is the nature of the training – is it “hands-on” or classroom based or both?
Can individuals be hired in unsubsidized employment at the end of the OJT?
What can participants expect upon completion of the program?
Do they earn any specific credentials?
If so, which ones?
If so, are these recognized by employers or industry?
Is this an established credential or was it created for this program?
Does earning that credential require simply completing the course? Or is there some external exam or practicum that must be passed?
What connections to employers and/or career pathways are available to participants?
What changes to the training program have occurred over time? Please describe.
Employment, Counseling, Financial, and Support Services – Program Staff, Partners (Question 8)
What are the major support services available to program participants through your grant-funded program? Please explain in detail.
Probe for: counseling (academic, personal, and financial), employment assistance, financial assistance or needs related payments, child care and transportation, and soft/life skills training.
Who provides these services?
How/where/when are they delivered?
If they are not available in your grant-funded program, do students in your program access these services from other programs/places within your organization or in the community? Please describe.
Are there any requirements placed on program participants to utilize these services (e.g., meeting with a counselor or case manager every month)?
Does the program provide referrals to participants for other services?
If so, which organizations do you provide referrals to?
What services do these organizations provide?
Do you know if participants have been successful in obtaining services from these providers? If not, why (e.g. high demand)?
What services/supports are available upon program completion?
Probe for: connections to job openings, job search services, job retention, work supports, and internships/apprenticeships.
How long are these services generally provided to individuals after they leave the program?
What data are collected about/from individuals who utilize services after program completion?
To what extent do students use these services?
Topic 4: Service Utilization, Data Collection, and Performance Management
Participation Levels and Course/Program Completion - Program Managers; Program Staff
What assessments do you use to track student progress and performance in each component of the training program (ask for each component if there are multiple phases of a program)?
How are these data collected, when (timing and frequency), and from whom?
How would you determine if a participant was having problems? What action would be taken?
How many individuals have completed the training to date?
What proportion of those who start the program complete the program?
Of those who complete, what pass the certification or exam (if applicable)?
What is the total number of individuals you are expecting will complete the training throughout this grant program? Is training completion proceeding on target? Are certifications or degrees proceeding on target?
How many individuals have been placed in permanent jobs after completion to date (if there are any completions and this information is tracked)? Are these jobs in the same industry or occupation as the training? (Please do not include any incumbent workers in these numbers).
What strategies do you employ to maintain participant engagement, attendance, and ultimately completion?
Do you have issues with participants dropping out before completion? If so, about what percent of participants have dropped out in the course of this grant? Do you use any particular retention strategies?
What options are available to participants who aren’t able to complete the program?
What other pathways are they likely to pursue?
Do you provide referrals for other services in the community to participants who are unable to complete the program or otherwise meet program requirements? If so, please describe.
Support Service Utilization - Program Managers and Staff
Do you track how often participants utilize program support services?
How are these data collected, when (timing and frequency), and from whom?
Have you noticed any trends in the utilization of support services? If so, please describe.
Performance Management and Quality Assurance – Program Managers and Staff, Program Partners (If applicable)
What measures are in place to track the performance/quality of the training program?
Are these measures mandated by your H-1B TST grant?
Are the measures stored in an electronic system?
How do your track the performance of program components that are delivered by partner organizations?
What measures are in place to track the performance/quality of the support services?
Are these measures mandated by your H-1B TST grant?
Are the measures stored in an electronic system?
How do you track the performance of program components that are delivered by partner organizations?
Do you collect information from participants about their experiences in the program?
If so, please describe how you collect this information and what you do with it.
Can you describe the recent trends in the performance of the program overall?
Additional Data Collection – Program Managers
What other data are collected by/about the program and its participants?
How are these data collected, when (timing and frequency), and from whom?
How do you use or plan to use these data (i.e. what is the purpose of the data collection)?
Topic 5: Counterfactual Condition
Control Group Services– Program Managers, Program Partners (Questions 2-4)
What are the other training programs individuals can access in the community?
What type of training do they provide?
Who operates these programs?
Where are these services located?
Is there an application process/waiting period to access these services
To what extent are you aware that control group members are utilizing these services?
What barriers might they have to accessing these services?
Do you track what services control group members access?
When individuals are assigned to the control group what service referrals do you provide them?
Topic 6: Implementation Accomplishments and Challenges
Program Successes and Challenges- Program Managers and Staff, Program Partners
Are there ways in which the program has exceeded or fallen short of meeting its goals? Please describe.
What do you believe have been the program’s greatest successes?
What are the main factors that contributed to this success?
What challenges did the program encounter during start-up? How were these challenges overcome?
What challenges has the program encountered since then? How were these challenges overcome?
What are the most important areas for program improvement that your program or your organization/institution has identified?
How do you plan to address these?
Sustainability & Future Plans – Program Managers
Based on the experience to date, are you planning to continue to offer this program in the future?
If so, are you planning on making any modifications and/or changes to the program?
If not, what is the primary reason that you will not continue the program?
Are there other approaches, strategies, or services that you believe would contribute to better outcomes for the target constituency?
What resources are required to continue to operate this program?
What sources of support are you pursuing to replace the funding provided by your H-1B TST grant?
OMB No. # 1205-0507, Exp: 6/30/2019
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |