1205-0426_Supporting Statement B_12 23 25

Post Separation Data Collection for Former Job Corps Participants

OMB Control No. 1205-0426

Expiration Date: 12/31/2028

SUPPORTING STATEMENT B

Post Separation Data Collection for

Former Job Corps Participants

OMB Control No. 1205-0426

B. Collections of Information Employing Statistical Methods

This information collection does employ statistical methods.

B1. Description of the Population and Sampling to Be Used

Goals of the Data Collection

No statistical methods leading to inferences of larger populations are used.

There are four primary goals of the data collected through the instruments and methods described in this request:

1. to fulfill WIOA-specified performance measurement and reporting requirements for the Job Corps program, specifically regarding reporting on the employment and education outcomes of program participants to determine the performance of each center and the larger program and to provide one part of an employer effectiveness measure;

2. to determine the long-term placement and wage results for the Outcome Measurement System (OMS) which is used to manage many aspects of the program including center-level performance;

3. to report the percentage of employers and work-based learning partners that are satisfied with the performance of Job Corps Information Technology Academy (JCITA) students; and,

4. to enhance the quality of the Job Corps program by collecting information on customer satisfaction with former participants, employers, and work-based learning partners.

Populations to be surveyed

There are two populations to be surveyed:

1. All former Job Corps participants. Participants are defined under WIOA as individuals who meet Job Corps eligibility criteria, have been accepted and enrolled into the program, and have demonstrated a commitment to the program by either completing the Career Preparation Program (CPP) or having at least 60 days of continuous enrollment.

For the purposes of this collection, the surveys conducted with former participants following the second and fourth quarters after their exit from Job Corps are termed: Q2 Survey; and Q4 Survey, respectively. All former Job Corps participants are invited to take both the Q2 and Q4 surveys. Additionally, all former participants will be invited to take the Continued Engagement Survey about midway between their Q2 and Q4 surveys. Table 1 below shows the expected size of the population to be surveyed and anticipated response rates.

Table 1: Estimated Population size and Response Rates: Former Job Corps Participants

	Population Frame	Survey Frequency	Total Annual Responses	Response Rate
Q2 Survey (Appendix A)	28,000	1	17,640	.63
Continued Engagement Survey (Appendix B)	28,000	1	17,360	.62
Q4 Survey (Appendix C)	28,000	1	17,080	.61
Total	84,000	-	52,080	.62

The current Q2 and Q4 surveys under OMB 1205-0426 have been used since last approved in December 2022. The response rate estimates used in this current revision consider the former Job Corps participant response rates using the existing survey and associated incentive and outreach methodologies but also consider some of the improvements to the survey processes which are expected to improve future response rates above current levels. The response rate for the Continued Engagement Survey is estimated based on the historical performance of the Q2 and Q4 surveys

2. Employers and work-based learning partners. This population consists of 1,200 employers and work-based learning partners where former Job Corps participants were reportedly placed. This population will be offered the Employer and Work-Based Learning (WBL) Partner Satisfaction Survey. Table 2 below shows the expected size of the population to be surveyed and anticipated response rate.

Table 2: Estimated Population size and Response Rates: Employers & work-based learning partners

	Population Frame	Survey Frequency	Total Annual Responses	Response Rate
Employer Survey (Appendix D)	1200	1	600	.50

The response rate estimates for the Employer and Work-Based Learning (WBL) Partner Satisfaction Survey are based on contractor experience conducting previous employer surveys.

Uses of the Collected Data

Former Participant Surveys

The data collected with the former participant surveys serves a variety of purposes:

• Job Corps uses the survey data to fulfill part of the performance measurement and reporting requirements for the Job Corps program specified under WIOA. Specifically, the survey results lead to performance outcomes for five of the six primary WIOA reporting measures. The survey results require precision at the level of the center and the Career Transition Services (CTS) provider. Job Corps uses performance-based contracting to select center and career transition service (CTS) contractors, and as a result, bases incentives and award fees, in part, on the results from these surveys. Job Corps' ranking of center and CTS contractor performance uses data collected from the surveys. The ranking of centers and CTS agencies provides one of the major criteria used by Job Corps to determine whether a contract option is renewed and to evaluate contractor past performance for new contracts;

• Job Corps uses the survey data to determine the results of long-term placement metrics in the Outcome Measurement System (OMS). The OMS provides Job Corps with the long-term placement and wage results for former participants. This information is essential for Job Corps to manage the center operators and career transitional services providers and evaluate their outcomes;

• The data is used for independent verification of contractor reported outcomes regarding initial placement and wages. Having an alternate verification source is important, as these metrics are included in assessing the performance of the very contractors who report the outcomes. Additionally, the metrics are part of the system used for determining incentive and award fees for performance-based service contracts.

• Job Corps will use the continued engagement survey to collect updated contact information between the second quarter and fourth quarter surveys;

• Data collected supports the continuous program improvement activities regularly conducted by Job Corps and program operators. The questions in module 6 in both the Q2 and Q4 surveys (Appendices A and C)of the former participant surveys are used to assess former students’ satisfaction with the Job Corps program and determine their reason for leaving prior to graduation, if applicable. This information is essential to Job Corps’ continuous efforts for program enhancement and renovation.

Employer and Work-Based Learning Partners Satisfaction Surveys

The data collected from the work-based learning partners and employers supports the continuous improvement efforts and are designed to assess employer satisfaction with the former Job Corps participants they hired, how well they were prepared to meet the requirements of the position for which they were hired and a rating of their problem-solving skills. The results collected by this survey provide qualitative information about the relevance and effectiveness of the education and training services delivered by Job Corps center operators. The data will also be used to report the percentage of employers and work-based learning partners that are satisfied with the performance of Job Corps Information Technology Academy (JCITA) students.

B2. Statistical Methodology for Stratification and Sample Selection

As indicated above, all groups are a census of the population. No sample selection or stratification is applicable.

For the Work-Based Learning Partner and Employer Satisfaction Survey, a survey will be sent to all of the Work-Based Learning partners or employers of all Job Corps Information Technology Academy (JCITA) students that were placed into Work-Based Learning or with an employer partner.

Former Participant Surveys

The key variables to be collected in the surveys are presented in the following diagram. The questionnaire will begin with questions designed to re-verify placement information obtained from JCDC (Job Corps Data Center). For those who are not placed, the questionnaire will skip the re-verification questions and begin with questions about their current job. Once information on the current job is captured, the questionnaire is designed to capture information on additional jobs held during the quarter. For each job, we will ask information on employer name, start date and end date, weekly hours worked (including overtime), hourly wage rate, any additional payments (such as tips, bonuses, or commissions) and other information needed to calculate total earnings over the quarter.

Using the data collected in these surveys, the contractor will compare post-program outcomes for former Job Corps participants who completed the program and other former participants who left the program prior to completion. In addition to providing supplemental data for WIOA reporting the detailed data in the participant surveys serve to inform program management.

Questions are also asked about other placement and post-program activities, and about safety on center and satisfaction with the program.

Employer and Work-Based Learning Partners Satisfaction Survey

The key variables to be collected in the employer and work-based learning satisfaction survey are presented in the following diagram.

The data collected from the work-based learning partners and employers supports the continuous improvement efforts and are designed to assess employer satisfaction with the former participants they hired, how well they were prepared to meet the requirements of the position for which they were hired and a rating of their problem-solving skills. The results collected by this survey also provides data to report the percentage of employers and work-based learning partners that are satisfied with the performance of Job Corps Information Technology Academy (JCITA) students.

B3. Methods to Maximize Response Rates and Address Nonresponse

Former Participant Surveys

As noted in Table 1 above, expected response rates for the former participant surveys range from 61 to 63 percent based on historical data. Strategies used to maximize response rates for the former participant surveys include:

• The data collection instrument consists primarily of well-tested items.

• The survey contractors work closely with Job Corps placement staff and JCDC to obtain adequate tracking and locating information about all Job Corps graduates and former enrollees.

• An online web version of the survey can be accessed through computers, tablets, and smartphones—modalities that have been shown to be more popular with this age group (see Section Supporting Statement A—Question 3 response).

• The survey contractor will use multiple contact sources such as United States Postal Service (USPS) address, email address, and cell phone number to maximize the opportunity for contacting the respondent through a variety of modes including standard mail, electronic contacts, and consented text messaging. Maintaining contact via multiple modes such as the continued engagement survey between Q2 and Q4 timeframes is expected to improve response rates during the Q4 data collection period. Other touchpoints through text messaging will also help keep the participants engaged between the Q2 and Q4 surveys. In addition, regular contact between students and career transition specialists following separation, and the enhancements to the Career Information System will provide access to updated student locating information.

• The survey contractor will employ sample-locating and refusal-avoidance techniques that have been proven to maximize locating and enlisting the cooperation of youth populations. Graduates also receive monetary incentives from Job Corps during the 12-month service eligibility period that will likely enhance their cooperation with the data collection effort.

• Former participants who complete the surveys are paid an incentive for their cooperation: $25 for completing the Q2 survey; $15 for completing the continued engagement survey; and $50 for completing the Q4 survey.

• The survey contractor will provide periodic reminders via a variety of contact methods including hard copy and electronic means, which helps track graduates and former enrollees who have moved since their separation from Job Corps. Additionally, they will use electronic outreach, which is particularly beneficial for respondents for whom traditional contact information such as phone numbers and USPS mailing addresses are inadequate. Job Corps employs additional searching techniques to maximize responses and elicit cooperation, including:

  • contacting parents, relatives, and neighbors for military placements to obtain current contact information;

  • sending address-correction letters;

  • searching on-line nationwide databases (for example, Accurint, White Pages, Directory Assistance, Lexis/Nexus, reverse lookups, among others);

  • requesting, where possible, information from public agencies _{(for example, motor vehicle} departments and corrections departments);

  • providing a toll-free line for respondents to call; and

  • providing information on the surveys on the contractor’s website.

Job Corps uses online survey techniques and telephone procedures through a support contract that maximizes response rates after respondents have been located. Telephone interviewers are trained to carefully follow these procedures. The contractors carefully monitor and retrain interviewers to correct any weaknesses in their contact and interviewing techniques. Two to three weeks after the first contact with a respondent who initially refused to participate, a senior interviewer will contact the respondent and address the respondent's concerns about completing the interview. The data collection contractors will maintain databases to track survey data, will generate regular reports to identify non-responders, and will support follow-up efforts. These procedures have been used successfully in Job Corps’ data collection efforts to achieve response rates consistent with those projected for these follow-up surveys.

Despite extensive efforts to maximize the response rate, there will inevitably be non-respondents. The non-respondents to the survey create a potential for non-response bias. That is, the respondent sample may not be representative of the population. If, as expected, response rates fall below 80 percent, Job Corps will conduct a thorough non-response bias analysis by comparing the characteristics of non-respondents, as captured when they enter the program, with the characteristics of survey respondents who complete the Q2 and Q4 surveys. Fortunately, as students enter Job Corps, there is a great deal of information on their baseline characteristics, such as age, sex, location, prior education level, in-program accomplishments, length of stay in the program and other important characteristics. Job Corps will compare these characteristics for respondents and non-respondents to determine if the respondent population is systematically different from the non-respondent population.

In past non-response bias analyses, Job Corps found that both male and female response rates exhibit the same pattern over time. Female graduates on average exhibited a slightly higher response rate than males. Similarly, response rates were stable across age groups, with a minor uptick at the older end of the spectrum. Overall, in past surveys, Job Corps found little non-response bias, except those students who were placed in jobs were more likely to respond. For that reason, Job Corps will continue to conduct regular non-response analyses (as described below) to ensure that any non-response bias is identified and accounted for with the new survey results.

Employer and Work-Based Learning Partners Satisfaction Survey

As noted in Table 2 above, we anticipate a 50% response rate for the Employer and Work-Based Learning Partners Satisfaction Survey. This population is stable and relatively easy to reach compared with the former participant surveys, but we expected them to be less motivated to complete the survey. However, Job Corps will conduct a thorough non-response bias analysis by comparing the characteristics of non-respondents with the characteristics of survey respondents who complete the surveys. Fortunately, there is a great deal of information on the baseline characteristics of employers in the survey population obtained from placement data such as organizational size, type, NAICS code, and familiarity with the placed participant. Job Corps will compare these characteristics for respondents and non-respondents to determine if the respondent sample is systematically different from the non-respondent sample.

Strategies used to maximize response rates for the employer and work-based learning partners survey include:

• The survey contractors work closely with Job Corps staff and JCDC to obtain adequate tracking and locating information about employer and work-based learning partners of all placed Job Corps participants.

• A minimum of three contact attempts will be made with employers unless the contact information is found to be invalid.

• Business factors results will be used when determining survey techniques including varying call times to increase chances of availability during business and training staff about reaching the correct person, especially in larger firms.¹

• The survey staff will provide a toll-free line for respondents to call and provide information on the survey on the contractor’s website to reduce refusals.

Missing Data

Former Participant Surveys

Throughout the survey, Job Corps makes a concerted effort to avoid missing data. For example, when the questionnaire includes a question on earnings, the response may be “Don’t Know” or “Refused.” The interviewers are trained to probe further and obtain useful information by providing the respondent opportunities to respond in categories rather than in exact earnings amount (e.g., ranges). This approach is usually effective in obtaining a response and avoiding missing data. Similarly, if the respondent refuses to provide information, interviewers are directed to a probe that reminds the respondent of the private nature of the survey, that information provided by the respondent will only be used for analysis of the Job Corps program, and individual information will not be shared.

To date, item non-response has not been a significant problem in using the former participant survey data to meet reporting requirements. While missingness is expected to be largely missing at random (MAR), if item level nonresponse does exceed 30 percent, Job Corps will conduct an item-level nonresponse assessment. This assessment will focus on key outcomes as well as demographic and program engagement variables that may influence response patterns. We will model item nonresponse using observed covariates and paradata and compare respondent and nonrespondent distributions with an approach that will rely on diagnostic nonresponse bias analyses (with no weighting/imputation) and conservative classification rules. Findings will inform whether potential missingness appears to be MAR and help guide any necessary adjustments.

To mitigate item-level nonresponse, in some cases, Job Corps has been able to use answers to questions in one part of the survey to solve missing data problems in another part of the survey (e.g., answers about the wage or hours of the initial job placement for students who work at that same job as answered in a different part of the survey). When faced with missing data issues, Job Corps has not used any statistical methods for data imputation (e.g., random forest, regression-based techniques) to assign values when key individual data items needed in a calculation are missing. Instead, for example, Job Corps has adopted a conservative approach of not counting a student as placed in a quarter unless all the underlying questions needed to make that determination have been answered and indicate the placement criteria have been met.

To minimize this issue, follow-up probes with responses expressed in relatively narrow ranges have been added to wage, hours and earnings questions to help obtain more accurate and complete data. In other cases, questions related to specific criteria (e.g., at least minimum wage, and at least 20 hours a week) have been added and/or modified to minimize the extent of missing data.

Employer and Work-Based Learning Partner Satisfaction Survey

The Employer and Work-Based Learning Partners Satisfaction survey is a short 10-minute interview, which is not anticipated to have missing data. In previous employer surveys, Job Corps has not used any statistical methods for data imputation (e.g., random forest, regression-based techniques) to assign values when key individual data items needed in a calculation are missing.

Weighting

Former Participant Surveys

The Former Job Corps Participant surveys use a census of former Job Corps participants; weighting is not necessary or appropriate.

Employer and Work-Based Learning Partner Satisfaction Survey

The Employer and Work-Based Learning Partners Satisfaction survey uses a census of work-based learning and employer partners; weighting is not necessary or appropriate.

B4. Test Procedures

Former Participant Surveys

The second quarter and fourth quarter surveys are short surveys of participants (12 minutes when self-administered and 17 minutes when interviewer-assisted). No additional testing has been conducted since valid and reliable data has been obtained from the current survey instrument.

Table 3: Q2 and Q4 Survey Participation and Duration by Mode

Participation Mode	Total Responses	Hours/Respondent	Estimated Hours
Self-administered Web	26,040	12/60 minutes	5,208.00
Interviewer-assisted Telephone Interview	8,680	17/60 minutes	2,459.33
Total	34,720	-	7667.33

The continued engagement survey is an approximately 5-to-10-minute survey, depending on the mode, self-administered online or with a telephone interviewer. The survey—which is based on the previous initial engagement survey—will be administered between the Q2 and Q4 surveys. Like the initial engagement survey, the continued engagement survey is primarily meant to collect updated contact information and to keep the participants engaged between the Q2 and Q4 surveys. No additional testing has been conducted since valid and reliable data has been obtained from previous initial engagement surveys.

Table 4: Continued Engagement Survey Participation and Duration by Mode

Participation Mode	Total Responses	Hours/Respondent	Estimated Hours
Self-administered text (abbreviated survey)	347	2/60 minutes	11.57
Self-administered Web	12,846	5/60 minutes	1,070.50
Interviewer-assisted Telephone Interview	4,166	10/60 minutes	694.33
Total	17,359	-	1,776.40

Employer and Work-Based Learning Partner Satisfaction Survey

The Employer and Work-Based Learning Partners Satisfaction survey is a short 10-minute survey of work-based learning partners and employers of placed Job Corps students that have participated in the Job Corps Information Technology Academy. No additional testing has been conducted. However, we used the Fairfax University of America’s Employer Satisfaction Survey form as a basis for the questions.

Table 5: Employer and Work-Based Learning Partner Participation and Duration by Mode

Participation Mode	Total Responses	Hours/Respondent	Estimated Hours
Self-administered Web	240	10/60 minutes	40
Interviewer-assisted Telephone Interview	360	10/60 minutes	60
Total	600	-	100

B5. Contact Information & Privacy

The survey contractor, Decision Information Resources (or DIR), employs data collection policies and procedures to ensure that survey data is kept private. Respondents’ names and contact information are stored in a computer database that is separate from the database that stores the answers to survey questions. The survey contractor’s computer files are password protected and stored on secure servers. In addition, all the survey contractor’s staff sign a confidentiality agreement as a condition of their employment.

All privacy related data is housed in a secure Citrix environment maintained by the Job Corps Data Center (JCDC).

Contact information for the contractor that will collect and analyze the survey information is:

Decision Information Resources, Inc.
24 Greenway Plaza, Suite 1050

Houston, Texas 77046

Phone: 713-650-1425

No individuals outside of the contractor were consulted on statistical aspects of this design.

1 Fisher, S., Bosley, J., Goldenberg, K., Mockovak, W., & Tucker, C. (2003). A qualitative study of nonresponse factors affecting BLS establishment surveys: results. In 163rd Annual Joint Statistical Meetings.

9