Part B

The Employment and Training Administration (ETA) of the U.S. Department of Labor (DOL) seeks approval to conduct a survey of state Eligible Training Provider List (ETPL) coordinators in all 50 states, as well as Puerto Rico and Washington, D.C. DOL has contracted with IMPAQ International, LLC (IMPAQ) to conduct the Feasibility Study of Using Eligible Training Provider List (ETPL) and Workforce Data Quality Initiative (WDQI) Data for Consumer Reports. The purpose of the project is to explore potential ways that states might provide measures of education and job training program performance to potential customers. The goals of this study are to: 1) provide an overview of how ETPLs are managed and 2) to assess the capacity of states to use information gathered in the process of creating the ETPL to produce program report cards. The survey is a key component of understanding these processes. It will be administered via a Web-based survey; corresponding telephone surveys will be conducted with state ETPL coordinators who do not respond to the Web-based survey within the three week survey period.

The survey is intended to document how states and local areas are administering their ETPLs as well as to gather information on whether and how existing ETPL processes could support education and training program report cards. The survey will focus on how ETPLs are managed in all 50 states, Puerto Rico, and the District of Columbia. Questions will cover topics including: who is responsible for managing the state’s ETPL, what (if any) data is used to certify programs for inclusion on the ETPL, the specific criteria that programs must meet to be certified, whether and how the ETPL is updated, whether providers from other states may be included on the ETPL, whether there are procedures for removing programs from the list, and how current ETPL processes might support report card systems.

Because the administration of the ETPL varies significantly by state and the goal of this survey is to get a complete picture of how ETPLs are administered, it will be necessary to survey one representative from each state, as well as from the District of Columbia and Puerto Rico. To identify the appropriate survey respondents, we are working with officials from DOL/ETA who will provide the relevant state contacts at their disposal.

B1. Respondent Universe and Sampling Methods

B1.1 Sampling Design / Respondent Selection

The survey will be conducted with the universe of state ETPL coordinators across the 50 states, Puerto Rico, and Washington, DC. Therefore, no sampling will be necessary.

Exhibit 1: Survey Respondents

Respondent Group	Number of Respondents	Sampling Method
ETPL Coordinators	52	No sampling

B1.2 Potential Respondent Universe and Sampling Unit

The potential respondent universe is the current state ETPL coordinator for each of the 50 states, Puerto Rico and Washington, D.C. Individual state ETPL coordinators will be asked to complete the survey, as these individuals typically have insight into and knowledge of the state-wide administration of the ETPL. However, there will be no sampling as all state ETPL coordinators will be invited to participate.

B1.3 Population Frame and Estimated Sample Size

Exhibit 2 shows the number of entities in the universe covered by the collection. All individuals in the universe will be contacted so there will be no sampling.

Exhibit 2: Respondent Universe and Sample

Respondent Group	Sample/Universe	Sampling Method
State ETPL Coordinators	52/52	No sampling.

B1.4 Expected Response Rates

We expect an 80 percent response rate for the Web-based survey, based on extensive pre-survey notification and follow-up activities coordinated with the U.S. Department of Labor, as well as the IMPAQ team’s experience on related efforts. Examples of similar efforts that yielded such a response rate include Project GATE, which was conducted for the Employment and Training Administration, U.S. Department of Labor. In this project, three waves of surveys were conducted with response rates of 82.2%, 88.1%, and 80.6% respectively. To capture the remaining 20 percent of ETPL coordinators, IMPAQ staff will call them directly and conduct the survey over the phone. Therefore, in total, we expect a 100% response rate.

B2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

B2.1 Statistical Methodology for Stratification

No stratification will be necessary since the universe of ETPL coordinators will receive the survey.

B2.2 Sample Selection Methodology

The Web-based and phone surveys are being administered to the universe of ETPL coordinators so there is no sampling.

B2.3 Estimates of Variance

N/A

B2.4 Analysis Plans

The results of the survey will be analyzed using basic descriptive statistics. To assess the distribution of responses, we will calculate frequencies, means, and standard deviations, as appropriate, for each survey item. For example, we will examine the frequency at which certain data sources are used to calculate program performance and the average ease of conducting specific tasks associated with managing the ETPL. We will also use cross-tabs to examine relationships between survey items. For example, we will assess how frequently states with subsequent eligibility waivers require the collection of training program performance information. These analyses will be conducted using statistical packages such as SPSS, SAS, and STATA.

B2.5 Minimal Substantively Significant Effect

N/A

B2.6 Unusual Problems

N/A

B2.7 Periodic / Cyclical Data Collection

N/A

B3. Maximizing Response Rates and Addressing Nonresponse

B3.1 Methods to Maximize Response Rate / Issues of Non-Response

The survey will be preceded by an email from DOL/ETA notifying respondents of the upcoming survey. A request to complete the Web-based survey will be delivered in an email that will explain the importance of the project. A unique Web link will be provided for every potential respondent, making the survey very easy to access. The survey will be designed so that respondents can enter and exit it if they wish to complete it in more than one sitting. Based on expert review, it will be brief, taking approximately 15 minutes to complete. All non-respondents will receive weekly follow-up emails reminding them of the survey. If they have not completed the Web-survey within the three week period, they will receive a phone call from IMPAQ staff and be asked to complete the survey over the phone.

B3.1a Nonresponse Bias Analyses

Through combining Web and telephone survey modes, along with a survey pre-notification from ETA, we expect to achieve a 100% response rate. If the response rate is lower than 100%, we will examine observable characteristics such as region, waiver status, etc. to assess the extent to which there are any systematic differences between states with ETPL coordinators who responded to the survey and those who did not. However, we will not conduct formal statistical tests to compare respondents to non-respondents. Because we will limit our analysis to the states that respond to the survey, and thus will not attempt to make inferences regarding the broader universe of states, non-response bias will not affect the conclusions we draw from the survey results.

B3.1b Nonresponse Weights

The purpose of this study is to document specific ETPL practices in as many states as possible. It is not appropriate in this setting to attempt to infer state ETPL policies more broadly based on the responses we receive (assuming a response rate of less than 100%). Therefore, we will not construct non-response weights for use in our analysis. We will limit our analysis to those states whose ETPL coordinators have completed the survey.

B3.1c Other Procedures to Address Missing Data

N/A

B3.2 Accuracy and Reliability of Information Collected

The state ETPL Coordinator in each state is the person that is likely to have the most comprehensive and up to date knowledge about how the ETPL is administered. However, in case there are some questions this individual is unable to answer, the survey instructions ask the respondent to take the time to look up answers or ask other individuals who might be knowledgeable about that particular question.

B3.3 Justification for non-systematic data-collection

N/A

B4. Test Procedures

B4.1 Test of Procedures and Methods to Minimize Burden and Improve Utility

The survey has been reviewed by DOL staff members who are familiar with how ETPLs are managed and administered. It has also been pre-tested with 2 current ETPL coordinators. An initial review by DOL staff members provided important insight into how questions will be interpreted by potential respondents and whether the questions being asked are clear and relevant. As a result of this review, some of the questions have been reworded and skip patterns have been modified to streamline the survey. Additionally, the survey has been pilot tested with two current ETPL coordinators using a cognitive interview approach to elicit feedback about how questions are interpreted and answered by respondents. In these cognitive interviews, IMPAQ staff met with individual ETPL coordinators who were asked to answer the survey questions while providing feedback on their comprehension of survey items and their ability to provide reliable answers. Additional changes to questions and response options based on this pre-test were incorporated into the final survey instrument.

B4.2 Approval for Pilot Tests with 10 or More Respondents

N/A

B5. Contact Information and Confidentiality

B5.1 Consultant Contact Information

No uncompensated individuals were consulted on any aspect of this design.

B5.2 Analyst Organization Information

No uncompensated agency unit, contractors, grantees, or other persons will collect and/or analyze the information.

B6. Other Considerations

B6.1 Itemized Project Costs

Item	Hours	Cost
Survey Design and Pre-Testing	362	$62,406.27
Survey Implementation	152	$14,810.96
TOTAL	514	$77,217.23

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	SUPPORTING STATEMENT
File Modified	0000-00-00
File Created	2021-01-28