1205-0436 Part B - FINAL -5-8-23

Employment and Training Administration Quick Turnaround Surveys

OMB Control No. 1205-0436

SUPPORTING STATEMENT

QUICK-TURNAROUND SURVEYS AND SITE VISITS, Part B

OMB Control No. 1205-0436

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.       Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used.  Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample.  Indicate expected response rates for the collection as a whole.  If the collection had been conducted previously, include the actual response rate achieved during the last collection.

This request is for eight (8) to twenty (20) surveys or sets of site visits over three years, which would involve collecting information at the state or local level, or some combination of both, from workforce agencies or boards, partner programs, services providers businesses, or other relevant entities. Given the nature of this ICR, individual collections will vary as to type of respondents and their number both for site visits and surveys.

Site visits, involving semi-structured discussions, focus groups, on-site observations, document or website reviews, and questionnaires will involve purposive samples with locations selected either to cover a broad range of sites, or to assure that there are sites with specific characteristics related to the topic of interest. Questionnaires may also be administered as an adjunct to other methods during site visits, in order to obtain key background information or more details on respondents’ experience and perceptions, and to enhance the likelihood that information is obtained from a broad range of respondents. However, no statistical methods will be used in the analysis of any data obtained during site visits.

Surveys administered to either a universe of respondents or a random sample, including possibly stratified random samples, will allow for statistical analyses Drawing samples using stratified random selection can improve the efficiency of sample estimates when the dimension on which the sample is stratified is strongly related to the question being examined.  For example, when administrative and governance structures are being explored in a survey, program size (based on funding) is likely to be a reasonable stratification criterion.  

On this basis, the calculations on the following page offer examples of sampling rates and strategies for drawing samples among State workforce Agencies, local workforce areas, and American Job Centers.  The calculations are based on a likely maximum sample size of 250 respondents; however, it is understood that an occasional survey may exceed that size, in order to improve the robustness and generalizability of the survey results.

A calculation of the precision and necessary sample size will be done for each survey. Information on those, and details of minimal substantively significant effect sizes, will be included in Part B of the abbreviating support statement, which will be provided to DOL’s Chief Evaluation Office (CEO) for review and approval, and will be included in the material submitted to OMB under this ICR.

Examples of Sampling Rates and Sampling Strategies

 

Location/Types of Respondents	Size of Universe	Number of Respondents	Sampling Strategy	Description of Strategy
State Workforce Agencies	67	67	Universe
Local Workforce Areas	540	250	Stratified random selection	After ranking local areas by amount of funding, five strata will be defined.   The first strata will include the largest local areas that make account for 20% of the areas, the second stratum will include local areas that make up the next 20% and so on.  An equal number of local of workforce areas (50) will be selected randomly from within each stratum
American Job Centers (AJC)	2,294	250	Stratified random selection	Of 250 local areas will be selected using the method described above.  One Center will be randomly selected within each of the local workforce areas, using a simple random sampling

While past surveys under previous versions of this ICR often achieved high response rates (including several that exceeded 80 percent), expected response rates will vary for individual information collection requests. Information on expected rates will be discussed in any requests submitted under this ICR.

As noted above, future sets of site visits (involving structured discussions, focus groups, observations, document reviews and questionnaires) will involve purposive samples with locations selected either to cover a broad range of sites, or to assure that there are sites with specific characteristics related to the topic of interest. For such site visits, the (purposive) sampling methods will be discussed in the abbreviated supporting statement even though no statistical methods will be used.

 

2.       Describe the procedures for the collection of information including:

 

        Statistical methodology for stratification and sample selection,

        Estimation procedure,

        Degree of accuracy needed for the purpose described in the justification,

        Unusual problems requiring specialized sampling procedures, and

        Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

As noted above, site visits will not involve use of statistical methods in the sampling, nor in the analysis of any data obtained. The information from site visits, which will be qualitative in nature, will, however, be analyzed using various methods, including possible use of Natural Language Processing programs, to identify themes and patterns, as well as variations among respondents.

However, for each survey to be conducted, the abbreviated supporting statement (Part B) will identify the population(s) of interest, the population parameters (including formulas) about which the ETA wishes to make inferences, the sampling design, the survey instrument, point estimation and variance estimation methods to be used (again, including formulas), and nonresponse adjustment or imputation methods, if any, to be used in the analysis.

Below is an example of procedures for a typical survey:

a) Sample Selection.  Sample selection will use one of procedures described above, in answer to question 1.

b) Estimation Procedures.  The analysis will primarily make use of frequency distributions and cross-tabulations, which will provide basic information about agency opinions and service strategies.  Basic characteristics of all local areas are known from existing sources (e.g., their level of funding, types of persons served, type of administrative structure, location, etc.); the data can be used to suggest whether non-respondents differ in any substantial way from respondents and to develop weights for respondents in order to accurately depict the frequencies or cross-tabulations.

c) Degree of Accuracy.  For surveys directed to state agencies, sampling will not be employed.  Thus, results should be an accurate reflection of the relevant universe, subject to the constraints of reporting error, after appropriate adjustments for non-response bias are made. 

For surveys directed to local areas or programs, sampling will be utilized. For example, in regard to American Job Centers, we may select a sample of 250 respondent entities, as described above.  Assuming a response rate of 70% would yield completed interviews for 175 entities.

The degree of accuracy for a survey directed to local areas or American Job Centers will be determined as follows. Suppose H strata of sizes N_h, h = 1, 2, …, H, from each of which n_h units are sampled respectively by simple random sampling (srs).  Suppose the proportion of “successes” in the hth stratum is p_h.  Then the overall proportion of successes in the population is

,

where N is the total population size and S is the total number of successes in the population.

 

An unbiased estimator of p is

,

where , s_h  being the number of successes in the sample from the hth stratum, so that  is an estimate of the number of successes in the population.

 

The variance of this estimator is given by

 

.

d) Unusual problems.  There are no unusual problems.

e) Periodic Data Collection.  Each survey and site visit data collection will be administered only once.

3.    Describe methods to maximize response rates and to deal with issues of non-response.  The accuracy and reliability of information collected must be shown to be adequate for intended uses.  For collections based on sampling, a special justification must be provided for any collection that will not yield reliable data that can be generalized to the universe studied.

In order to maximize response rates and minimize non-responses, both during site visits and for surveys, ETA and its contractor will conduct outreach to inform respondents about upcoming site visit activities or survey, the reason for the data collection, how privacy will be protected, and to request their response.

Maximining responses during site visits, during structured discussions and focus groups, with managers, program staff, and current, former, or potential program participants will multiple activities. These will include careful advance scheduling, offering virtual or remote options, , and communicating with potential respondents about the nature of the questions and the importance of their views and experiences.

ETA will attempt to maximize response rates for surveys by contacting respondents multiple times until a completed survey form is obtained.  The follow-up will involve efforts by phone (such by calls or texts), and if a response is obtained, there will be an attempt to administer the survey at the time. 

For surveys of state agencies or local areas, key characteristics from existing sources (e.g., funding level, types of persons served, type of administrative structure, location, etc.) will be used for exploring whether non-respondents differ in any substantial way from respondents, and to develop weights for respondents in order to accurately depict the frequencies or cross-tabulations.  Because the sample will be drawn using probabilistic selection methods, results will be generalized to the universe after making appropriate corrections for potential selection bias.

4. Describe any tests of procedures or methods to be undertaken.  Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility.  Tests must be approved if they call for answers to identical questions from 10 or more respondents.  A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Potential instruments, such as survey questionnaires and the component parts of site visits protocols (for structured discussions and focus groups) will be pre-tested with 9 or fewer individuals. The pre-tests will be used to assess the clarity of content and wording, respondent burden time, potential sources of response error, and, for surveys, the organization and format of the questionnaire.  The pretest will be used to modify the instruments as appropriate prior to submission to OMB and will be described in the abbreviated supporting statement.

If additional pre-tests of a revised instrument appear to be needed, these will be discussed in the abbreviated supporting statement before the pre-tests are conducted. Further, if, based on experiences during data collection, changes to wording or to procedures are needed, a change request will be submitted with updated materials to OMB for review.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Advice on statistical aspects of each individual data collection will be sought as the collection plan and instruments are developed. Information about the consultation process and contact information any individuals consulted on statistical aspects will be included in the abbreviated supporting statement in the package submitted to OMB.

ETA will also consult with CEO prior to the submission of an individual request to CEO for approval under this ICR.

 

3