1205-0436 Supporting Statement Part B_FINAL 6.1.17

Employment and Training Administration Quick Turnaround Surveys

OMB Control No. 1205-0436

June 2017

SUPPORTING STATEMENT

QUICK-TURNAROUND SURVEYS, Part B

OMB Control No. 1205-0436

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.       Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used.  Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample.  Indicate expected response rates for the collection as a whole.  If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Depending on the survey, the administration will be either to the universe of respondents or to a stratified random sample.  Drawing samples using stratified random selection in such circumstances can improve the efficiency of sample estimates when the dimension on which the sample is stratified is strongly related to the research question being examined.  Given that administrative and governance structures and service design and delivery will encompass the most likely research questions being investigated under this clearance, we propose that program size is the most reasonable stratification criterion.  

On this basis, Table 3 offers an example of sampling rates and strategies for drawing samples of states, local areas, and American Job Centers.  The calculations in Table 3 below are based on a sample size of 250 respondents; however, it is understood that an occasional survey may exceed that size, in order to improve the robustness and generalizability of the survey results. A calculation of the precision and necessary sample size will be done for each survey and provided to OMB for approval.  Details of minimal substantively significant effect sizes will be included in the OMB submission for each survey.

Table 3. Examples of Sampling Rates and Sampling Strategies

 

Universe		# of Respondents		How Drawn
State agencies		All in Universe		All in universe

 

Local workforce areas		250 local areas (of 548 in universe)		Stratified random selection.  Five strata will be defined.  After ranking local areas by amount of funding, each stratum will be defined such that each will make up equal proportions of total WIOA funding.  Thus, the first stratum will include the largest local areas. As that make up the top 20% of aggregate WIOA funding nationwide; the next stratum will include local areas that make up the next 20% of funding, and so on.  An equal number of LWIAs (50, or 250/5) will be selected randomly from within each stratum.
American Job Centers		250 Centers (of approximately 2,400 in universe)		Stratified random selection of 250 local areas will be selected using the method described above.  One Center will be randomly selected within each of these LWIAs using a simple random sampling.

 

We expect to receive a response rate of at least 80% for all surveys associated with this information collection plan.  This estimate is consistent with results from previous data collection efforts of similar nature and magnitude, including the Job Corps High School Diploma Initiative Survey administered by the National Office of Job Corps, which received an 82% response rate.

 

2.       Describe the procedures for the collection of information including:

 

        Statistical methodology for stratification and sample selection,

        Estimation procedure,

        Degree of accuracy needed for the purpose described in the justification,

        Unusual problems requiring specialized sampling procedures, and

        Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

For each survey to be conducted, the OMB package will give a very detailed point-by-point statement of the population(s) of interest, the population parameters (including formulas) about which the ETA wishes to make inferences, the full sample design, the data collection instrument to be used, point estimation and variance estimation methods to be used (again, including formulas), and nonresponse adjustment or imputation methods, if any, to be used in the analysis.

For a typical survey:

a) Sample Selection.  Sample selection will be carried out using the procedures described above, in answer to question 1.

b) Estimation Procedures.  The analysis will primarily make use of frequency distributions and cross-tabulations, which will provide basic information about agency opinions and service strategies.  Basic characteristics of all local areas are known from existing sources (e.g., their level of funding, types of persons served, type of administrative structure, location, etc.); the data can be used to suggest whether non-respondents differ in any substantial way from respondents and to develop weights for respondents in order to accurately depict the frequencies or cross-tabulations.

c) Degree of Accuracy.  For surveys directed to state agencies, sampling will not be employed.  Thus, results should be an accurate reflection of the relevant universe, subject to the constraints of reporting error, after appropriate adjustments for non-response bias are made.  For surveys directed to local areas and American Job Centers, we may select a sample of 250 respondent entities, as described above.  However, on occasion, we may choose to sample all of the local areas and American Job Centers. Assuming a response rate of 80% would yield completed interviews for 200 entities from among local areas. 

The degree of accuracy for a survey directed to local areas or American Job Centers will be determined as follows. Suppose H strata of sizes N_h, h = 1, 2, …, H, from each of which n_h units are sampled respectively by simple random sampling (srs).  Suppose the proportion of “successes” in the hth stratum is p_h.  Then the overall proportion of successes in the population is

,

where N is the total population size and S is the total number of successes in the population.

 

An unbiased estimator of p is

,

where , s_h  being the number of successes in the sample from the hth stratum, so that  is an estimate of the number of successes in the population.

 

The variance of this estimator is given by

 

.

d) Unusual problems.  There are no unusual problems.

e) Periodic Data Collection.  Each survey will be administered only once.

3.    Describe methods to maximize response rates and to deal with issues of non-response.  The accuracy and reliability of information collected must be shown to be adequate for intended uses.  For collections based on sampling, a special justification must be provided for any collection that will not yield reliable data that can be generalized to the universe studied.

In order to maximize response rates and minimize non-responses, ETA’s contractor will attempt to contact respondents multiple times until a completed survey is obtained.  When web surveys are used primarily, the follow-up will be by telephone; an attempt to administer the survey by telephone will be attempted on the spot. 

When phone surveys are used primarily, the contractor will attempt to reach each respondent by phone multiple times.  Should this approach not yield a completed survey, telephone calls will be made to another contact person within the targeted agency.  

Basic characteristics of state agencies and of local areas are known from existing sources (e.g., their level of funding, types of persons served, type of administrative structure, location, etc.).  These data can be used to suggest whether non-respondents differ in any substantial way from respondents and to develop weights for respondents in order to accurately depict the frequencies or cross-tabulations.  Because the sample will be drawn using probabilistic selection methods, results will be generalized to the universe studied after making appropriate corrections for potential selection bias.

4. Describe any tests of procedures or methods to be undertaken.  Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility.  Tests must be approved if they call for answers to identical questions from 10 or more respondents.  A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

ETA’s contractor will pretest each survey with up to, but no more than, nine respondents.  The pretest will assess the clarity of content and wording of the survey, the organization and format of the questionnaire, respondent burden time, and potential sources of response error.  The pretest will be used to modify the questionnaire as appropriate.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

ETA’s contractor(s) will refine the design and carry out the sampling plan, collect the data, and prepare data tabulations and brief analyses.  The contractor that ETA uses will vary from survey to survey.  However, the contractors will be selected because their substantive background makes them very familiar with the context of the questions being asked and because of their expertise in survey design and administration and statistical methods. Information about any individuals consulted on statistical aspects of the survey design will be included in the OMB submission for each survey. Contact information for the collecting agency will also be included.

 

5