VERS Justification Part B

B. Collection of Information Employing Statistical Methods

B1. Respondent Universe and Sampling Methods

The target population for the Veterans Employability Survey comprises five cohort groups of veterans. The current status of the veteran will be defined in terms of the sampling frame at the time the sample is drawn. The cohort groups are:

1. Veterans who applied to the VR&E Program, were found to be eligible, but have a current status of “did not show up for an initial appointment;”

2. Veterans who had to temporarily interrupt the evaluation and planning phase of the VR&E Program, and have a current status of “dropped out rather than returning to the program;”

3. Veterans who continued into the evaluation and planning phase of the VR&E Program, and have a current status of “dropped out before a plan was developed;”

4. Veterans who completed the evaluation and planning phase of the VR&E Program, began a plan of rehabilitation, and have a current status of “dropped out or were otherwise discontinued from the program;” and

5. A control group of veterans who have a current status of “successful completion” of the VR&E Program.

The sampling frame will be limited to veterans who had a disposition of completed or interrupted/discontinued participating in the VR&E Program. Between January 2004 and November 2005, the population size of each cohort group is shown below. The time period of the sampling frame will be updated prior to the conduct of the study.

Cohort 1	27,490
Cohort 2	11,714
Cohort 3	14,754
Cohort 4	9,730
Cohort 5	15,898

The objective of the proposed sampling design is to select a sample from each of these five cohort groups so as to achieve 1,000 completed interviews in each. A response rate of 80% is expected for the Veterans Employability Survey. Based on the expected response rate alone, it will be necessary to sample around 6,250 veterans in order to complete 5,000 interviews.

B.2. Procedures for the Collection of Information

Sample Selection. The sampling frame, which is essential for the selection of a probability sample of veterans from each cohort group, will be abstracted from the Chapter 31 Master Record, a database maintained by the VR&E Program. The VA will provide Abt Associates the sampling frame in the form of computerized lists of the five cohort groups. The lists will contain names, addresses, telephone numbers, and other pertinent contact information. Abt Associates will work with the VA to ensure that correct veteran contact information is available for the survey.

As noted above, to achieve 1,000 completed interviews for each of the five groups, we will select a larger sample. In addition to an allowance for nonresponse, this larger sample will allow for the possibility that the contact information for some of the sampled veterans might not be correct. The sampled veterans will be matched with the National Change of Address (NCOA) service and names with undeliverable addresses will be removed. Abt Associates will work with the VA to ensure that the number of names deleted from the frame is kept to a minimum. In addition, it might not be possible to locate all of the NCOA-matched cases. Given these factors, a larger than needed initial sample size of 3,000 veterans will be drawn from each cohort group (15,000 in total). We will group the initial sample of 3,000 veterans per cohort group into replicates (or random subsamples) of 50 cases each and release replicates in a controlled fashion to obtain close to the required 1,000 interviews per cohort, while working towards achieving the target response rate of 80%.

The sampling design for the selection of the sample is relatively straightforward, since separate computerized lists for the five cohorts will be provided. We plan to sort the population for each cohort group by Census Region to ensure geographic representation of the sample. Within each Census Region, the sample will be sorted by Service-Connected Disability rating (<50%, >=50%, and unknown). For cohorts 3 to 5, within each Service-Connected Disability rating, the population will then be sorted by Serious Employment Handicap (yes versus no). Because each cohort sample will produce 1,000 completed interviews, the VA will be able to analyze data on such characteristics as age, race, ethnicity, and sex.

The required sample from each cohort group will be selected using systematic random sampling. The sorting described above followed by systematic sampling will ensure proportional representation of key subgroups.

In stratified sampling, such as that to be used for the Veterans Employability Survey, we sometimes find that some units were either misclassified at the time of selection and were therefore selected in the wrong stratum or changed strata after sample selection. This should not cause problems in obtaining unbiased estimates. If we find that after selection, some selected veterans are in a different stratum (i.e., cohort group) than at the time of selection, we will classify the veteran in the correct stratum but retain the sampling weights computed on the basis of their original probabilities of selection. An estimate for that stratum is obtained using the corresponding sampling weights. The units in the same stratum may have different weights, but the estimates will be unbiased. The variance will slightly increase due to differential weights if stratum movement is small.

Estimation Procedure. For obtaining population-based estimates of totals, means, etc., we will assign a sampling weight to each responding veteran in each of the cohort groups. This combines the base sampling weight, which depends on the probability of selection of the veteran, and an adjustment for unit nonresponse. The sampling frame contains variables that can be used to rake the sample to known population control totals. These final weights will be used for all statistical analyses.

Degree of Accuracy. The sample design for the Veterans Employability Survey recognizes that the five cohort groups have very different levels of experience with the VR&E Program and that it is important to understand how their exposure to the program relates to their current status and prospects in the labor force. The sample design calls for separate estimates, with a reasonable level of precision, for each cohort group. We selected a target sample size of 1,000 interviews because this sample size will provide estimates of population percentages with a 95% confidence interval half-width no larger than about 3.3 percentage points, assuming a design effect from weighting of around 1.1. A target of 1,000 interviews per cohort group will also provide reliable subgroup estimates within cohorts. Examples of important subgroup estimates include estimates by degree of Service-Connected Disability and Serious Employment Handicap status, which apply only to cohort groups 3 to 5. Table B.2-1 shows the expected 95% confidence interval half-widths, for a percentage equal to 50%, for Service-Connected Disability categories within each cohort group. Table B.2-2 shows the expected 95% confidence interval half-widths, for a percentage equal to 50%, for Serious Employment handicap categories within cohorts 3 to 5.

Table B.2-1: Expected 95% confidence interval half-widths for a percentage equal to 50% for Service-Connected Disability categories within each cohort group

Service-Connected Disability rating	Cohort 1	Cohort 2	Cohort 3	Cohort 4	Cohort 5
<50%	4.4	4.7	4.8	4.6	4.3
>=50%	5.2	4.8	4.6	4.9	5.5
Unknown	13.1	12.8	15.1	13.4	11.5

Table B.2-2: Expected 95% confidence interval half-widths for a percentage equal to

50% for Serious Employment Handicap categories within cohort groups 3 to 5

Serious Employment Handicap	Cohort 3	Cohort 4	Cohort 5
Yes	4.9	4.8	5.3
No	4.3	4.4	4.1

An equally important aspect of the design is to allow for comparisons between cohort groups. A sample size of 1,000 completed interviews in each cohort group will make it possible to detect a difference in percentages, at the .05 significance level for a 2-tailed z-test, of around 5.7 percentage points between any two cohort groups, with 80% power. We expect some of the differences to be of this order of magnitude or somewhat larger.

Unusual Problems Requiring Specialized Sampling Procedures. There are no unusual problems associated with these sampling procedures.

Use of Periodic Data Collection Cycles to Reduce Burden. This clearance request is for a one-time survey.

Data Collection Procedures. The VA will have oversight responsibility for all activities conducted by Abt Associates Inc. These activities include: specifications for data collection procedures, sample design, specific content of the questionnaire, instructions for the administration of the interview, and procedures to measure quality control. The contractor is responsible for drawing the veteran sample, performing the necessary mailing and telephone-interviewing operations related to the survey, and monitoring quality throughout the data collection phase.

Quality Control Procedures. Telephone data collection for the Veterans Employability Survey will be subject to strict quality control procedures. Five percent of all telephone interviewing work will be monitored by supervisory staff to ensure that introductory material is properly read, the item wording and sequence of the questionnaire are followed, respondent questions are properly answered, responses are properly probed, and interviewers’ recording of responses into the CATI system is accurate. Interviewers will be given active coaching and feedback on their performance.

B.3. Methods to Maximize Response Rates and Deal with Non-response

A variety of methods will be used to maximize response rates, such as limiting the time required to complete the questionnaire to 20 minutes, quality control of telephone interviews, and careful monitoring of production rates on a weekly basis.

Survey Non-response. As a result of the survey topic and the ease of participation, a high response rate is expected. Nonetheless, careful monitoring to ensure that the data are collected in an accurate and timely manner and that non-response is minimized will be maintained. Efforts to ensure high response rates will include efficient scheduling of callbacks through the CATI call scheduler, training interviewers in refusal aversion techniques to prevent initial refusals, and training refusal converters to overcome resistance to the survey.

Interview data will be weighted to represent the population from which the sample was drawn. The weight combines a basic weight reflecting the probability of selection and an adjustment for veterans not responding to the survey. For non-response adjustment, suitable unit nonresponse weighting classes will be created after evaluating non-response bias in the sample. The following nonresponse bias evaluation is planned. Because we are using a list sample for this survey, there are background variables available for both nonrespondents and respondents:

• Census Region

• Date of birth

• Gender

• Severe employment disability indicator

• Degree of disability

Once the data collection has been completed, respondents within each cohort group will be compared with the population on these five characteristics. Any major differences will be addressed with poststratification weighting adjustments.

B.4. Tests of Procedures or Methods to be Undertaken

A 150-case pretest (30 interviews with veterans in each of the five sample strata) to refine data collection methods is planned after OMB approval is obtained.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Michael P. Battaglia Abt Associates Inc. 55 Wheeler Road Cambridge, MA 02138

David C. Hoaglin, Ph.D. Abt Associates Inc. 55 Wheeler Road Cambridge, MA 02138

Christine Elnitsky Office of Policy Department of Veterans Affairs 810 Vermont Ave., N.W. Washington, DC 20005

Abt Associates Inc. will complete the data collection and will conduct the analyses in consultation with the VA.