Justification B

SUPPORTING STATEMENT B
FOR SURVEY OF VETERAN ENROLLEES’ HEALTH AND RELIANCE UPON VA

VA FORM 10-21034g, OMB CONTROL NUMBER 2900-0609

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

Overall basic survey methodology is described below, but commencing with the May 2006 Terms of Clearance for this survey it is essential to note that, in addition to the descriptions below, VA has agreed to gather additional information on outcomes, test alternative procedures, and conduct methodological research to improve the methodology of this survey and assess the quality of information in the sampling frame of administrative records as well as potential non-response bias. More specifically, VA will work with their contractor to enhance the coding of case level outcomes to better understand reasons for non-response and conduct research that may include tracing Veterans with inadequate contact information, increasing the number of call attempts, and testing strategies for identifying and handling Veterans in institutions, etc. VA will share specific plans and will consult with OMB on this research before conducting these studies.

1. Provide a numerical estimate of the potential respondent universe and describe any sampling or other respondent selection method to be used. Data on the number of entities (e.g., households or persons) in the universe and the corresponding sample are to be provided in tabular format for the universe as a whole and for each stratum. Indicate expected response rates. If this has been conducted previously include actual response rates achieved.

The Health and Reliance Survey universe to be sampled is the enrollees specified at some point in time. For example, the approximately 7 million living enrollees as of April 2008 was the population of interest for the 2008 survey.

For previous surveys, the sampling frame of enrollees had been stratified into 294 strata based on VISN (21), enrollee type (2: pre or post) and priority group (7: 1-6, 7/8). The target number of interviews for each combination was 200 for priority groups 1-6 and 400 in priority group 7/8. To increase the data utility for OEF/OIF, VHA added additional strata based on OEF/OIF status. The stratification and sample allocation have been modified to achieve target OEF/OIF sample sizes for each VISN (200) and the target sample sizes for the original 294 combinations of VISN, enrollee type, and priority.

A random sample of approximately 125 enrollees in each of priority categories 1-8, for the pre and post enrollees, in each of 21 health care networks is expected to yield an optimally stratified sample of approximately 42,000. The attached 2008 Health and Reliance Stratification show the potential priority group definitions and stratifications as well as the projected universe and sample size by strata, based on the 2008 enrollee survey. For 2008 survey a response rate of 35percent and a cooperation rate of 58percent were achieved.

To achieve the goal of 200 OEF/OIF (Operation Enduring Freedom / Operation Iraqi Freedom) enrollees per VISN, we stratified each VISN into one of four OEF/OIF size strata. The reason for grouping the VISNs into size strata is to achieve the sample size targets for each VISN while adding as few new strata as possible (for operational efficiencies). To achieve approximately 200 per VISN, we allocated 4600 interviews to the OEF/OIF sample.

2. Describe the procedures for the collection of information, including: Statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose in the proposed justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

a. SURVEY:

(1) For the Health and Reliance Survey, VA and the contractor have arrayed the VA enrolled population into strata by VISN, priority level, OEF/OIF status (Operation Enduring Freedom / Operation Iraqi Freedom) and pre and post enrollee status. The sampling methodology is to generate an optimally stratified random sample of approximately 100 per strata cells, and call the Veterans up to a maximum of 7 times according to a defined protocol in order to complete an interview.

Beginning in 2008 the VHA Survey included pre-notification letters to all sampled Veterans; the sample was selected and released in four waves. The size and timing of the waves was determined to operationally coordinate the pre-notification mailings, telephone interviewing, and fielding schedule. The overall number of Veterans selected was determined by reviewing historical response rates and estimating the impact of all Veterans receiving a pre-notification in 2008. The first wave included 100,009 enrollees. One week later, a second sample of 44,226 enrollees was selected. During week three an additional 45,114 Veteran enrollees were selected. The fourth wave (9,267 Veteran enrollees) was much smaller and designed to reach the targets in strata where additional sample was required. Most strata did not require a fourth sample selection to be brought to protocol.

For a sample size of approximately 42,000, we expect survey estimates based on the total sample to have error margins of approximately +/-0.5 percentage points at the 95 percent confidence level. For each priority level combining pre and post enrollees within VISN, with a sample size of approximately 200, we expect survey estimates to have error margins in the range of approximately +/-7 percentage points at the 95 percent confidence level. Confidence interval projections are based on measuring a population percentage equal to 50 percent. These projections do not account for sample design effects, which may increase the actual error margins for the survey estimates. VA will provide the contractor a list of enrollees from which to draw. This survey will be repeated on an approximately annual basis.

(2) The 95percent confidence intervals for data from the enrollee survey are a function of the number of cells in the tabulations. The data collected for this survey are from a survey sample. Inherent in a sample is sampling error. Since sampling error can be estimated, it is important to consider standard errors when comparing subpopulations, such as among VISNs.

(3) Over the years that the survey has been conducted, the questionnaire has evolved to meet VHA’s increasingly complex data needs. In particular, some of the questions have changed little over time, as they probe for basic socioeconomic, demographic, and health-related data. However, other questions have been added, deleted, or expanded upon to address more and different sorts of issues of timely and topical interest to VHA. When questions change to reflect the Departmental emphasis and VHA’s specific, forward-looking data needs and interests, OMB will be asked to approve the new survey questions.

b. FOCUS GROUP PROGRAM:

(1) Key Informant and Stakeholders to be interviewed (Phase I) would probably be selected by a VHA enrollee survey committee on the basis of the potential participants' knowledge and experience of Veterans and with Veterans issues. For example, a committee might identify certain people here at VACO who it feels can contribute to the survey effort, and/or the committee might decide to send a global e-mail out to Network Planners in the field requesting recommendations of names of potential participants and then just go down the list that we get back till we have enough people who agree to participate in Phase I.

(2) On the other hand, Veteran participants in individual In-Depth Interviews or in Focus Groups (Phases II and III/IV) ) would be selected by random sampling from the VHA enrollment file, based on stratification by the basic characteristics of interest - which will vary from one survey year to the next. In order to include non-enrolled Veterans, as well, we would probably use Random Digit Dialing techniques to locate potential participants. The number of potential respondents will be increased until a sufficient number of individuals have agreed to participate.

(3) However, as Focus Group activities are "qualitative" and not "quantitative" research, none of these participant selection methods are statistical samples in the true sense; there are no weights or statistical errors to compute. However, the methods outlined above do allow for covering fairly broad cross-sections of the enrollee and Veteran populations, as might be necessary or desirable in conducting focus groups.

3. Describe methods used to maximize the response rate and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Veterans generally identify with the need to provide VA information to improve the system’s ability to provide for their care. The response rates for the Health and Reliance Survey of approximately 35 percent in 2008 is an improvement from the 2007 response rate of 24 percent. VA continues work to improve that rate, which is primarily due to a lack of adequate phone numbers. Nevertheless, once Veterans were contacted, their cooperation rate was 58 percent in 2008; up from the cooperation rate of 49 percent in the 2007 survey. This compares to response rates obtained in other national telephone surveys. Terms of clearance state that OMB continues to encourage VA to work with their contractor to enhance the coding of case level outcomes to better understand reasons for nonresponse and conduct research and test strategies for identifying and handling Veterans in institutions. Enhancements to the survey in 2008 included a pre-survey letter, an increase from 6 to 7 callback attempts, and a reverse lookup function of a portion of the sample.

Given the wide variety of covariates that are related to the likelihood of an enrollee responding, we instituted a weighting adjustment that corrects for this differential. The weighting adjustment includes health utilization and demographic information. Stratum level disproportionate sampling is adjusted for with base weights (or design weights) prior to the nonresponse adjustments. We also instituted a propensity score weighting adjustment, in which individual propensities to respond are measured with a probability model (such as the logistic regression modeling presented herein). The estimated probabilities are used to group the enrollees into classes with similar probabilities. Typically five classes (or quintiles) are formed. Within each class, the respondents are weighted up to account for the nonrespondents. These weighting adjustments reduce potential bias to the extent that the nonrespondents and respondents with similar response probabilities are also similar with respect to the survey statistics of interest

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions of 10 or more individuals.

Earlier (1999, 2000, 2002, 2003, 2005, 2007 and 2008) survey questions on the Health and Reliance Survey have been clarified based upon Veteran or interviewer questions and input. Any proposed survey questions will be pre-tested on fewer than 10 Veterans in order to work out any problems with wording or Veterans’ comprehension of questions, etc., before full implementation of the CATI surveys. The attached VHA enrollee survey instrument includes all necessary modifications to date in the survey instrument.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Jim Schaefer (Tel. 919-334-8040)

VHA Office of the Assistant Deputy Under Secretary for Health for Policy and Planning,

Department of Veterans Affairs

810 Vermont Avenue NW

Washington, DC 20420

Marybeth Matthews (Tel. 414-384-2000 ext 42359)

VHA Office of the Assistant Deputy Under Secretary for Health for Policy and Planning,

Department of Veterans Affairs

5000 West National Avenue

Milwaukee WI 53295

Karen Lentz (Tel 414-384-2000 ext 42365)

VHA Office of the Assistant Deputy Under Secretary for Health for Policy and Planning,

Department of Veterans Affairs

5000 West National Avenue

Milwaukee WI 53295

Milliman Inc.

Kathi Patterson, FSA, MAAA, (Tel. 206-504-5539)

Principal and Consulting Actuary, Contractor

Ed Jhu, Principal Statistician, Contracter (Tel. 206-504-5828)

Macro, Inc. (Tel. 802-863-9600)

Andrew Dyer

Randall ZuWallack

126 College Street

Burlington Vermont 05401

Burlington, VT, Contractor for the survey