Analysis of Methodological Experiments and Non-Response Bias for the 2008 Annual Survey of Veteran Enrollees Health and Reliance Upon VA

Analysis of Methodological Experiments and Non-response Bias for the 2008 Annual Survey of Veteran Enrollees’ Health and Reliance UPon VA

Final Report

--- Not For Distribution ---

Office of the Assistant Deputy Under Secretary for Health for Policy and Planning

Date: May 1, 2009

In 2006, Macro International Inc. (Macro) reviewed the research design for the Veterans Health Administration’s (VHA) 2005 Survey of Veteran Enrollees’ Health and Reliance Upon VA (SoE). The review examined the survey process and potential biases resulting from missing or outdated contact information and survey non-response—including both the inability to make contacts and respondent refusals. The report, submitted to the Office of Management and Budget (OMB), made several actionable recommendations for improving the research design. During discussions about this report, VHA, Macro, and the OMB developed a design improvement plan with long- and short-term goals for the SoE.

The 2007 SoE included several methodological experiments to gauge the impact of design enhancements. The experiments included survey pre-notification letters sent by the Under Secretary for Health and extending the maximum number of call attempts from six to 10. The results of these experiments are documented in the 2007 Report “Supplementary Analysis and Technical Assistance for the 2007 Annual Survey of Veteran Enrollees Health and Reliance on VA,” February 14, 2008. Based on the experimental evidence, Macro recommended that both of these design enhancements be adopted for the 2008 SoE. Based on this recommendation, VHA adopted the use of pre-notification letters and increased the maximum call attempts to seven (concern for increased respondent burden and budgetary restrictions prevented an increase to 10).

2008 Recommendations

1. Send pre-notification letters to sampled enrollees and increase the number of call attempts from six to ten.

2. Experiment with reverse phone look-ups based on address information.

3. Experiment with alternative response options for enrollees without a telephone number listed in the database.

4. Continue using the propensity score weighting.

The 2008 recommendations also included two experiments to take advantage of address information found in VHA’s Enrollee database. VHA has address information for nearly all enrollees. Since the 2007 database was missing telephone numbers for about 25 percent of enrollees and another 25 percent were incorrect, addresses can be used to improve the contact information through directory look-ups and/or via other modes of data collection such as mail. On this recommendation, VHA opted to experiment with a telephone look-up using Lexis-Nexis database services. VHA did not offer alternative response options for the survey, but Macro increased capacity and staffing to handle the increased in-bound calling that was expected as a result of mailing pre-notification letters to the selected enrollees.

This report summarizes the methodological enhancements adopted for the 2008 SoE and presents findings for the 2008 experiments. The format for this analysis is largely the same as it was for the 2005 and 2007 reports. Macro evaluates potential biases caused by various steps in the survey process and makes general summary observations based on the 2008 results. When relevant, Macro compares the 2008 results to those from 2007. Finally, in 2006, Macro recommended that the survey weighting include a non-response adjustment for utilization of VHA services. This weighting was first implemented for the 2007 survey. Macro evaluates this non-response weighting based on its ability to mitigate the risk of potential bias.

Background

The Department of Veterans Affairs (VA) serves American Veterans by providing primary and specialized care as well as related medical and social support services. It administers the country’s largest, most comprehensive, integrated health care system. In 2007, VHA served over five million Veteran enrollees. The number of Veterans turning to VHA for health care increases every year, and their need is expected to grow. More and more Veterans are turning to VHA as a result of changes in our nation’s economy, the demographics of the Veteran population, and as benefits available to them under Medicare diminish. In addition, rising health care costs and increasing financial burden placed on the consumer, will also contribute to more Veterans relying on VHA for assistance.

While demand for health care services grows, VHA's ability to meet this demand is circumscribed by the Veteran’s Health care Eligibility Reform Act of 1996 (Public Law 104-262). This law instituted a priority-based enrollment system designed to balance the needs of those Veterans most in need of services, with the necessity to control health care costs and demands on the system. Under this law, the number of priority levels to which VHA can deliver care is a function of annual funding levels and utilization of health care services by enrollees.

The 1996 law also requires VHA to fully understand the reliance of enrolled Veterans on VHA health care services and programs compared to their use of non-VHA services and programs (also known as “VA reliance”). This understanding comes from data gathered through the SoE. The SoE was developed with core and supplemental sections to gather a variety of information to determine the relationship among demographic, socioeconomic, and morbidity characteristics of Veteran enrollees, as well as enrollees’ choice of health care providers and their utilization of health care.

VHA has conducted seven cycles of the SoE (1999, 2000, 2002, 2003, 2005, 2007, and 2008). The data gathered by the VHA SoE also establishes the number of priority levels that VHA can support. It is used to develop health care budgets and to assist the Department for Veterans Affairs with its annual enrollment decisions. This data is also used as inputs into VHA’s Enrollee Health Care Projection Model. Forecasts developed from this model are used for a number of purposes, such as Millennium Bill projects, budgeting, and scenario-based policy and planning analyses.

Any collection of information conducted or sponsored by a Federal agency requires OMB clearance. As part of the FY06 OMB clearance package, VHA was tasked with conducting a non-response bias assessment of the SoE, as well as with examining the quality of the information in the sampling frame. The 2006 analysis satisfied this task. VHA and Macro met with OMB to discuss the 2006 analysis and agreed to develop methods to improve the survey program. OMB granted clearance to VHA with the condition that VHA take steps to improve the design, starting with the 2007 survey. Since then, the SoE has:

• Added a pre-notification letter sent from the Under Secretary for Health that describes the purpose of survey, explains that Macro is conducting it on VHA’s behalf, and provides a number to call with questions or concerns;

• Increased the maximum number of call attempts from six to seven; and

• Improved the weighting methodology to use a propensity score adjustment based on demographics and health care utilization administrative records.

This report assesses the 2008 SoE. The report includes an analysis of the directory matching used to update and locate additional enrollee phone numbers. This report is organized as follows:

• Summary of 2008 methodological enhancements and results of the 2008 survey experiment—directory matching;

• A summary of the sample design for the SoE;

• The sample design and its relation to interview outcomes;

• Results of the bias analysis; and

• Evaluation of weighting adjustments.

2008 Experiments

In 2007, VHA experimented with an extended calling protocol. Specifically, for a small percent of the sample, Macro increased the maximum number of call attempts from six to 10. The analysis of the experiment found that the extended call protocol, coupled with sending pre-notification letters (described below), increased response rates. For 2008, VHA extended the calling protocol from six maximum attempts to seven for all sampled enrollees. Macro estimates that that change increased the overall response rate by two percentage points. Six percent of the interviews were completed on the seventh or higher attempt. The number of attempts exceeded seven for one of two reasons: 1) to fulfill the minimum number of attempts on weekends, weekday evenings, or weekdays; or 2) to honor scheduled call-backs. Response rates are calculated with American Association of Public Opinion Research (AAPOR) Response Rate 1 (RR1), which is a strict definition that assumes all unresolved records are eligible respondents. This response rate is described in a later section of this report.

One of the 2006 findings was that the VHA enrollee database had an address listed for nearly all of the enrollees, whereas it only listed a valid phone number for about three-quarters of enrollees. In that report, Macro discussed a variety of ways to leverage the address information, including:

1. Designing a mixed-mode methodology with mail and telephone data collection;

2. Sending pre-notification letters; and

3. Using the address information to identify additional or updated telephone numbers by running the address and the Veteran name against a reverse look-up database.

Pre-notification Letters. Sending pre-notification letters to selected enrollees was tested during the 2007 survey. Letters were sent to 42,000 randomly selected enrollees who had a valid address. The details of this experiment are discussed in the 2007 report. The experiment demonstrated that the pre-notification had a positive impact on response rates. For this reason, pre-notification letters were sent to all selected enrollees in 2008. In total, 199,505 enrollees were mailed a pre-notification letter. Nine percent of the letters were returned by the U.S. Postal Service as undeliverable mail. Priority groups 4 and 5 had the highest percentage of returned letters with 12 percent each. The lowest return rate was six percent in Priority group 1. The percentage of letters returned as undeliverable as addressed is presented in the Appendix by VISN, Priority group, Enrollee type, and OEF/OIF status.

While Macro has no controlled experiment to suggest that the letters had a positive effect on response, the empirical evidence from the administration of the 2008 SoE suggests the result to be positive. The overall response rate was 35 percent, which is lower than the expected response rate based on the experimental data in 2007. However, the 2008 sample included an OEF/OIF oversample. OEF/OIF enrollees had a lower response rate than non-OEF/OIF enrollees, 22 percent versus 37 percent.

Reverse Look-up Database Match. For 2008, VHA designed an experiment to test the efficacy of telephone look-ups based on name and address of the enrollee. For this experiment, VHA drew a sample of 62,516 enrollees who had address information. About 90 percent also had a valid telephone number. This sample was sent to Lexis-Nexis and matched to a database of addresses and telephone numbers. The database match resulted in a valid phone number for 3,256 of the 6,346 enrollees who did not have a valid telephone number. Further, the match provided an updated telephone number for 16,583 of the 56,170 enrollees with a valid telephone number. The address matching increases the number of enrollees eligible for the frame to 95 percent. The percentage of enrollees with valid contact information is significantly higher based on the database match, 77 percent versus 66 percent. The results of the database matching are in Table 1.

Table 1. Results of Database Matching

Original Status	Results of Match	Number of Enrollees
Valid VHA number	Update through Lexis-Nexis	16,583
	Not updated	39,587
Invalid VHA phone number	Update through Lexis-Nexis	3,256
	Not updated	3,090

The experimental sample resulted in 12,765 completed interviews with a response rate of 33 percent. The remaining 29,695 completed interviews were conducted as part of the non-experimental survey, conducted with enrollees who had a valid telephone number. The response rate for the non-experimental sample was 36 percent. The cooperation rate among the experimental sample is slightly lower, 63 to 65 percent. This might be due to the fact that the matched sample is revealing contact information for enrollees that the VHA does not have, which infers that these enrollees are less likely to rely on VHA for services and thus may be less likely to participate in the survey. This hypothesis is supported by the higher response rates for patients (home health care service, inpatient care, outpatient care, or pharmacy services) versus non-patients, 40 percent to 25 percent. Similarly, the cooperation rate for patients is much higher than for non-patients, 68 percent versus 54 percent.

The matching did not seem to improve the quality of the address information. The percentage of returned pre-notification letters in the experimental and non-experimental sample were nine percent each.

To evaluate whether the database match impacts survey data, four questions about health insurance coverage are examined:

PREA. Are you enrolled in VA health care?

A1. Are you covered by Medicare?

A7. Are you currently covered by Medicaid for any of your health care?

A9. Are you currently covered by any other individual or group health plan that either you, or an employer, or someone else, such as a family member obtains for you?

The percentage of enrollees who report that they are not enrolled is lower in the sample that was matched, 84 percent to 87 percent (p-value < 0.0001). A slightly higher percentage of enrollees in the matched sample report coverage by Medicare (64.9 percent to 63.6 percent, p-value=0.0882). There is no difference in the percentage enrolled in Medicaid or covered by another health plan.

Table 2a. Comparison of Survey Responses for Matched and Unmatched Sample

		Matched Sample	Unmatched Sample	Total
Total responding enrollees		12,765	29,695	42,460
PREA. Are you enrolled in VA health care?	Yes	84.0%	86.9%	85.9%
	No	13.0%	10.3%	11.3%
	DR/DK*	3.0%	2.7%	2.8%
		p-value < 0.0001
A1. Are you covered by Medicare?	Yes	65.0%	63.5%	64.0%
	No	35.0%	36.5%	36.0%
		p-value = 0.0570
A7. Are you currently covered by Medicaid for any of your health care?	Yes	8.2%	8.9%	8.7%
	No	91.8%	91.1%	91.3%
		p-value = 0.1261
A9. Are you currently covered by any other individual or group health plan that either you, or an employer, or someone else, such as a family member obtains for you?	Yes	28.1%	28.3%	28.2%
	No	71.9%	71.7%	71.8%
		p-value = 0.8169

^*DR/DK = I don’t remember enrolling or Don’t know

When evaluating usage of VHA services as measured by administrative records (described in a later section), there are three significant differences: outpatient care for mental health or substance abuse (p-value = 0.0513), outpatient care unrelated to mental health and substance abuse (p-value = 0.0017), and pharmacy services (p-value = 0.0022). As shown in the sections below, the sample of enrollees (prior to weighting) severely overestimates the percentage of enrollees receiving outpatient care unrelated to mental health and substance abuse as well as the percentage of enrollees receiving pharmacy services. The overestimation of these statistics is about 10-12 percentage points. In both cases, the matched sample results in estimates that are two points lower than the unmatched sample. Although the match sample still overestimates the true value, the matching appears to be reducing bias.

Table 2b. Comparison of VHA Utilization for Matched and Unmatched Sample

		Matched Sample	Unmatched Sample	Total
Total responding enrollees		12,765	29,695	42,460
1. Received home health services	Yes	0.13%	0.11%	0.11%
	No	99.87%	99.89%	99.89%
		p-value = 0.6018
2a. Inpatient treatment for mental health or substance abuse	Yes	0.47%	0.48%	0.48%
	No	99.53%	99.52%	99.52%
		p-value = 0.8432
2b. Inpatient treatment for non-mental health and non-substance abuse	Yes	4.8%	4.5%	4.6%
	No	95.2%	95.5%	95.4%
		p-value = 0.3879
3a. Outpatient treatment for mental health or substance abuse	Yes	3.6%	4.2%	4.0%
	No	96.4%	95.8%	96.0%
		p-value = 0.0513
3b. Outpatient treatment for non-mental health and non-substance abuse	Yes	70.7%	73.0%	72.2%
	No	29.3%	27.0%	27.8%
		p-value = 0.0017
4. VHA pharmacy services	Yes	68.5%	70.8%	70.0%
	No	31.5%	29.2%	30.0%
		p-value = 0.0022

Sample Design

VHA provides Macro with a sample of records from its database of enrollees. The sample for the SoE is selected in the following manner:

• VHA considers the entire universe of enrollees who are listed as of a certain date—this list includes both institutionalized and non-institutionalized Veterans.

• VHA eliminates all records lacking a telephone number.

• VHA then eliminates all records for which the telephone number is incomplete or lacks a valid exchange-area code combination.

• Note that VHA experimented with a sample of enrollees from the full database (including those with no phone number). This experiment is described in the previous section.

• VHA eliminates all records for which at least one of the sample stratification variables is absent−namely VISN, pre/post enrollee status, or priority group status.

• The file of enrollees is then stratified by OEF/OIF status, pre/post enrollee status, priority group, and VISN--and independent random samples are drawn for each stratum.

Sampling Design and Interview Outcomes

The final sample of enrollees responding to the SoE must pass through many stages:

• First, to be in the final sample of respondents, an enrollee must be in the sampling frame—meaning that contact information and all stratification variables are available;

• Then, the enrollee must be sampled via the stratified random selection process;

• Next, the enrollee’s contact information must be valid and lead to the correct enrollee; and

• Finally, the enrollee must elect to respond to the survey.

The only stage that is a controlled random process, and therefore not subject to potential bias, is the random sample selection. All other stages have the potential to introduce non-random systematic bias into enrollee estimates. Figure 1 presents enrollee totals at each of the sample stages for the 2008 survey. Table 3 presents the enrollee frequencies for each of the major stratum levels: OEF/OIF status, VISN, enrollee type, and priority group.

The VHA experimented with sampling enrollees without a telephone number and conducting an address match to obtain a telephone number. The 3,256 enrollees where a number was obtained are not included in the frame for this analysis since they otherwise would have been invalid. They are included in the final sample of 179,956 enrollees.

Figure 1. Stages and Enrollee Totals for the 2008 Survey of Enrollees

Table 3. Stages and Enrollee Totals for the 2008 Survey of Enrollees

		Enrollee Population	Frame Eligible	Enrollees Selected	Correct Contact	Survey Responses
Total		7339531	6486807	179956	121376	42460
OEFOIF	N	6988756	6159531	153343	104099	38737
	Y	350775	327276	26613	17277	3723
VISN	1	324436	302502	8077	5722	1881
	2	197695	172153	7804	5499	1939
	3	316569	285126	8955	5995	1707
	4	425748	397519	7953	5838	2008
	5	183262	171015	9085	6110	1761
	6	404312	370527	8514	5706	1898
	7	458197	338092	9503	6121	1769
	8	573439	526396	8807	6047	1874
	9	343107	311080	8038	5556	2001
	10	272635	234174	8314	5742	1967
	11	326891	289934	8246	5652	2017
	12	314717	264928	8363	5777	2146
	15	299398	276967	8228	5526	2234
	16	605778	546113	8933	5884	2178
	17	345483	322079	9053	5773	2045
	18	312841	232777	9096	5679	2111
	19	220811	197793	7926	5332	2234
	20	326671	293610	8373	5462	2185
	21	317673	287832	9076	6198	2173
	22	398204	319654	9653	6083	1944
	23	371664	346536	7959	5674	2388
Priority Group	1	966204	855893	23686	17387	6625
	2	527789	468367	21456	14680	5457
	3	975774	854258	23347	15557	5475
	4	203821	177420	22534	13932	5021
	5	2139811	1870529	32668	21042	6315
	6	325303	294416	14311	9486	2423
	7/8	2200829	1965924	41954	29292	11144
Enrollee type	POST	5098703	4565637	91687	62887	21520
	PRE	2240828	1921170	88269	58489	20940

Frame Eligibility

About 12 percent of the enrollee population was ineligible to be in the sampling frame due to incomplete telephone information or incomplete stratification information—much lower than 2007, when 27 percent was ineligible (and 25.6 percent in 2005). A telephone number may be missing from the sample completely, be missing digits, or not have a valid exchange-area code combination. The improved frame has contact information for a higher percentage of enrollees and reduces the risk of bias due to incomplete coverage. Further reducing the risk of frame coverage bias is the sample of enrollees selected from the full frame of enrollees with and without phone numbers. While this is an experiment, it has potential to increase the number of enrollees who we are able to reach and to whom we may administer a survey.

According to administrative records, about 60 percent of enrollees received services (home health care, inpatient or outpatient care) in the past 12 months. Frame eligibility percentage is slightly higher than for those who have received services, 90 compared to 86 percent. Similarly, the frame eligibility percentage is slightly higher for enrollees receiving the prescription drug service (about 55 percent of enrollees), 90 compared to 86 percent.

Sample Selection

A total of 179,956 enrollees were sampled from the frame in order to meet the sample size requirements for each stratum; this was less than the 2007 survey but comparable in size to the 2005 survey. The sample was stratified, with 298 strata defined by four OEF/OIF groups, 21 VISNs (1-12, 15-23), two enrollee groups (pre and post), and seven priority groups (one through six; combined seven and eight). The sample selection is described in the methodology report.

The sample design is a disproportionate sample with smaller strata receiving higher shares of sample than the larger strata. For analysis at the sampling stage, Macro used design weights equal to the ratio of the frame total to the sample total in each stratum.

Survey Eligibility

All of the enrollees sampled for the survey were called at least once in order to initiate an interview. During data collection, many telephone numbers were classified as ineligible, including: non-working numbers, wrong numbers where selected enrollee is not known, out of service numbers, fax or modem telephone numbers, and business numbers where the enrollee is not known. Although these were ineligible for the survey since they did not lead to the selected enrollee, this loss of sample may impose bias on the survey estimates since these enrollees were part of the population, yet cannot be reached for interview. There were no protocols for identifying an alternative telephone number other than the ability to contact an alternative number if provided. The dialing of telephone numbers during data collection was a second form of frame validation since, albeit the enrollee was included in the frame, the frame information did not lead to the selected enrollee. The percentage of sampled enrollees with invalid contact information was 33 percent; this is the same as in 2007. Compounded with the fact that 12 percent of enrollees are excluded from the frame due to invalid telephone numbers, this suggests that 41 percent of the enrollees are not reachable by telephone.

For enrollees who had received services (home health care, inpatient, or outpatient care) in the previous 12 months, the survey eligibility rate was much higher than for those who had not received services, 75 compared to 60 percent. Similarly, the survey eligibility rate was much higher for enrollees receiving the prescription drug service, 76 compared to 62 percent. It is speculated that enrollees who received services had more opportunities to keep their contact information current and accurate.

Design weights are used in the analysis of the enrollees with correct and incorrect contact information.

Non-response

After determining that the telephone contact information was accurate, the final stage of the process became either a complete interview with the enrollee (response) or unsuccessful interview attempts. Macro classifies non-response into two forms: enrollee refusal and enrollee non-contact. Enrollee refusals result when an enrollee (or an enrollee agent) is contacted, the sponsor (VHA) and purpose of the survey are communicated, and the enrollee elects not to participate by verbal refusal, hang-up, or other form of termination. A non-contact means that the enrollee (or an enrollee agent) is never reached directly; this includes answering machines and other technological barriers, language barriers, hang-ups and refusals before or during the survey introduction (where an enrollee’s presence in not yet confirmed), busy phone numbers, etc.

In general, non-response is evaluated by examining a survey’s response rate (i.e., the proportion of completed interviews relative to the selected sample, minus the identified ineligible sample elements); response rates of less than 70-80 percent are frequently considered to imply that there is the potential for significant non-response bias in the results. For the 2008 SoE, the final response rate using AAPOR RR1 calculations was 35 percent for the overall sample. This is higher than in previous years, but low nonetheless. Therefore, the potential for non-response bias is considerable. The low response rate is primarily due to inability to reach and confirm the enrollee is eligible. Once reached, most enrollees cooperated. The overall cooperation rate was 65 percent.

Where:

i is a completed interview

p is a partial interview

r is a refusal

nc represents non-contacts ( i.e. answering machines, fax machines, callbacks, etc.)

o represents “other” (i.e. language barrier, no eligible proxy, etc.)

uo represents unknown others (i.e. no answer/ no previous contact, busy/no pervious contact, hang-ups, etc)

uh represents working telephone number but unknown if Veteran located there (i.e. no opportunity to screen for eligibility)

Design weights are used in the analysis of non-response.

Bias Analysis

With the exception of the controlled random sampling process, all stages described in the previous section have the potential to introduce bias into the survey estimates. The impact of coverage (or frame) bias and non-response bias are difficult to assess since data are not available for those who do not participate in the survey. Therefore, there is no way to compare the groups and draw inferences about the survey data. In lieu of survey responses for individuals who do not participate in the survey, we rely on secondary information available for both survey respondents and non-respondents. This information generally comes from the sampling frame and/or the population. In most cases, this information is limited, but in the case of VHA, there is considerable administrative data available about the population of enrollees. This information allows review of frame coverage and non-response biases for the survey with respect to enrollees’ use of various VHA services.

For the purpose of conducting this bias analysis, VHA provided Macro with a file based on administrative records that indicated if an enrollee had utilized any of the following services in the past year (the file did not indicate the frequency or amount for any of these benefits):

1. Received home health services

2. Inpatient treatment

1. Mental health or substance abuse

2. Non-mental health and non-substance abuse

3. Outpatient treatment

a. Mental health or substance abuse

b. Non-mental health and non-substance abuse

4. VHA pharmacy services

The following sections detail the bias analysis using this information.

1. Receiving Home Health Services

A small proportion of enrollees, 0.11 percent, receive home health services. The percentage is slightly higher for those not eligible for the frame (0.13 percent). This is a different pattern from previous years, where the frame-ineligible enrollees were less likely to have received home health care services. There was no evidence of a difference when comparing the percentage of enrollees receiving home health services with valid contact information versus those with invalid contact information (p-value=0.6431); there was no evidence of a difference for responding and non-responding enrollees either (p-value=0.8993).

Figure 2. Percentage of Enrollees Receiving Home Health Care

In Priority Group 4, the percentage receiving home health care is higher than the rest of the strata, 1.23 percent. This percentage increases to 1.50 percent for enrollees with valid contact information versus 0.87 percent for enrollees with invalid contact information (p-value<0.0001). There are eight strata (VISNs 7, 8, 11, 12, 16, 21, 22; Priority Group 4) where the enrollees with invalid contact information are significantly different (usually lower) than those with valid contact information (p < 0.1) and seven strata (VISNs 3, 5, 7, 11, 18, 22, 23) where respondents are significantly different (usually lower) than non-respondents. These differences are likely due to random variation in the very small percentage of enrollees receiving home health care services.

Table 4. Percentage of Enrollees Receiving Home Health Care

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		0.11	0.11	0.13	0.11	0.11	0.11	0.6431	0.12	0.11	0.8993
OEFOIF	N	0.12	0.11	0.13	0.12	0.12	0.11	0.7289	0.12	0.12	0.9664
	Y	0.00	0.00	0.00	0.00	0.00	0.00	0.0909	0.00	0.01	0.1554
VISN	1	0.12	0.12	0.12	0.13	0.12	0.15	0.7715	0.08	0.15	0.3475
	2	0.29	0.30	0.22	0.34	0.34	0.34	0.9164	0.49	0.25	0.1483
	3	0.10	0.11	0.07	0.10	0.09	0.12	0.7005	0.05	0.11	0.0440
	4	0.09	0.09	0.07	0.09	0.10	0.04	0.1861	0.11	0.09	0.7311
	5	0.10	0.10	0.04	0.08	0.08	0.07	0.8830	0.03	0.10	0.0087
	6	0.06	0.06	0.03	0.03	0.02	0.04	0.3969	0.01	0.03	0.2634
	7	0.11	0.09	0.18	0.13	0.17	0.06	0.0005	0.02	0.24	0.0000
	8	0.09	0.09	0.06	0.07	0.09	0.03	0.0028	0.14	0.06	0.1978
	9	0.09	0.09	0.08	0.10	0.08	0.14	0.2548	0.07	0.08	0.9104
	10	0.27	0.26	0.31	0.32	0.32	0.29	0.7666	0.47	0.24	0.3058
	11	0.17	0.18	0.14	0.21	0.13	0.41	0.0776	0.06	0.18	0.0659
	12	0.13	0.12	0.22	0.08	0.09	0.04	0.0210	0.06	0.11	0.3323
	15	0.06	0.06	0.03	0.03	0.03	0.02	0.5840	0.01	0.05	0.1858
	16	0.10	0.10	0.12	0.05	0.07	0.02	0.0131	0.08	0.05	0.5586
	17	0.09	0.09	0.09	0.13	0.15	0.11	0.5603	0.12	0.16	0.6885
	18	0.07	0.06	0.11	0.05	0.03	0.08	0.2255	0.01	0.05	0.0313
	19	0.07	0.08	0.01	0.09	0.08	0.11	0.5579	0.11	0.06	0.2183
	20	0.05	0.05	0.02	0.05	0.04	0.05	0.8145	0.06	0.03	0.4735
	21	0.17	0.17	0.20	0.15	0.11	0.24	0.0590	0.10	0.11	0.8680
	22	0.12	0.11	0.15	0.12	0.17	0.03	0.0216	0.45	0.03	0.0728
	23	0.09	0.09	0.10	0.19	0.19	0.20	0.9355	0.08	0.29	0.0546
Priority	1	0.19	0.19	0.23	0.18	0.17	0.20	0.4789	0.14	0.19	0.3363
	2	0.07	0.07	0.08	0.06	0.06	0.06	0.8619	0.05	0.07	0.4291
	3	0.08	0.07	0.08	0.06	0.05	0.07	0.4947	0.08	0.04	0.1556
	4	1.23	1.22	1.31	1.26	1.50	0.87	0.0000	1.31	1.60	0.1810
	5	0.09	0.09	0.10	0.09	0.09	0.09	0.9771	0.14	0.07	0.2211
	6	0.01	0.01	0.01	0.00	0.00	0.00	.	0.00	0.00	.
	7/8	0.03	0.03	0.03	0.05	0.05	0.04	0.5148	0.04	0.06	0.4795
Enrollee Type	POST	0.07	0.06	0.08	0.06	0.06	0.06	0.8834	0.07	0.06	0.9150
	PRE	0.21	0.22	0.20	0.22	0.23	0.20	0.2807	0.24	0.23	0.8211

Notes: 1. Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

2. N/A indicates no observed cases.

2. Inpatient Treatment

Overall, less than one percent of enrollees have been admitted to a hospital or medical facility for mental health or substance abuse reasons (0.82 percent). This percentage drops to 0.68 percent for those who have insufficient information to be frame-eligible. This is the opposite of what was observed in 2007, during which admitted enrollees were less likely to be frame-eligible.

Similar to last year, there is a considerable difference between enrollees with valid contact information and enrollees without valid contact information, 0.68 to 1.19 percent (p-value<0.0001). If all eligible enrollees responded to the survey, the percentage of enrollees admitted to a hospital or medical facility for mental health or substance abuse reasons would underestimate the true value by 0.14 percentage points (about 20 percent). However, this underestimation is further compounded by the fact that non-respondents were more likely to have received inpatient treatment (0.81 percent versus 0.48 percent, p-value<0.0001). Both of these effects results in an estimate that underestimates the true value by 71 percent.

In Priority Groups 1 and 4 (the two groups with the highest population percentage of enrollees admitted to a hospital or medical facility for mental health or substance abuse reasons), the percentages, as measured from the responding enrollees (1.16 and 3.40 percent) underestimate the population percentages of 2.07 and 5.67 percent. This was due both to differences in enrollees with valid versus invalid contact information and differences between those who responded to the survey or did not. This is very similar to the pattern in 2007.

For all VISNs, the percentage of enrollees who have been admitted to a hospital or medical facility for mental health or substance abuse is higher for enrollees with ineligible contact information (most are significantly different). Non-respondents have a higher percentage than do respondents for 18 VISNs, eight of which are significantly different. This pattern is consistent with 2007. The underestimation at these stages is considerable and results in very biased results. For instance, in VISNs 16 and 22, the final estimate of enrollees admitted for mental health or substance abuse is 0.17 and 0.15 percent respectively. However, the actual values for these VISNs are 0.64 and 0.80 percent.

For enrollees admitted to a hospital or medical facility for reasons unrelated to mental health or substance abuse, the final estimate is only slightly higher than the actual percentage, 4.63 versus 4.34. However, the percentage for respondents is significantly higher than for non-respondents, 4.63 to 4.01 (P-value < 0.0001).

When comparing enrollees with and without eligible contact information, five VISNs (3, 6, 9, 11, 21) are significantly different (P < 0.1). The pattern of the differences is inconsistent. In Priority Group 1, enrollees with ineligible contact information were admitted more often than enrollees with valid contact information in 2007 and again in 2008. In 2007, nearly 10 percent of enrollees with ineligible contact information were admitted versus 8.51 for enrollees with eligible information. In 2008, the percentages were 9.71 and 8.43 percent.

When comparing the respondents and non-respondents, there are seven significant differences for the VISNs, three for priority groups and both enrollee types. This is slightly better than 2007 which had significant differences in nine VISNs and all priority groups. As with 2007, the significant differences result in overestimates of the population percentages.

Figure 3. Percentage of Enrollees Receiving Inpatient Treatment

(a) For Mental Health or Substance Abuse

(b) Not for Mental Health nor Substance Abuse

Table 5. Percentage of Enrollees Receiving Inpatient Treatment

(a) For Mental Health or Substance Abuse

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		0.82	0.84	0.68	0.83	0.68	1.19	0.0000	0.48	0.81	0.0000
OEFOIF	N	0.81	0.83	0.67	0.84	0.68	1.21	0.0000	0.47	0.80	0.0000
	Y	0.99	0.99	0.98	0.82	0.81	0.85	0.6478	0.65	0.85	0.1527
VISN	1	1.01	1.03	0.67	1.10	0.89	1.68	0.0007	0.51	1.11	0.0018
	2	0.71	0.74	0.47	0.67	0.62	0.79	0.2117	0.52	0.67	0.3520
	3	0.65	0.68	0.32	0.51	0.46	0.60	0.1588	0.19	0.58	0.0002
	4	0.86	0.88	0.70	1.08	0.96	1.48	0.3690	0.53	1.21	0.0056
	5	1.07	1.10	0.68	1.10	1.09	1.12	0.8845	0.91	1.17	0.3631
	6	0.98	1.01	0.65	1.29	1.06	1.81	0.0191	0.42	1.41	0.0000
	7	0.81	0.83	0.76	0.70	0.59	0.93	0.0809	0.48	0.64	0.4197
	8	0.62	0.63	0.46	0.58	0.45	0.90	0.0100	0.37	0.49	0.2707
	9	0.98	1.01	0.68	1.27	1.12	1.65	0.0646	1.23	1.05	0.7231
	10	1.04	1.02	1.18	1.24	1.04	1.78	0.0319	0.29	1.49	0.0000
	11	0.88	0.85	1.09	0.93	0.72	1.47	0.0051	0.57	0.82	0.1431
	12	0.90	0.89	0.92	0.70	0.62	0.89	0.0402	0.39	0.78	0.0316
	15	0.92	0.94	0.63	0.95	0.80	1.32	0.0619	0.97	0.67	0.3339
	16	0.80	0.82	0.61	0.62	0.41	1.07	0.0014	0.17	0.56	0.0002
	17	0.90	0.91	0.85	0.94	0.77	1.27	0.0367	0.57	0.88	0.2090
	18	0.74	0.77	0.64	0.85	0.41	1.69	0.0046	0.49	0.36	0.3294
	19	0.80	0.83	0.51	0.70	0.61	0.89	0.0491	0.48	0.71	0.1663
	20	0.86	0.89	0.64	0.74	0.63	0.96	0.0708	0.52	0.70	0.3122
	21	0.68	0.69	0.63	0.73	0.57	1.08	0.0976	0.40	0.67	0.1124
	22	0.60	0.63	0.46	0.71	0.51	1.11	0.0020	0.15	0.69	0.0000
	23	0.64	0.63	0.69	0.47	0.45	0.53	0.5034	0.31	0.56	0.1139
Priority	1	2.07	2.11	1.77	1.87	1.69	2.39	0.0001	1.16	2.03	0.0000
	2	0.68	0.69	0.55	0.64	0.57	0.82	0.0103	0.48	0.62	0.1474
	3	0.51	0.53	0.37	0.53	0.42	0.78	0.0004	0.29	0.49	0.0204
	4	5.67	5.85	4.44	5.85	4.79	7.66	0.0000	3.40	5.59	0.0000
	5	0.77	0.80	0.60	0.90	0.70	1.31	0.0001	0.48	0.81	0.0212
	6	0.21	0.21	0.19	0.16	0.17	0.14	0.5449	0.21	0.15	0.6057
	7/8	0.12	0.12	0.10	0.13	0.12	0.17	0.2726	0.07	0.15	0.0221
Enrollee Type	POST	0.54	0.55	0.46	0.59	0.48	0.86	0.0000	0.34	0.56	0.0003
	PRE	1.44	1.51	1.05	1.39	1.17	1.86	0.0000	0.81	1.37	0.0000

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Table 5. Percentage of Enrollees Receiving Inpatient Treatment

(b) Not for Mental Health or Substance Abuse

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		4.34	4.35	4.29	4.26	4.24	4.31	0.5151	4.63	4.01	0.0000
OEFOIF	N	4.50	4.52	4.37	4.43	4.40	4.51	0.3230	4.73	4.19	0.0004
	Y	1.09	1.08	1.22	0.92	0.88	0.99	0.2935	0.80	0.91	0.4679
VISN	1	3.69	3.75	2.89	3.67	3.63	3.78	0.7094	3.42	3.75	0.4682
	2	3.71	3.84	2.86	3.82	3.98	3.42	0.1250	3.57	4.20	0.2136
	3	3.49	3.64	2.17	3.95	3.40	5.14	0.0003	3.08	3.53	0.3024
	4	3.40	3.29	4.99	3.14	3.00	3.57	0.1808	2.63	3.22	0.1650
	5	4.50	4.64	2.65	4.36	4.46	4.11	0.3888	4.34	4.52	0.7304
	6	4.17	4.26	3.22	4.54	4.81	3.94	0.0705	5.15	4.63	0.4366
	7	3.84	3.42	5.04	3.64	3.66	3.59	0.8651	3.58	3.70	0.8328
	8	4.59	4.66	3.73	4.37	4.54	3.95	0.1867	5.20	4.21	0.2008
	9	5.54	5.65	4.53	5.23	5.48	4.62	0.0911	6.18	5.07	0.0865
	10	4.24	4.03	5.54	3.93	3.91	4.01	0.8367	3.25	4.31	0.1123
	11	3.83	3.90	3.30	3.41	3.17	4.02	0.0731	3.03	3.25	0.5946
	12	4.58	4.30	6.07	4.43	4.61	3.95	0.1120	5.62	3.93	0.0275
	15	4.81	4.92	3.45	4.81	4.68	5.14	0.4254	4.45	4.86	0.5067
	16	4.91	4.99	4.21	5.08	4.99	5.26	0.6361	6.26	4.19	0.0042
	17	4.80	4.85	4.22	4.53	4.66	4.29	0.3909	4.85	4.55	0.6411
	18	4.93	4.77	5.40	5.12	5.27	4.83	0.3345	5.38	5.20	0.8110
	19	4.48	4.45	4.81	4.18	4.20	4.11	0.8368	5.96	2.91	0.0000
	20	4.45	4.63	2.90	4.23	4.41	3.87	0.1730	4.77	4.17	0.2792
	21	4.30	4.38	3.52	4.05	3.68	4.90	0.0256	4.49	3.21	0.0326
	22	4.21	4.25	4.05	3.94	3.83	4.15	0.4331	4.81	3.35	0.0777
	23	4.31	4.28	4.66	4.63	4.47	5.10	0.2632	5.29	3.79	0.0219
Priority	1	8.72	8.69	8.99	8.76	8.43	9.71	0.0008	8.47	8.40	0.8612
	2	3.46	3.47	3.33	3.47	3.33	3.77	0.0520	3.47	3.25	0.4330
	3	2.92	2.97	2.57	2.98	3.09	2.77	0.1319	3.38	2.91	0.1072
	4	15.90	16.06	14.86	16.00	16.10	15.83	0.5049	16.10	16.10	0.9982
	5	5.42	5.48	5.07	5.21	5.41	4.81	0.0241	6.56	4.82	0.0001
	6	1.26	1.27	1.13	1.03	1.09	0.90	0.3562	1.52	0.89	0.0743
	7/8	1.58	1.59	1.54	1.54	1.57	1.44	0.3303	1.78	1.43	0.0184
Enrollee Type	POST	2.94	2.95	2.92	2.83	2.81	2.87	0.6585	3.14	2.61	0.0035
	PRE	7.51	7.67	6.56	7.52	7.62	7.29	0.0510	8.30	7.24	0.0000

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

3. Outpatient Treatment

As in 2007, there is evidence of extreme systematic bias for outpatient treatment unrelated to mental health or substance abuse. Overall, the population percentage is 56.76 percent and is very similar for frame eligible enrollees, 57.91 percent. A one-to-two percentage point difference is fairly consistent across the strata. The one exception is VISN 7, where the frame percentage is 3.5 points lower than the population. For enrollees with eligible contact information, the percentage then climbs to 62.08 percent, significantly higher than enrollees without contact information, 45.24 percent (p-value<0.0001). The percentage climbs again to 72.17 percent when measured for the responding enrollees, much higher than the non-responding enrollees at 56.15 percent (p-value<0.0001). This pattern of overestimation is consistent across VISNs, priority groups, enrollee types, and OEF/OIF status.

Overall, 4.22 percent of enrollees receive outpatient treatment for mental health or substance abuse, and this percentage is similar when restricted to frame-eligible enrollees (4.28 percent) and enrollees with eligible contact information (4.12 percent). The percentage for enrollees without eligible contact information is significantly higher at 4.34 (0.0304), but the minor difference does not seem to be a significant contributor to bias. There is no evidence of a significant difference between respondents and non-respondents (p-value=0.1845).

The number of VISNs where there are significant differences between enrollees with eligible contact information and ineligible contact information is higher than in 2007, nine versus four. As with the inpatient mental health and substance abuse treatment, when a significant difference exists (nine VISNs, two priority groups, and post-enrollee type), the percentage for enrollees with invalid contact information is generally higher than those with valid information. There are a high number of VISNs where the percentage of enrollees receiving outpatient care for mental health and substance abuse is higher for non-respondents than respondents. There were only three in 2007.

Priority Groups 1 and 4 have the highest percentage of enrollees receiving outpatient care for mental health or substance abuse (11.94 and 9.34 percent). In Priority Group 1, the percentage drops only slightly to 11.83 percent for enrollees with valid contact information, and then down to 11.24 percent for responding enrollees--significantly different from the non-respondents, 12.21 percent (p-value=0.0340). The same pattern holds for Priority Group 4—9.17 percent for valid contacts, but a further drop to 8.07 for responding enrollees, which is significantly different from the 9.81 percent for non-respondents (p-value<0.0001). This is similar to the pattern observed in 2007, but the bias in the end result—a 1.3 point underestimate is less for 2008 than for 2007—a 2.5 point underestimate.

Figure 4. Percentage of Enrollees Receiving Outpatient Treatment

(a) For Mental Health or Substance Abuse

(b) Not for Mental Health nor Substance Abuse

Table 6. Percentage of Enrollees Receiving Outpatient Treatment

(a) For Mental Health or Substance Abuse

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		4.22	4.28	3.72	4.19	4.12	4.34	0.0304	4.01	4.18	0.1845
OEFOIF	N	4.00	4.05	3.59	4.01	3.94	4.16	0.0496	3.90	3.97	0.5664
	Y	8.61	8.65	8.06	7.74	7.89	7.47	0.1179	8.43	7.74	0.1398
VISN	1	5.03	5.14	3.64	5.06	4.83	5.71	0.0628	3.97	5.32	0.0036
	2	4.32	4.49	3.14	4.14	4.21	3.98	0.5725	3.44	4.64	0.0219
	3	4.01	4.22	2.17	4.33	4.51	3.93	0.1002	4.70	4.43	0.6233
	4	4.09	4.12	3.71	4.21	4.42	3.48	0.0169	5.09	4.04	0.0507
	5	4.22	4.32	2.74	3.90	3.74	4.27	0.1597	3.42	3.88	0.3229
	6	4.31	4.43	2.96	4.93	5.26	4.17	0.0168	5.46	5.16	0.7114
	7	4.39	4.16	5.06	4.64	4.94	3.99	0.0334	5.03	4.89	0.8792
	8	3.43	3.51	2.58	3.05	2.87	3.45	0.1703	2.92	2.85	0.8651
	9	4.37	4.49	3.24	4.63	4.29	5.47	0.0710	4.64	4.09	0.3284
	10	5.13	5.06	5.51	5.06	5.10	4.96	0.7834	4.20	5.64	0.0322
	11	4.19	4.27	3.53	4.10	3.70	5.12	0.0149	2.86	4.23	0.0032
	12	4.24	4.19	4.51	3.65	3.57	3.86	0.4531	3.59	3.55	0.9425
	15	4.28	4.36	3.31	4.00	4.01	3.99	0.9663	3.97	4.04	0.8898
	16	4.37	4.46	3.54	4.10	3.91	4.53	0.2009	3.38	4.24	0.0736
	17	4.29	4.35	3.52	4.40	4.29	4.61	0.4616	5.05	3.87	0.0669
	18	3.56	3.66	3.28	3.43	3.17	3.92	0.0411	2.89	3.36	0.2940
	19	4.43	4.55	3.39	4.12	3.73	5.05	0.0037	3.29	4.05	0.0899
	20	4.39	4.57	2.85	4.46	4.63	4.13	0.1982	4.60	4.64	0.9384
	21	4.39	4.43	4.00	4.26	4.36	4.04	0.4377	4.78	4.12	0.2850
	22	4.11	4.20	3.74	4.12	3.97	4.41	0.2913	4.11	3.90	0.7117
	23	3.75	3.77	3.52	3.85	3.40	5.13	0.0012	3.21	3.55	0.4592
Priority	1	11.94	12.03	11.24	11.94	11.83	12.26	0.3178	11.24	12.21	0.0340
	2	5.67	5.82	4.51	5.50	5.44	5.64	0.4644	5.85	5.18	0.0808
	3	3.04	3.15	2.28	2.89	2.98	2.71	0.1568	2.83	3.07	0.3700
	4	9.34	9.53	8.06	9.43	9.17	9.86	0.0359	8.07	9.81	0.0002
	5	3.50	3.57	3.04	3.59	3.37	4.01	0.0062	3.35	3.38	0.9329
	6	3.34	3.42	2.57	3.47	3.56	3.29	0.3711	3.22	3.71	0.3768
	7/8	1.36	1.38	1.17	1.31	1.29	1.38	0.4660	1.29	1.29	0.9936
Enrollee Type	POST	3.46	3.52	2.99	3.47	3.31	3.84	0.0001	3.23	3.36	0.3815
	PRE	5.93	6.10	4.93	5.82	6.03	5.39	0.0000	5.95	6.07	0.5596

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Table 6. Percentage of Enrollees Receiving Outpatient Treatment

(b) Not for Mental Health or Substance Abuse

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		56.76	57.91	47.98	57.05	62.08	45.24	0.0000	72.17	56.15	0.0000
OEFOIF	N	57.37	58.60	48.21	57.79	62.92	45.62	0.0000	72.79	56.92	0.0000
	Y	44.60	44.96	39.60	42.25	44.09	38.86	0.0000	48.52	42.89	0.0000
VISN	1	58.68	60.12	38.87	60.20	65.10	46.65	0.0000	75.08	59.48	0.0000
	2	51.35	54.14	32.53	51.67	56.58	39.19	0.0000	68.72	49.77	0.0000
	3	43.41	45.59	23.59	43.24	46.97	35.09	0.0000	58.89	42.07	0.0000
	4	57.93	58.61	48.24	57.70	62.09	43.27	0.0000	72.11	56.22	0.0000
	5	49.38	51.05	26.09	50.55	55.92	38.27	0.0000	66.82	51.20	0.0000
	6	56.61	58.24	38.70	56.16	60.19	47.06	0.0000	69.00	55.44	0.0000
	7	54.57	50.36	66.43	54.34	58.21	45.85	0.0000	65.71	54.67	0.0000
	8	63.52	65.20	44.72	63.92	70.26	48.94	0.0000	79.57	65.53	0.0000
	9	60.15	61.85	43.70	60.05	65.37	47.00	0.0000	73.84	60.41	0.0000
	10	55.40	54.69	59.73	56.75	62.18	42.16	0.0000	73.62	55.22	0.0000
	11	57.77	59.90	41.07	58.36	64.14	43.73	0.0000	73.52	58.19	0.0000
	12	58.58	58.18	60.72	59.60	64.96	45.45	0.0000	76.25	57.22	0.0000
	15	61.81	63.51	40.70	62.31	67.53	49.57	0.0000	76.19	60.59	0.0000
	16	59.59	61.06	46.12	59.52	64.46	48.76	0.0000	74.74	57.96	0.0000
	17	57.29	58.41	41.93	57.17	62.34	47.37	0.0000	71.37	57.38	0.0000
	18	58.44	59.33	55.84	58.08	62.90	48.84	0.0000	72.54	56.58	0.0000
	19	55.90	57.18	44.95	56.47	61.48	44.69	0.0000	70.32	54.95	0.0000
	20	53.51	55.59	34.99	54.02	59.04	43.92	0.0000	69.87	51.56	0.0000
	21	52.82	54.19	39.64	53.61	57.71	44.18	0.0000	68.53	51.52	0.0000
	22	49.85	50.77	46.09	49.53	55.01	38.99	0.0000	65.59	49.78	0.0000
	23	63.02	64.51	42.38	63.88	68.68	50.29	0.0000	77.04	61.79	0.0000
Priority	1	71.64	72.13	67.89	71.63	73.65	65.82	0.0000	76.93	71.54	0.0000
	2	59.06	60.41	48.44	59.76	63.27	51.82	0.0000	69.22	59.63	0.0000
	3	52.96	55.04	38.36	53.13	58.51	41.69	0.0000	69.60	52.04	0.0000
	4	66.51	67.32	61.08	67.61	72.94	58.55	0.0000	80.27	68.69	0.0000
	5	55.65	56.68	48.48	56.30	62.57	43.94	0.0000	74.09	56.72	0.0000
	6	40.16	41.19	30.38	39.72	43.55	31.17	0.0000	50.69	40.34	0.0000
	7/8	53.98	55.21	43.74	54.27	59.63	39.05	0.0000	72.11	51.35	0.0000
Enrollee Type	POST	54.12	55.18	45.04	54.38	59.28	42.41	0.0000	69.92	52.90	0.0000
	PRE	62.76	64.40	52.86	63.10	68.71	51.11	0.0000	77.71	63.64	0.0000

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

4. VHA Pharmacy Services

The percentage of enrollees receiving the VHA pharmacy service follows very closely to the observed patterns for outpatient treatment unrelated to mental health or substance abuse. The percentage of enrollees receiving the service is 55.12 percent and increases to 56.23 percent for frame-eligible enrollees. There is a minor increase to 59.93 percent when limiting to sampled enrollees with valid contact information and a significant increase to 69.96 percent when measuring responding enrollees. This pattern is consistent across all strata—a slight increase in the percentage from population to frame-eligible and significant increases in the percentage for enrollees with valid contact information and responding enrollees. All comparisons between enrollees with valid information to those without are significant. Further, all comparisons of responding to non-responding enrollees are significant.

Figure 5. Percentage of Enrollees Receiving Prescription Drug Services

Table 7. Percentage of Enrollees Receiving Prescription Drug Services

			In Frame		Samp-led	Eligible			Respond
		Popul-ation	Yes	No	Yes	Yes	No	P-value	Yes	No	P-value
Total		55.12	56.23	46.72	55.24	59.93	44.23	0.0000	69.96	54.03	0.0000
OEFOIF	N	56.12	57.33	47.13	56.37	61.14	45.03	0.0000	70.82	55.27	0.0000
	Y	35.22	35.44	32.25	32.70	33.69	30.87	0.0000	37.01	32.79	0.0000
VISN	1	57.08	58.50	37.50	58.70	63.01	46.80	0.0000	72.99	57.38	0.0000
	2	50.19	52.93	31.72	50.80	55.68	38.42	0.0000	66.77	49.45	0.0000
	3	41.75	44.01	21.29	42.29	45.90	34.43	0.0000	56.29	41.62	0.0000
	4	55.74	56.37	46.79	55.87	60.37	41.08	0.0000	71.27	53.99	0.0000
	5	46.57	48.20	23.82	47.11	51.87	36.22	0.0000	62.45	47.30	0.0000
	6	56.08	57.73	38.06	56.27	60.18	47.44	0.0000	69.08	55.38	0.0000
	7	53.88	49.07	67.43	52.77	56.31	44.99	0.0000	63.10	53.12	0.0000
	8	60.45	62.05	42.47	59.68	65.60	45.67	0.0000	75.71	60.47	0.0000
	9	59.47	61.20	42.63	59.27	63.93	47.81	0.0000	71.41	59.56	0.0000
	10	55.52	54.61	61.07	56.65	61.35	44.02	0.0000	71.06	55.45	0.0000
	11	56.88	59.03	40.08	56.39	61.51	43.44	0.0000	70.39	55.88	0.0000
	12	57.36	56.71	60.80	56.95	61.62	44.61	0.0000	73.75	53.31	0.0000
	15	61.09	62.82	39.74	61.15	66.19	48.83	0.0000	73.76	60.13	0.0000
	16	59.45	60.99	45.31	59.45	63.84	49.88	0.0000	73.85	57.52	0.0000
	17	55.28	56.36	40.36	55.30	60.46	45.54	0.0000	69.91	55.27	0.0000
	18	55.57	56.64	52.43	55.34	59.57	47.23	0.0000	69.69	52.94	0.0000
	19	54.13	55.35	43.61	54.34	59.09	43.17	0.0000	68.42	52.19	0.0000
	20	51.90	54.01	33.16	52.47	57.66	42.02	0.0000	68.66	50.06	0.0000
	21	49.99	51.21	38.26	50.44	54.41	41.32	0.0000	66.70	47.37	0.0000
	22	46.44	47.46	42.30	46.56	51.87	36.34	0.0000	62.04	46.85	0.0000
	23	60.18	61.61	40.48	61.26	65.43	49.41	0.0000	73.98	58.40	0.0000
Priority	1	76.90	77.35	73.44	77.16	78.80	72.44	0.0000	81.75	76.91	0.0000
	2	54.27	55.53	44.35	54.13	56.81	48.05	0.0000	62.75	53.18	0.0000
	3	46.32	48.14	33.56	46.28	50.70	36.86	0.0000	60.44	45.02	0.0000
	4	72.73	73.67	66.47	73.97	79.31	64.88	0.0000	85.82	75.53	0.0000
	5	55.37	56.43	47.98	55.89	61.99	43.87	0.0000	74.32	55.72	0.0000
	6	32.39	33.15	25.17	30.95	34.28	23.50	0.0000	41.54	31.02	0.0000
	7/8	51.16	52.40	40.74	51.35	56.52	36.66	0.0000	68.80	48.36	0.0000
Enrollee Type	POST	50.95	51.92	42.63	51.00	55.50	40.01	0.0000	66.16	49.11	0.0000
	PRE	64.62	66.47	53.54	64.86	70.41	52.97	0.0000	79.32	65.39	0.0000

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Survey Weighting

The weighting methodology for the 2005 SoE used a base weight as the inverse of the probability of selection in each stratum (OEF/OIF group, enrollee type, VISN, and priority group), with a non-response adjustment by age group (under 45, 45-64, and 65+). A recommendation stemming from the 2005 survey analysis was to add utilization statistics to the non-response adjustment. This adjustment was implemented for 2007 and continued for 2008. The details of the non-response modeling and weighting adjustment are presented in a separate report: Veterans Health Care System Survey Methodology Report 2008.

The preceding bias analysis is based on weighted data that accounts for the differential sampling probabilities for each stratum and does not adjust for non-response. Macro also performed the bias analysis using the weights used for the SoE to analyze if the non-response adjustment reduces the biases observed for the health estimates. This non-response adjustment was successful in reducing bias for the 2007 SoE and continues to be for the 2008 SoE. Overall, the non-response weighting tends to reduce bias in measuring the health estimates—five of six estimates are closer to the population. The significant biases for outpatient treatment unrelated to mental health or substance abuse and pharmacy services are eliminated.

Table 8. Survey Estimates and Bias for Weighted and Weighted and Adjusted Data

		Base weight only				Base weight and non-response adjustment
	Popul-ation	Est	Bias	L95	U95	Est	Bias	L95	U95
1. Home Health care	0.11	0.12	0.00	-0.03	0.04	0.13	0.02	-0.02	0.06
2. Inpatient treatment
(a) Related to MH/SA	0.82	0.48	-0.34	-0.41	-0.27	0.85	0.04	-0.08	0.15
(b) Unrelated to MH/ SA	4.34	4.62	0.25	0.04	0.54	4.25	-0.09	-0.33	0.14
3. Outpatient treatment
(a) Related to MH/SA	4.22	4.01	-0.21	-0.41	0.00	4.28	0.06	-0.18	0.30
(b) Unrelated to MH/ SA	56.76	72.17	15.41	14.87	15.96	56.91	0.16	-0.55	0.86
4. VHA Pharmacy service	55.12	69.96	14.84	14.29	15.40	55.31	0.19	-0.51	0.88

The new weighting procedure has eliminated the bias for each of the six health measures. This is expected since these health measures contribute to the propensity score estimates that are used to make the adjustment. The weighting adjustment will succeed in reducing bias when survey responses are correlated with the probability to respond and with one of the six health measures in the model. This was demonstrated in the 2007 analysis by examining the weighting adjustment impact for self-report utilization.

Discussion

There are some noticeable differences in the 2008 survey when compared to the 2007 and 2005 results. The database of enrollees seemed to have improved telephone contact information. The percentage of invalid phone numbers for 2008 was roughly half the ineligible rate observed in 2007 and 2005. This greatly improves frame coverage from about 75 percent of enrollees to 88 percent.

While the increased frame coverage is a welcome improvement, the stages that have historically introduced bias into the estimates are whether the contact information was valid or not and whether the enrollees responded or not. There continues to be a high number of telephone numbers that are invalid and a high number of non-responding enrollees. There are noticeable differences between enrollees with valid and invalid contact information as well as between responding and non-responding enrollees. As with 2005 and 2007, these differences are producing biased results in terms of VHA utilization as measured by administrative records. Survey items (satisfaction, awareness perceptions, etc.) that are correlated to utilization will also be biased unless appropriate corrections are made. The non-response weighting introduced in 2007 and continued for 2008 mitigates the bias in the utilization statistics. It follows that bias in the survey items that are correlated to utilization are also mitigated.

2008 Design Enhancements

Pre-notification letters and increased call attempts—two recommendations from the 2007 analysis—both had a positive impact on response. Increasing the number of call attempts from six to seven improves response by about two percentage points. While there is no causal evidence for improvements due to the pre-notification letters in 2008, the 2007 survey demonstrates that the letters increase response. Further, the response rate for 2008 was comparable to the 2007 response rate for the sample of enrollees who were sent the pre-notification letters.

A recommendation from the 2005 analysis report was to use the address information to identify a telephone number by running the Veteran's name and address against a reverse look-up database. VHA experimented with this recommendation as part of the 2008 survey. This database matching had a very positive result. The matching results included updated telephone numbers for 30 percent of the enrollees who had a phone number listed with VHA. Further, the matching returned a telephone number for 50 percent of the enrollees without a phone number listed with VHA. The success of this match was observed in the percentage of enrollees with valid contact information. In the non-matched sample, 33 percent of the telephone numbers were not valid. In the matched sample, only 25 percent were not valid.

Macro recommends full adoption of these operational changes for the next survey. Further, Macro recommends continued use of pre-notification letters, maintaining the call attempts at a maximum of seven and continuing to adjust the data for non-response using the propensity score model.

		Mailings					Undeliverable as addressed
		Sep 22	Sep 26	Oct 10	Nov 4/5	Total
Total		100292	44322	45621	9270	199505	8.9%
OEFOIF	N	89338	39460	40576	3172	172546	8.6%
	Y	10954	4862	5045	6098	26959	10.1%
VISN	1	4787	2144	2138	293	9362	8.5%
	2	4783	2082	2186	293	9344	8.5%
	3	4789	2092	2166	376	9423	7.7%
	4	4779	2074	2163	236	9252	6.7%
	5	4766	2133	2137	533	9569	9.8%
	6	4759	2112	2194	286	9351	8.1%
	7	4776	2106	2166	1063	10111	8.9%
	8	4784	2116	2177	362	9439	7.5%
	9	4782	2105	2174	238	9299	7.7%
	10	4790	2112	2177	291	9370	9.0%
	11	4758	2124	2135	347	9364	8.4%
	12	4762	2129	2183	447	9521	8.5%
	15	4783	2114	2160	356	9413	7.9%
	16	4774	2076	2201	373	9424	8.7%
	17	4772	2151	2153	471	9547	9.6%
	18	4758	2144	2173	647	9722	11.1%
	19	4771	2136	2150	275	9332	9.6%
	20	4794	2116	2201	348	9459	10.7%
	21	4784	2091	2182	514	9571	9.6%
	22	4770	2100	2173	1235	10278	11.1%
	23	4771	2065	2232	286	9354	6.7%
Priority	1	14664	6513	6648	468	28293	5.6%
	2	12520	5577	5701	525	24323	7.2%
	3	12703	5640	5814	1059	25216	9.0%
	4	12560	5406	5655	927	24548	12.2%
	5	16715	7400	7645	2329	34089	12.2%
	6	6093	2711	2784	3164	14752	9.6%
	7/8	25037	11075	11374	798	48284	7.0%
Enrollee Type	POST	50227	21716	22874	6958	101775	8.0%
	PRE	50065	22606	22747	2312	97730	9.7%
Unmatched sample		100292	.	28529	9270	138091	8.7%
Matched sample		.	44322	17092	.	61414	8.9%

100292	44322	45621	9270	199505	17745
89338	39460	40576	3172	172546	14827
10954	4862	5045	6098	26959	2723
4787	2144	2138	293	9362	799
4783	2082	2186	293	9344	793
4789	2092	2166	376	9423	728
4779	2074	2163	236	9252	618
4766	2133	2137	533	9569	936
4759	2112	2194	286	9351	757
4776	2106	2166	1063	10111	896
4784	2116	2177	362	9439	707
4782	2105	2174	238	9299	719
4790	2112	2177	291	9370	846
4758	2124	2135	347	9364	788
4762	2129	2183	447	9521	810
4783	2114	2160	356	9413	741
4774	2076	2201	373	9424	818
4772	2151	2153	471	9547	920
4758	2144	2173	647	9722	1082
4771	2136	2150	275	9332	895
4794	2116	2201	348	9459	1011
4784	2091	2182	514	9571	917
4770	2100	2173	1235	10278	1142
4771	2065	2232	286	9354	627
14664	6513	6648	468	28293	1588
12520	5577	5701	525	24323	1754
12703	5640	5814	1059	25216	2263
12560	5406	5655	927	24548	2995
16715	7400	7645	2329	34089	4142
6093	2711	2784	3164	14752	1414
25037	11075	11374	798	48284	3394
50227	21716	22874	6958	101775	8105
50065	22606	22747	2312	97730	9445
100292	.	28529	9270	138091	12077
.	44322	17092	.	61414	5473

Analysis of Methodological Experiments and Non-response Bias for the 30 2008 Annual Survey of Veteran Enrollees Health and Reliance on VA