Download:
pdf |
pdfSupplementary Analysis and Technical Assistance for the 2007
Annual Survey of Veteran Enrollees Health and Reliance on VA
Final Report
Submitted to the Veterans Healthcare Administration
By Macro International Inc.
Date: Feb 14, 2008
In 2006, Macro International Inc. (Macro) reviewed the research design for the Veterans
Healthcare Administration’s (VHA) 2005 Survey of Enrollees. The review examined the survey
process and potential biases resulting from missing or outdated contact information and survey
non-response—including the inability to make contacts and respondent refusals. This resulting
report, presented here and submitted to the Office of Management and Budget (OMB), made
several actionable recommendations for improving the research design. During discussions
about this report, VHA, Macro and OMB developed a design improvement plan that includes
long and short-term goals.
This report summarizes many of the enhancements suggested for the 2007 Survey of Enrollees.
First, we follow-up on and extend the original analysis. The format for this analysis is largely
the same as it was for the 2005 Survey of Enrollees. We also evaluate potential biases caused by
various steps in the survey process and make general summary observations based on the 2007
results. When relevant, we compare the 2007 results to our 2005 observations. The 2007
research includes two experiments to improve response rates—sending a pre-notification letter
and increasing the number of call attempts. We describe these two experiments and examine
their impact on response.
Finally, in 2006 Macro recommended that the survey weighting include a non-response
adjustment for utilization of VHA services. We evaluate this new weighting adjustment and its
success at mitigating the impact of non-response.
Discussion
The results from 2007 are largely consistent with the 2005 results—the sample of respondents
produced biased estimates with respect to utilization as measured by VHA administrative data.
Since one of the objectives of the Survey of Enrollees is to measure healthcare utilization (selfreported), it is very likely that the survey estimates are overestimating true utilization for the
population of enrollees. There is no way to confirm which survey estimates are biased or whether
other survey concepts (awareness, perceptions, etc.) are affected by a sample of respondents that
is overrepresented by enrollees who tend to use VHA healthcare services.
The purpose of this report and its 2006 predecessor is to quantify the impact of each operational
stage in the survey process. Breaking the process into stages and evaluating the probability that
1
an enrollee passes through each stage allows us to develop actionable strategies for mitigating
the effects of bias at each stage. These strategies may involve operational changes to the survey
to reduce the likelihood of biased estimates or post-survey adjustments to correct for observed
biases.
The three stages are: frame eligibility, valid contact information, and survey non-response. In
general, each stage in the process introduces bias.
Frame eligibility
Although frame eligibility, with about 75 percent of the population eligible for the frame, is the
stage that contributes least to the bias, there are utilization differences between eligible and noneligible enrollees. For each utilization statistic, the frame estimate is within 10 percent of the
true population value. This is consistent with the 2005 survey.
Valid Contact Information
As with 2005, about one-third of the sampled enrollees in 2007 had invalid contact information.
This stage seems to be a major contributor to the bias observed in the six health statistics.
Enrollees with valid contact information differed significantly from enrollees with invalid
contact information on all six health measures. Compounding the invalid contact information
(frame error) is the 27 percent that are frame ineligible, leaving about 48 percent of the enrollee
population for conducting the survey.
The percentage of enrollees without valid contact information who received inpatient treatment
for substance abuse or mental health is twice as high as the percentage for the enrollees with
valid contact information. Of the other statistics, the minimum percent difference between
eligible and ineligible groups is six percent and the maximum is over 35 percent.
Survey Response
Response rates for the 2007 Enrollee Survey are lower than the 2005 response rates, yet the
difference between the respondents and non-respondents is very similar to 2005. The only
utilization statistic where the difference in the groups is much larger than in 2005 is for enrollees
who received inpatient care for substance abuse or mental health, where 0.46 percent of
respondents received care versus 0.67 percent for non-respondents.
2007 Design Enhancements
In the 2006 report, we recommended altering the survey weighting to include measures of
healthcare utilization. Including these variables in the weighting will mitigate the potential for
bias if the survey measures are correlated with response likelihood and healthcare utilization.
We also concluded that frame accuracy ought to be the top priority for improving the survey
operations. Much of the population is excluded from participation due to lack of, or inaccurate,
contact information. Since accurate contact information seems to correlate with utilization, the
frame (contactable enrollees) produces biased utilization estimates right from the outset. We
offered a variety of suggestions, most leveraging the address information:
•
The most radical use of addresses is moving from telephone survey mode to a mail
survey. There are many considerations surrounding this shift in methodology—such as
2
•
•
survey administration (self-administered with pen and paper versus an intervieweradministered, computer-assisted instrument), field time, data entry, etc. However, if the
address information is better than the telephone information, then the risk of coverage
bias will be reduced.
With addresses, we can send a pre-notification letter, stating the survey's purpose and
importance, indicating the sponsor (VHA), and introducing the research company
conducting the survey. The letter elicits participation by communicating that the survey is
a critical tool for VHA to effectively administer benefits. Further, it legitimizes the
survey and provides reassurance that the information provided will be kept completely
confidential. In an era of information thievery, an official letter will most likely improve
response among enrollees.
One further possibility would be to use the address information to identify a telephone
number by running the veteran's name and address against a reverse look-up database.
Security restrictions may impede this matching considering that this service would be
performed by an outside vendor.
Two operational changes stemming from the 2006 report are the sending of pre-notification
letters and increasing the number of attempts. The analyses of these two experiments suggest
that both methods increase response rates considerably. Macro recommends full adoption of
these operational changes for the 2008 survey. The success of the additional attempts is
conditional on the pre-notification letter. For sampled enrollees not receiving pre-notification,
there was an insignificant increase in response rate for those receiving additional attempts. Yet,
when a pre-notification letter is sent, the additional attempts have a positive impact on response
rates.
Another improvement for the 2007 survey is the use of a more complex weighting scheme.
Incorporating health measures into the non-response (and frame coverage) adjustment mitigates
biases in the survey response data due to varying frequencies of use among VHA enrollees. This
adjustment will improve estimates of utilization.
In summary, the 2007 results are very similar to the 2005 results. The experiments implemented
for the 2007 survey are promising enhancements and should be applied to the entire sample.
Future experiments should continue to focus on improving frame information with reverse phone
look-ups or potentially a mail or internet survey for enrollees without a phone number.
2008 Recommendations:
1. Send pre-notification letters to sampled enrollees and increase the number of call attempts
from 6 to 10.
2. Experiment with reverse phone look-ups based on address information.
3. Experiment with alternative response options for enrollees with no telephone.
4. Continue using the propensity score weighting.
Background
VHA serves American veterans by providing primary and specialized care as well as related
medical and social support services. It administers the country’s largest, most comprehensive,
3
integrated healthcare system. In 2007, VHA served over seven million veteran enrollees. The
number of veterans turning to VHA for healthcare increases every year, and their need is
expected to grow. More and more veterans are turning to VHA as a result of changes in our
nation’s economy, the demographics of the veteran population, and as benefits available to them
under Medicare diminish. In addition, rising healthcare costs and a financial burden that is more
often being placed on the consumer will also contribute to more veterans relying on VHA for
assistance.
While demand for healthcare services grows, VHA's ability to meet this demand is
circumscribed by the Veteran’s Healthcare Eligibility Reform Act of 1996 (Public Law 104262). This law instituted a priority-based enrollment system designed to balance the needs of
those veterans most in need of services, with the necessity to control healthcare costs and
demands on the system. Under this law, the number of priority levels to which VHA can deliver
care is a function of annual funding levels and utilization of healthcare services by enrollees.
The 1996 law also requires VHA to fully understand the reliance of enrolled veterans on VHA
healthcare services and programs compared to their use of non-VHA services and programs (also
known as “VA reliance”). This understanding comes from data gathered through a survey of
veteran enrollees (the VHA Survey of Enrollees). The VHA Survey of Enrollees was developed
with core and supplemental sections to gather a variety of information that determines the
relationship among demographic, socioeconomic, and morbidity characteristics of veteran
enrollees, as well as enrollees’ choice of healthcare providers and their healthcare utilization.
VHA has conducted six cycles of this survey of veteran enrollees (1999, 2000, 2002, 2003, 2005,
and 2007). The data gathered by the VHA Survey of Enrollees also establishes the number of
priority levels that VHA can support. It is used to develop healthcare budgets and to assist the
Department for Veterans Affairs with its annual enrollment decisions. This data is also used as
inputs into the VHA Healthcare Enrollment Projection Model. Forecasts developed from this
model are used for a number of purposes, such as the Capital Asset Realignment for Enhanced
Services (CARES), Millennium Bill Projects, budgeting, and scenario-based policy and planning
analyses.
Any collection of information conducted or sponsored by a Federal agency requires OMB
clearance. As part of the FY07 OMB clearance package, VHA was tasked with conducting a
non-response bias assessment of the VHA Survey of Enrollees, as well as with examining the
quality of the information in the sampling frame. The 2006 analysis satisfied this task. VHA
and Macro met with OMB to discuss the 2006 analysis and agreed to develop methods to
improve the survey program. OMB granted clearance to VHA with the condition that VHA take
steps to improve the design, starting with the 2007 survey. Since the 2007 survey was already
under development, OMB and VHA agreed to experiment with design enhancements that could
be seamlessly integrated into the 2007 survey. Two experiments were added for 2007—sending
a pre-notification letter and increasing the number of call attempts.
This report details Macro’s findings of the non-response bias assessment for the 2007 Survey of
Enrollee results and an assessment of the sampling frame and design. This report is organized as
follows:
4
•
•
•
•
•
•
•
Results of the survey experiments—pre-notification and additional attempts;
A summary of the sample design for the Survey of Enrollees;
The sample design and its relation to interview outcomes;
Findings for the sample bias analysis (frame eligibility);
Results of the non-response bias analysis;
Survey weighting; and
Weighting adjustments.
2007 Experiments
One of the 2006 findings was that the VHA enrollee database had an address listed for nearly all
of the enrollees, whereas it only listed a valid phone number for about three-quarters of
enrollees. One way to take advantage of address listings is to send a pre-notification letter.
Survey pre-notification letters generally increase response rates by informing respondents about
the survey—explaining the survey's purpose, describing the information that will be collected,
and indicating when the survey call should be expected. To evaluate the impact of sending prenotification letters, in 2007 VHA mailed these letters to a subsample of the sampled enrollees;
the subsample was comprised of 42,000 randomly selected enrollees with a valid address. For
operational feasibility, the pre-notification letter experimental group was the first group of
replicates released for data collection. This may have had a minor impact on the evaluation since
the sample of enrollees selected for the pre-notification letter experiment were in the field longer,
which tends to increase response rates slightly. However, we believe that any impact on the
experiment due to the unequal time in the field is minimal.
A second experiment added to the 2007 survey evaluated the impact of an increased number of
attempts. Ten percent of the enrollee sample received a 10-attempt protocol rather than the sixattempt protocol used in past survey cycles. More attempts tend to increase response, but each
additional attempt provides diminishing returns. Most completed interviews occur within the
first few calls, but the most difficult respondents to reach usually need more attempts.
As enrollees were assigned to each experiment independently, the design has four conditions:
1. Sent pre-notification and receiving additional attempts;
2. Sent pre-notification and not receiving additional attempts;
3. Not sent pre-notification and receiving additional attempts; and
4. Not sent pre-notification and not receiving additional attempts.
The response rates for each of the conditions are presented in Table 1. As expected, the highest
response is in the group that received the pre-notification letter and received a 10-attempt
protocol—43.3 percent. This is twice as high as the group that did not receive either
experimental condition—21.4 percent. Sending a pre-notification letter clearly improves
response to the survey. The response rate for enrollees who received the letter is 38.9 percent,
much higher than those who did not receive the letter.
About 7.3 percent of the letters mailed out were returned as undeliverable. Of these, about 45
percent of the letters were returned because of undeliverable addresses and another 33 percent
5
due to an unknown addressee. Nearly all enrollees have address information, and 92.6 percent of
the addresses for frame-eligible enrollees (valid phone number and stratification variables) are
deliverable. The response for undeliverable mail is 12.1 percent, while the delivered mail
response rate is 39.7 percent. The cost to print and mail pre-notification letters to 42,000
enrollees is $26,551.20, or $0.63 per letter. The 42,000 sampled enrollees yielded 10,502
interviews, so the cost per completed interview is $2.53. At $2.53 per completed interview and
an annual target of 42,000 completed interviews, sending letters to the entire sample will cost
about $106,185. Sending the letters increases response rate—and therefore, also the
interviewing productivity. However, another factor to consider is survey length. The survey
averaged 17.5 minutes for interviews conducted with enrollees assigned to the letter group while
the average length for the non-letter group averaged 16.8 minutes – the difference of only a
minute on the telephone may or may not be attributable to the letter The total time per
completed interview, which includes time spent on incomplete interviews as well as non-talk
time (between calls and after call work) is 30.8 for the letter group and 32.6 for the non-letter
group. This translates into a reduction of about 1,800 hours over the course of the full project.
However, this savings, about $42,000, does not completely offset the mailing costs.
Overall, the 10-attempt protocol seems to have had a slight impact on response rates relative to
the six-attempt, 25.3 percent versus 24.0 percent. The effect is exaggerated when paired with the
letter, suggesting that using the letter in conjunction with extra attempts may yield the best
results. Among those who were sent the letter, the 10-attempt protocol groups have a response
rate that exceeds the six-attempt group by five points, 43.3 percent to 38.4 percent. This effect is
not evident among those enrollees who were not sent letters, 22.1 percent and 21.4 percent.
Although this increase is significant, it is small in comparison to the difference in response rates
for those who were sent letters.
Table 1. Response Rates for the Experimental Conditions
Partial
513
103
102
1
410
Refused
interview
18,944
2,831
2,751
80
16,113
Other
noninterview
29,406
4,365
4,217
148
25,041
Unknown
status
85,090
9,232
8,696
540
75,858
4,441
38,147
51
462
1,971
16,973
2,804
26,602
8,281
76,809
1,149
9,353
3,292
28,794
10
93
41
369
308
2,523
1,663
14,450
348
4,017
2,456
22,585
841
8,391
7,440
68,418
Sampled
Enrollees
264,199
42,000
38,931
3,064
222,199
Response
Rate
24.1%
38.9%
39.7%
12.1%
21.5%
Complete
interview
42,588
10,502
10,396
106
32,086
10-attempt
6-attempt
26,413
237,786
25.3%
24.0%
Letter and 10-atts
Letter and 6-atts
No letter and 10-atts
No letter and 6-atts
4,200
37,800
22,213
199,986
43.3%
38.4%
22.1%
21.4%
Total
Letter
Delivered
Returned
No letter
The response rates were calculated with American Association of Public Opinion Research
(AAPOR) Response Rate 1 (RR1), which is a strict definition that assumes all unresolved
records are eligible respondents. This response rate is described in a later section of this report.
6
To further examine the impact of a more rigorous call attempt protocol and sending the letters,
we compare the survey responses for the respondents in each group:
• Respondents assigned to the 10-attempt protocol versus those assigned to the 6-attempt
protocol, and
• Respondents who were sent a pre-notification letter versus those who were not sent a
letter.
Further, for respondents who were assigned to the 10-attempt protocol, we examine the
respondents who were reached after six attempts (late responders, those who would not have
been interviewed under the existing protocol) versus those who were reached during the first six
attempts (early responders). We look at four questions about health insurance coverage:
PREA. Are you enrolled in VA healthcare?
A1. Are you covered by Medicare?
A7. Are you currently covered by Medicaid for any of your healthcare?
A9. Are you currently covered by any other individual or group health plan that either you, or an employer,
or someone else, such as a family member obtains for you?
We see significant differences in three out of four estimates when comparing the responses for
enrollees sent letters versus those who were not. The percentage of enrollees who reported that
they didn’t remember enrolling or don’t know if they are enrolled is slightly higher in the group
that was sent letters, 3.4 percent to 2.7 percent (p-value = 0.0289), but this difference seems to
have very little practical significance. The percentage of enrollees covered by Medicare is nearly
3 percentage points higher in the letter group than the non letter group (71.1% to 68.4%, p-value
= 0.0002), as is the percentage of enrollees covered by another group health plan (28.6% to
26.0%, p-value = 0.0002). There is no difference in the percentage enrolled in Medicaid.
When comparing results for enrollees assigned to the six-attempt versus those assigned to the 10attempt protocol, there are no significant differences among the four questions we compared.
However, the results of the comparison between the enrollees who were reached within six
attempts and those reached after six attempts do suggest that the enrollees interviewed later differ
from those interviewed early. The percentage of late responders who do not remember enrolling
or do not know if they are enrolled with VHA is considerably higher than the early responders,
7.9 percent versus 2.7 percent (p-value = 0.0021.) Likewise, the percentage of late responders
covered by Medicare is much lower than early respondents (54.7 percent to 68.7 percent, p-value
= 0.0005) while the percentage covered by other insurance is much higher (35.5 percent to 26.1
percent, p-value = 0.0102). There is no difference in the percentage enrolled in Medicaid.
7
Table 2a. Comparison of Survey Responses for Attempts and Letter Experiments
Letter
sent
Total responding enrollees
PREA. Are you enrolled in
VA health care?
A1. Are you covered by
Medicare?
Yes
No
DR/DK*
Yes
No
No
letter
sent
10,502
32,086
85.5%
85.3%
11.2%
11.9%
3.4%
2.7%
p-value = 0.0289
71.1%
68.4%
28.9%
31.6%
p-value = 0.0002
9.6%
9.4%
90.4%
90.6%
p-value = 0.7036
28.6%
26.0%
71.4%
74.0%
A7. Are you currently covered Yes
No
by Medicaid for any of your
health care?
A9. Are you currently covered Yes
by any other individual or
No
group health plan that either
you, or an employer, or
p-value = 0.0002
someone else, such as a family
member obtains for you?
*
DR/DK = I don’t remember enrolling or Don’t know
6attempt
10-attempt
Total
Early
Late
38,146
4,441
85.5%
84.6%
11.6%
12.4%
2.9%
3.0%
p-value = 0.6008
69.3%
67.9%
30.7%
32.1%
p-value = 0.2038
9.5%
9.3%
90.5%
90.7%
p-value = 0.8548
26.7%
26.6%
73.3%
73.4%
4,159
282
84.6%
84.2%
12.6%
7.9%
2.7%
7.9%
p-value = 0.0021
68.7%
54.7%
31.3%
45.3%
p-value = 0.0005
9.3%
9.1%
90.7%
90.9%
p-value = 0.9395
26.1%
35.5%
73.9%
64.5%
p-value = 0.9243
p-value = 0.0102
During the survey, enrollees are also asked about whether they received any services from VHA,
“In 2006, did you use any VA healthcare services, or did you have any of your health care paid
for by VA?” If the answer was yes, respondents were asked about these services including:
A13.
Do you currently have prescription drug coverage from VA?
B15. In 2006, did you stay overnight at any VA Medical Hospital or a VA Mental Health Facility, or have
any stays at Non-VA facilities that were paid for by VA?
If yes,
B16. In 2006, how many total overnight stays, if any, did you have at a VA Medical Hospital, or a
medical hospital paid for by VA? Please do not count stays for mental health and substance
abuse treatment?
B19. In 2006, how many overnight stays, if any, did you have for mental health or substance abuse
treatment at a VA Facility or at a facility paid for by VA?
B22. In 2006, how many outpatient visits for medical care did you make that were paid for by VA? That
would include the number of times you went to a VA doctor, hospital or clinic for medical care or
received medical care somewhere else that was paid for by VA. Do not count dental or mental
health visits or trips to a pharmacy.
B23. In 2006, how many home health care visits, if any, were made to you by VA providers or non-VA
providers paid for by VA?
B24. In 2006, how many outpatient visits for mental health or substance abuse treatment, if any, did you
make to VA or visits elsewhere that were paid for by VA?
From these questions we calculate six variables indicating whether the enrollee utilized VHA
services or not:
1. Received home health benefits
2. Inpatient treatment
a. Mental health or substance abuse
b. Non-mental health and non-substance abuse
3. Outpatient treatment
8
a. Mental health or substance abuse
b. Non-mental health and non-substance abuse
4. VHA pharmacy benefits
These utilization statistics are self-reported. Later in the report, we use administratively
measured utilization statistics to analyze frame coverage and non-response bias in the survey.
Table 2b. Comparison of Self-reported VHA Utilization for Attempts and Letter Experiments
Yes
10,502
97.3%
No
letter
sent
32,086
97.0%
No
2.7%
3.0%
Letter
sent
Total responding enrollees
1. Received home health benefits
2a. Inpatient treatment for
mental health or substance abuse
2b. Inpatient treatment for nonmental health and non-substance
abuse
3a. Outpatient treatment for
mental health or substance abuse
3b. Outpatient treatment for nonmental health and non-substance
abuse
4. VHA pharmacy benefits
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
p-value = 0.3891
0.8%
0.7%
99.2%
99.3%
p-value = 0.4397
5.8%
5.9%
94.2%
94.1%
p-value = 0.7084
8.2%
8.2%
91.8%
91.8%
p-value = 0.9900
57.6%
60.0%
42.4%
40.0%
p-value = 0.0999
25.7%
24.8%
74.3%
75.2%
p-value = 0.1647
10-attempt
6attempt
Total
Early
Late
38,146
2.9%
4,441
2.9%
4,159
2.9%
282
2.2%
97.1%
97.1%
97.1%
97.8%
p-value = 0.9291
0.7%
0.5%
99.3%
99.5%
p-value = 0. 3367
5.9%
5.9%
94.1%
94.1%
p-value = 0.9909
8.2%
9.1%
91.8%
90.9%
p-value = 0.1634
58.5%
59.3%
41.5%
40.7%
p-value = 0.4863
74.9%
75.2%
25.1%
24.8%
p-value = 0.7371
p-value = 0.6662
0.4%
1.9%
99.6%
98.1%
p-value = 0.0465
5.8%
8.0%
94.2%
92%
p-value = 0.2987
9.0%
9.4%
91.0%
90.6%
p-value = 0.8856
59.6%
53.6%
40.4%
46.4%
p-value = 0.1487
73.4%
73.1%
26.6%
26.9%
p-value = 0.4859
When comparing the enrollees who were sent a letter to those who were not, we see no
significant differences in utilization statistics. The largest difference comes in measuring
outpatient treatment unrelated to substance abuse or mental health, 57.6 percent for those sent a
letter versus 60.0 percent for those who didn’t (p-value = 0.0999).
We see no statistically significant differences in the service utilization statistics when comparing
the sample that was administered a 6-attempt protocol to the sample that was administered the
10-attempt protocol. When comparing the late responders to the early responders, only one
difference is significant, the percentage of enrollees receiving inpatient treatment for mental
health or substance abuse, 0.4 percent for early responders and 1.9 percent for late responders (pvalue = 0.0465).
Overall, only six percent of respondents completed the interview after six attempts so the overall
results aren’t significantly impacted, but the late respondents differ considerably from early
respondents with respect to health coverage. The distribution of completed interviews over the
call attempts is presented in Figure 1.
9
Figure 1. Percentage of Interviews Completed on Each Attempt
40.0%
35.0%
30.0%
Percent
25.0%
6-attempt
20.0%
10-attempt
15.0%
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17
Attempts
Sample Design
VHA provides Macro with a sample of records from its database of enrollees. We understand
that the sample for the Survey of Enrollees is selected in the following manner:
•
•
•
•
•
VHA considers the entire universe of enrollees who are listed as of a certain date—this
list includes both institutionalized and non-institutionalized veterans.
VHA eliminates all records that lack a telephone number.
VHA then eliminates all records where the telephone number is incomplete or lacks a
valid exchange-area code combination.
VHA eliminates all records where at least one of the sample stratification variables is
absent; namely VISN, pre/post enrollee status, or priority group status.
The file of enrollees is then stratified by pre/post enrollee status, priority group, and
VISN, and independent random samples are drawn for each stratum.
Sampling Design and Interview Outcomes
The final sample of enrollees responding to the Survey of Enrollees must pass through many
stages:
• First, to be in the final sample of respondents, an enrollee must be in the sampling
frame—meaning that contact information and all stratification variables are available;
• Then, the enrollee must be sampled via the stratified random selection process;
• Next, the enrollee’s contact information must be valid and lead to the correct enrollee;
and
• Finally, the enrollee must elect to respond to the survey.
10
The only stage that is a controlled random process, and therefore not subject to potential bias, is
the random sample selection. All other stages have the potential to introduce non-random
systematic bias into enrollee estimates. Figure 2 presents enrollee totals at each of the sample
stages for the 2007 survey. Table 3 presents the enrollee frequencies for each of the major
stratum levels: VISN, enrollee type, and priority group.
Figure 2. Stages and Enrollee Totals for the 2007 Survey of Enrollees
Enrollee Population
7,186,950
In Frame
5,251,999
Not in Frame
1,934,951
In Sample
264,199
Not in Sample
4 987 800
Eligible Contact
176,572
Response
42,680
Ineligible Contact
87,627
Nonresponse
133,892
Note: The number of respondents displayed in this chart includes 93 enrollees who responded to the survey, but did
complete the full interview.
11
Table 3. Stages and Enrollee Totals for the 2007 Survey of Enrollees
Enrollee
Population
Total
VISN
Priority Group
Enrollee type
Frame
Eligible
Enrollees
Selected
Correct
Contact
Survey
Responses
7,186,950
5,251,999
264,199
176,572
42,680
1
316,460
265,114
12,397
9,046
2,029
2
197,528
152,182
10,869
7,810
2,019
3
317,488
249,704
15,499
10,612
2,016
4
418,972
356,307
11,637
8,511
2,063
5
175,200
141,136
17,230
11,303
2,035
6
381,461
295,734
12,540
8,537
2,039
7
431,230
260,026
17,384
10,643
1,983
8
603,388
402,563
13,854
9,220
2,024
9
327,825
247,681
11,329
7,787
2,041
10
262,048
197,651
12,430
8,416
2,031
11
311,908
235,417
10,902
7,399
2,042
12
307,158
227,133
12,979
8,798
2,042
15
291,310
233,703
10,098
6,913
2,034
16
584,123
427,965
11,237
7,376
2,029
17
325,362
244,036
14,576
9,079
2,017
18
300,407
177,022
12,382
7,921
2,040
19
209,591
154,322
11,189
7,375
2,038
20
313,982
224,221
10,276
6,469
2,031
21
298,688
225,832
11,998
8,072
2,051
22
384,319
244,508
14,698
9,194
2,044
23
354,576
289,742
8,695
6,091
2,032
1
870,018
618,474
21,724
15,580
4,269
2
496,102
351,280
24,528
16,211
4,256
3
939,191
666,581
27,624
17,661
4,279
4
196,791
140,324
25,549
15,891
4,284
5
2,252,477
1,585,318
25,386
16,043
4,252
6
253,289
173,923
33,162
20,326
4,243
7/8
2,179,059
1,716,099
106,226
74,860
17,097
POST
4,796,634
3,557,789
122,530
84,041
21,378
PRE
2,390,316
1,694,210
141,669
92,531
21,302
Frame Eligibility
About 27 percent of the enrollee population was ineligible to be in the sampling frame due to
incomplete telephone information or incomplete stratification information—slightly higher than
12
the 25.6 percent in 2005. A telephone number may be missing from the sample completely, be
missing digits, or not have a valid exchange-area code combination. Less than one percent of the
frame exclusions have a valid telephone but are missing stratification variables, so the
overwhelming majority of exclusions are due to invalid telephone numbers. The implication is
that there is significant coverage error within the current sample design.
Currently, the presence or absence of an address for an enrollee is not considered when
determining frame eligibility. An address represents an alternative piece of information that
could be used to locate a veteran either via mail or to identify a telephone number by reverse
look-up procedures. The promise of accurate address information is realized in the prenotification experiment, which nearly doubled the response rate.
Looking at the frame exclusions with valid stratification information and invalid telephone
number, 98.7 percent had a valid address. Overall, 99.4 percent of the enrollees have valid
address information. Both of these statistics are similar to the 2005 frame analysis. A record
with a valid address has data in all relevant mailing address fields. This does not mean that the
ZIP code, city, or other mailing address fields have accurate data—it only means that there is
data in each field. Table 4 presents the distribution of enrollees based on telephone and address
validity. Table 13 in the Appendix provides the percentages by VISN, priority group, and
enrollee type.
Table 4. Frequency and Percent of Enrollees with Valid Addresses and Telephone Numbers
Valid Phone
No
Yes
Total
Valid Address
No
Yes
Total
No
Yes
Total
No
Yes
Total
Total
24,154
1,852,714
1,876,868
17,353
5,292,729
5,310,082
41,507
7,145,443
7,186,950
Percentage
0.3%
25.8%
26.1%
0.2%
73.6%
73.9%
0.6%
99.4%
100.0%
For enrollees who have received services (home healthcare, inpatient or outpatient care) in the
past 12 months, the frame eligible percentage is slightly higher than for those who have not
received services, 74 to 72 percent. Similarly, the frame eligibility percentage is slightly higher
for enrollees receiving the prescription drug benefit, 74 to 72 percent.
Sample Selection
A total of 264,199 enrollees were sampled from the frame in order to meet the sample size
requirements for each stratum; this was 72,551 more than last year’s sample. The sample was
stratified, with 294 strata defined by 21 VISNs (1-12, 15-23), two enrollee groups (pre and post),
and seven priority groups (one through six and, seven and eight). The target number of
completed interviews per stratum was 200 in priority groups one through six and 400 in priority
group seven and eight—totaling 42,000 overall. Within each stratum, a random sample of
enrollees was selected from the frame.
13
The equal allocation of sample to strata results in a disproportionate sample with smaller strata
receiving higher shares of sample than the larger strata. For analysis at the sampling stage, we
use design weights equal to the ratio of the frame total to the sample total in each stratum.
Survey Eligibility
All of the enrollees sampled for the survey were called at least once in order to initiate an
interview. During data collection, many telephone numbers were classified as ineligible
including: non-working numbers, wrong numbers where selected enrollee is not known, out of
service numbers, fax or modem telephone numbers, and business numbers where the enrollee is
not known. Although these were ineligible for the survey since they did not lead to the selected
enrollee, this loss of sample may impose bias on the survey estimates since these enrollees were
part of the population, yet cannot be reached for interview. There were no protocols for
identifying an alternative telephone number other than the ability to contact an alternative
number if provided. The dialing of telephone numbers during data collection was a second form
of frame validation since, albeit the enrollee was included in the frame, the frame information
did not lead to the selected enrollee. The percentage of sampled enrollees with invalid contact
information was 33 percent. Adding this sample estimate to the 27 percent of enrollees excluded
from the frame suggests that 51 percent of the frame may not be reachable by telephone using
the current data collection protocols and sample design.
For enrollees who had received services (home healthcare, inpatient, or outpatient care) in the
previous 12 months, the survey eligibility rate was much higher than for those who had not
received services, 74 to 59 percent. Similarly, the survey eligibility rate was much higher for
enrollees receiving the prescription drug benefit, 74 to 60 percent. Enrollees who received
services had more incentive to keep their contact information current and accurate. Further, an
enrollee receiving services presents an opportunity for VHA to confirm and update current
contact information.
The design weights described in Sample Selection above were used in the analysis of the
enrollees with correct and incorrect contact information.
Non-response
After determining that the telephone contact information was accurate, the final stage of the
process became either a complete interview with the enrollee (response) or unsuccessful
interview attempts. We classify non-response into two forms: enrollee refusal and enrollee noncontact. Enrollee refusals result when an enrollee (or an enrollee agent) is contacted, the sponsor
(VHA) and purpose of the survey are communicated, and the enrollee elects not to participate by
verbal refusal, hang-up, or other form of termination. A non-contact means that the enrollee (or
an enrollee agent) is never reached directly; this includes answering machines and other
technological barriers, language barriers, hang-ups and refusals before or during the survey
introduction (where an enrollee’s presence in not yet confirmed), busy phone numbers, etc.
In general, non-response is evaluated by examining a survey’s response rate (i.e. the proportion
of completed interviews relative to the selected sample, minus the identified ineligible sample
elements); response rates of less than 70-80 percent are frequently considered to imply that there
14
is the potential for significant non-response bias to be present in the results. For the FY07 Survey
of Enrollees, the final response rate using AAPOR RR1 calculations was 24 percent for the
overall sample. 1 Therefore, the potential for non-response bias is considerable.
i
Response Rate =
(i + p ) + (r + nc + o ) + (uo ) + (uh )
Where:
i is a completed interview
p is a partial interview
r is a refusal
nc represents non-contacts ( i.e. answering machines, fax machines, callbacks, etc.)
o represents “other” (i.e. language barrier, no eligible proxy, etc.)
uo represents unknown others (i.e. no answer/ no previous contact, busy/no pervious
contact, hang-ups, etc)
uh represents working telephone number but unknown if veteran located there (i.e. no
opportunity to screen for eligibility)
The design weights described in Sample Selection were used in the analysis of the enrollees with
correct and incorrect contact information.
Bias Analysis
With the exception of the controlled random sampling process, all stages described in the
previous section have the potential to introduce bias into the survey estimates. The impact of
coverage (or frame) bias and non-response bias are difficult to assess since data are not available
for those who do not participate in the survey. Therefore, there is no way to compare the groups
and draw inferences about the survey data. In lieu of survey responses for individuals who do not
participate in the survey, we rely on secondary information available for both survey respondents
and non-respondents. This information generally comes from the sampling frame and/or the
population. In most cases, this information is limited, but in the case of VHA, there is
considerable administrative data available about the population of enrollees. This information
allows us to review the frame coverage and non-response biases for the survey with respect to
enrollees’ use of various VHA services.
For the purpose of conducting this bias analysis, VHA provided Macro with a file based on
administrative records that indicated if an enrollee had utilized any of the following services in
the past year (the file did not indicate the frequency or amount for any of these benefits):
1. Received home health benefits
2. Inpatient treatment
a. Mental health or substance abuse
b. Non-mental health and non-substance abuse
3. Outpatient treatment
1
Response rates reported for the VHA Reliance Survey prior to 2005 reported the equivalent of AAPOR cooperation rates.
15
a. Mental health or substance abuse
b. Non-mental health and non-substance abuse
4. VHA pharmacy benefits
The following sections detail the bias analysis using this information.
1. Receiving Home Health Benefits
Only a fraction of enrollees receive home health benefits, 0.11 percent overall. This percentage
increases slightly to 0.12 percent for frame eligible enrollees, compared to 0.09 percent for frame
ineligible enrollees. The percentage of enrollees receiving home health benefits with valid
contact information is 0.11 percent compared to 0.08 percent for enrollees with invalid contact
information (p-value=0.0722). There is insufficient information to detect a difference between
the responding enrollees and non-responding enrollees (0.14 vs. 0.10 percent, p-value=0.1158).
This pattern of differences among groups is similar to the pattern found last year.
In Priority Group 4, the percentage receiving home healthcare is much higher than the rest of the
strata, 1.25 percent. Over the course of each stage of the process, this percentage increases to
1.30 percent in the frame, 1.48 percent with eligible contact information to 1.55 percent for
responding enrollees. Reviewing the remaining strata, the stage that seems to introduce bias
appears to be invalid contact information. There are several strata where the enrollees with
invalid contact information are significantly different than those with valid contact information.
For all significant stratum differences (p < 0.1; five VISNs, three priority groups, and preenrollee type), the percentages for enrollees without contact information are lower than those
with valid contact information. This seems intuitive in that contact information for enrollees
receiving home healthcare is bound to be updated more frequently and completely. Like last
year, there are fewer statistical differences between respondents and non-respondents. Four
VISNs (5, 16, 18, 21—none of the same groups as last year) and Priority Group 1 differed
between respondents and non-respondents, although not in a consistent direction.
16
Figure 3. Percentage of Enrollees Receiving Home Healthcare
Population
In Frame
0.11%
Yes
No
In Sample
Yes
Eligible
Yes
No
Respond
0.12%
0.09%
0.10% [0.09%, 0.12%]
0.11% [0.09%, 0.13%]
0.08% [0.06%, 0.10%]
Yes
No
0.14% [0.09%, 0.20%]
0.10% [0.08%, 0.12%]
0.0
0.2
0.4
0.6
Percent
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
17
0.8
1.0
Table 5. Percentage of Enrollees Receiving Home Healthcare
In Frame
Population
Total
VISN
Priority
Yes
No
Sampled
Yes
Eligible
Yes
No
Respond
Pvalue
Yes
No
Pvalue
0.11
0.12
0.09
0.10
0.11
0.08 0.0722
0.14
0.10 0.1158
1
0.13
0.14
0.08
0.15
0.18
0.05 0.0532
0.08
0.22 0.1406
2
0.32
0.35
0.23
0.32
0.31
0.34 0.8209
0.25
0.33 0.6711
3
0.10
0.12
0.04
0.09
0.11
0.04 0.0254
0.20
0.09 0.2005
4
0.09
0.10
0.05
0.08
0.09
0.04 0.1834
0.05
0.11 0.3589
5
0.10
0.11
0.06
0.13
0.12
0.16 0.6189
0.02
0.14 0.0025
6
0.06
0.06
0.04
0.04
0.04
0.03 0.5166
0.04
0.04 0.9746
7
0.11
0.10
0.12
0.08
0.06
0.10 0.4900
0.10
0.06 0.3515
8
0.10
0.09
0.10
0.09
0.10
0.05 0.0941
0.12
0.10 0.7650
9
0.05
0.06
0.03
0.07
0.06
0.09 0.6467
0.04
0.07 0.6061
10
0.28
0.30
0.23
0.16
0.19
0.11 0.1701
0.17
0.19 0.8221
11
0.14
0.14
0.11
0.15
0.17
0.11 0.4627
0.22
0.14 0.5200
12
0.11
0.11
0.12
0.11
0.13
0.08 0.3747
0.19
0.11 0.5486
15
0.07
0.08
0.04
0.06
0.06
0.04 0.5702
0.06
0.06 0.9888
16
0.09
0.09
0.07
0.13
0.13
0.15 0.8217
0.36
0.03 <.0001
17
0.09
0.10
0.06
0.05
0.04
0.08 0.1761
0.05
0.03 0.5628
18
0.08
0.09
0.07
0.07
0.10
0.02 0.0022
0.02
0.14 0.0031
19
0.10
0.11
0.07
0.10
0.12
0.06 0.2553
0.20
0.09 0.2478
20
0.06
0.07
0.03
0.06
0.08
0.03 0.3728
0.06
0.10 0.6534
21
0.19
0.20
0.15
0.16
0.17
0.14 0.7605
0.40
0.08 0.0022
22
0.12
0.11
0.15
0.10
0.12
0.08 0.5225
0.19
0.09 0.3441
23
0.09
0.10
0.06
0.07
0.07
0.07 0.9682
0.10
0.06 0.6312
1
0.19
0.21
0.14
0.18
0.18
0.17 0.8636
0.30
0.14 0.0661
2
0.08
0.08
0.06
0.07
0.04
0.12 0.0259
0.04
0.05 0.8539
3
0.08
0.08
0.05
0.08
0.09
0.07 0.7132
0.13
0.07 0.2757
4
1.25
1.30
1.10
1.32
1.48
1.06 0.0081
1.55
1.45 0.6628
5
0.09
0.10
0.07
0.08
0.10
0.03 0.0225
0.12
0.09 0.6915
6
0.01
0.01
0.01
0.02
0.02
0.02 0.7296
0.03
0.02 0.4309
7/8
0.03
0.03
0.03
0.02
0.03
0.01 0.0405
0.03
0.03 0.7260
0.07
0.07
0.06
0.06
0.07
0.05 0.2598
0.09
0.06 0.3118
0.20
0.22
0.15
0.19
0.21
0.15 0.0807
0.26
0.19 0.1978
Enrollee POST
Type
PRE
Notes: 1. Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
2. N/A indicates no observed cases.
18
2. Inpatient Treatment
Very few enrollees have been admitted to a hospital or medical facility for mental health or
substance abuse reasons—just less than one percent (0.85 percent). This percentage drops
slightly to 0.77 percent for those who have sufficient information to be frame eligible, versus
1.06 percent for those who are frame ineligible. There is a considerable difference between
enrollees with valid contact information and enrollees without valid contact information, 0.61 to
1.22 percent (p-value<0.0001). Overall, non-respondents were somewhat more likely to have
received inpatient treatment (0.67 percent versus 0.46 percent, p-value = 0.0003)—a difference
that was not as significant in last year’s data.
Across VISNs, the percentage ranges from 0.62 to 1.19 percent, with five VISNs exceeding one
percent. The VISN percentages based on the responding enrollees suggests that the percentage
has an average of 0.46 percent with a range of 0.11 to 1.04 percent. In Priority Groups 1 and 4
(the two groups with the highest population percentage of enrollees admitted to a hospital or
medical facility for mental health or substance abuse reasons), the percentages, as measured from
the responding enrollees, 1.24 and 3.60 percent, underestimate the population percentages of
2.17 and 6.40 percent. This was due both to differences in enrollees with valid versus invalid
contact information and differences between those who responded to the survey or did not.
For all of the significant differences in eligibility (19 of the 21 VISNs, six of the seven priority
groups, and both pre- and post-enrollee status), the percentage for enrollees with eligible contact
information is less than those without. This pattern holds when evaluating the percentage
between responding and non-responding enrollees (seven of the 21 VISNs, two of the seven
priority groups, and both pre- and post-enrollee status).
Turning to enrollees admitted to a hospital or medical facility for reasons unrelated to mental
health or substance abuse, we see a different pattern. In the case of home health and mental
health or substance abuse inpatient visits, there were some noticeable differences at the frame
and response stages, but the stage that seemingly impacts the final results most is the enrollee
contact stage. Those with contact information are frequently different from those without valid
information. In the case of inpatient treatment unrelated to mental health or substance abuse,
there are fewer significant differences with respect to eligibility (only four of 21 VISNs, two of
seven priority groups; and both enrollee types). However, there are more differences between
responding and non-responding enrollees (nine of the 21 VISNs, all seven priority groups, and
both enrollee types). Furthermore, where there are significant differences with respect to
response, estimates based on respondents generally overestimate population percentages.
Overall, the population percentage of 4.42 is overestimated by the responding enrollees, 4.98
percent. The non-responding enrollees have a much lower percentage of 3.84 (p-value<0.0001).
Since 4.15 percent of the enrollees with valid contact information have been admitted for reasons
unrelated to mental health or substance abuse, the overestimation is a result of non-response bias.
The opposite pattern exists when evaluating inpatient treatment unrelated to mental health or
substance abuse. The percentage of enrollees without valid contact information is systematically
lower than those with valid contact information. As with the home healthcare, this seems like a
19
logical result since inpatient treatment offers the opportunity to update contact information.
Curiously, this theory does not hold when evaluating inpatient treatment related to mental health
or substance abuse.
Figure 4. Percentage of Enrollees Receiving Inpatient Treatment
(a) For Mental Health or Substance Abuse
Population
In Frame
0.85%
Yes
0.77%
No
In Sample
Yes
Eligible
Yes
1.06%
0.80% [0.76%, 0.85%]
0.61% [0.56%, 0.66%]
No
Respond
1.22% [1.11%, 1.32%]
Yes
0.46% [0.38%, 0.54%]
No
0.67% [0.61%, 0.73%]
0
2
4
6
Percent
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
(b) Not for Mental Health nor Substance Abuse
20
8
10
Population
In Frame
4.42%
Yes
4.23%
No
In Sample
Yes
Eligible
Yes
4.94%
4.23% [4.12%, 4.35%]
4.15% [4.01%, 4.28%]
No
Respond
4.42% [4.22%, 4.63%]
Yes
4.98% [4.68%, 5.28%]
No
3.84% [3.69%, 4.00%]
0
2
4
6
Percen t
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
21
8
10
Table 6. Percentage of Enrollees Receiving Inpatient Treatment
(a) For Mental Health or Substance Abuse
In Frame
Population
Total
VISN
Priority
No
Yes
Eligible
Yes
No
Respond
Pvalue
Yes
No
Pvalue
0.85
0.77
1.06
0.80
0.61
1.22 <.0001
0.46
0.67 0.0003
1
1.08
0.97
1.63
1.07
0.85
1.71 0.0016
0.63
0.92 0.2664
2
0.77
0.71
0.97
0.70
0.54
1.15 0.0040
0.48
0.55 0.7402
3
0.64
0.62
0.70
0.70
0.51
1.12 0.0004
0.29
0.57 0.1313
4
0.89
0.81
1.37
0.90
0.69
1.54 0.0026
0.68
0.69 0.9927
5
1.19
1.10
1.56
1.12
0.90
1.53 0.0072
0.46
1.01 0.0540
6
1.04
0.98
1.26
0.95
0.80
1.28 0.0936
0.57
0.89 0.2725
7
0.81
0.73
0.95
0.78
0.62
1.03 0.0187
0.59
0.62 0.9030
8
0.67
0.56
0.88
0.50
0.41
0.70 0.0440
0.37
0.42 0.8103
9
1.01
0.93
1.23
0.98
0.85
1.27 0.1154
1.04
0.78 0.4178
10
1.10
0.92
1.66
0.82
0.62
1.25 0.0140
0.35
0.73 0.0750
11
0.88
0.80
1.12
0.75
0.53
1.23 0.0003
0.42
0.58 0.3851
12
0.93
0.76
1.43
0.93
0.74
1.36 0.0093
0.99
0.66 0.2581
15
0.93
0.85
1.27
0.76
0.43
1.51 <.0001
0.37
0.46 0.5863
16
0.80
0.73
0.97
0.96
0.76
1.37 0.0194
0.41
0.90 0.0641
17
0.95
0.86
1.22
0.90
0.68
1.28 0.0252
0.43
0.77 0.2154
18
0.75
0.68
0.86
0.65
0.50
0.93 0.0415
0.19
0.62 0.0138
19
0.86
0.76
1.13
0.68
0.48
1.10 0.0016
0.11
0.62 <.0001
20
0.94
0.86
1.13
0.95
0.76
1.30 0.0415
0.23
1.02 <.0001
21
0.72
0.60
1.06
0.65
0.35
1.31 <.0001
0.23
0.39 0.2871
22
0.62
0.61
0.64
0.53
0.45
0.67 0.1295
0.19
0.53 0.0573
23
0.63
0.54
1.02
0.64
0.40
1.29 0.0002
0.41
0.40 0.9603
1
2.17
2.01
2.57
2.10
1.82
2.80 <.0001
1.24
2.06 0.0022
2
0.69
0.63
0.83
0.66
0.54
0.89 0.0025
0.52
0.55 0.8312
3
0.54
0.50
0.65
0.56
0.49
0.69 0.0333
0.37
0.53 0.1941
4
6.40
6.11
7.13
6.20
5.01
8.16 <.0001
3.60
5.56 <.0001
5
0.78
0.73
0.90
0.76
0.50
1.24 <.0001
0.36
0.55 0.1794
6
0.28
0.27
0.32
0.24
0.21
0.30 0.2063
0.27
0.19 0.3853
7/8
0.12
0.10
0.19
0.12
0.08
0.23 <.0001
0.03
0.09 0.1168
0.53
0.46
0.73
0.50
0.38
0.78 <.0001
0.29
0.41 0.0375
1.48
1.41
1.65
1.44
1.13
2.05 <.0001
0.84
1.24 0.0030
Enrollee POST
Type
PRE
Note:
Yes
Sampled
Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
22
Table 7. Percentage of Enrollees Receiving Inpatient Treatment
(b) Not for Mental Health or Substance Abuse
In Frame
Population
Total
VISN
Priority
No
Yes
Eligible
Yes
No
Respond
Pvalue
Yes
No
Pvalue
4.42
4.23
4.94
4.23
4.15
4.42 0.0320
4.98
3.84 <.0001
1
3.74
3.68
4.05
3.69
3.62
3.90 0.5950
4.16
3.44 0.2403
2
3.79
3.84
3.61
3.58
3.44
3.94 0.3473
3.56
3.40 0.7885
3
3.54
3.61
3.31
4.01
3.70
4.68 0.0310
4.78
3.45 0.0280
4
3.48
3.23
4.92
3.35
3.18
3.87 0.2179
3.86
2.95 0.1496
5
4.79
4.77
4.85
4.87
4.88
4.86 0.9744
5.99
4.61 0.0685
6
4.29
4.35
4.06
4.15
4.21
4.01 0.7207
4.47
4.12 0.6219
7
3.91
3.35
4.75
3.31
2.92
3.93 0.0077
3.70
2.73 0.0640
8
4.62
4.23
5.38
4.17
3.92
4.72 0.1228
4.97
3.61 0.0313
9
5.77
5.73
5.89
5.98
5.88
6.23 0.6314
7.64
5.21 0.0095
10
4.38
3.94
5.71
3.72
3.47
4.24 0.2042
2.90
3.70 0.2168
11
4.00
3.95
4.16
3.80
3.55
4.36 0.1579
3.59
3.53 0.9230
12
4.76
4.19
6.38
4.37
4.29
4.55 0.6107
4.78
4.13 0.3307
15
4.97
4.88
5.30
4.84
4.95
4.60 0.5941
6.31
4.31 0.0191
16
4.81
4.67
5.20
4.89
5.23
4.21 0.1201
6.98
4.49 0.0053
17
4.76
4.58
5.31
4.60
4.66
4.49 0.7594
5.53
4.37 0.1354
18
4.93
4.56
5.46
4.43
4.50
4.28 0.6996
4.75
4.41 0.6557
19
4.64
4.42
5.26
4.55
4.81
4.03 0.1547
6.60
4.10 0.0003
20
4.53
4.46
4.70
4.11
3.69
4.86 0.0373
4.13
3.46 0.3029
21
4.22
4.10
4.59
4.12
4.19
3.98 0.7180
4.96
3.89 0.1433
22
4.41
4.31
4.60
4.35
4.60
3.93 0.1538
5.85
4.21 0.0248
23
4.65
4.40
5.74
4.15
3.75
5.25 0.0135
4.29
3.48 0.1860
1
8.87
8.79
9.07
8.93
8.51
9.98 0.0038
9.23
8.21 0.0873
2
3.58
3.50
3.79
3.51
3.80
2.93 0.0013
4.56
3.51 0.0037
3
3.00
2.96
3.11
2.93
3.04
2.72 0.1340
3.74
2.81 0.0023
4
16.50
16.22
17.20
16.33
16.13
16.64 0.3387
17.16
15.73 0.0517
5
5.58
5.48
5.83
5.44
5.54
5.26 0.3964
6.62
5.12 0.0008
6
1.38
1.37
1.41
1.20
1.15
1.28 0.4201
1.74
0.98 0.0007
7/8
1.51
1.40
1.95
1.40
1.37
1.48 0.3273
1.65
1.28 0.0035
2.94
2.71
3.59
2.60
2.46
2.91 0.0020
3.05
2.25 <.0001
7.40
7.43
7.33
7.66
7.85
7.29 0.0299
9.24
7.34 <.0001
Enrollee POST
Type
PRE
Note:
Yes
Sampled
Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
23
3. Outpatient Treatment
For outpatient treatment unrelated to mental health or substance abuse, there is extreme
systematic bias evident. Overall, the population percentage is 58.85 percent and is slightly
higher for frame eligible enrollees, 59.86 percent. A one-to-two percentage point increase is
fairly consistent across the strata. The percentage then climbs to 65.02 percent for enrollees with
valid contact information. This is significantly different (p-value<0.0001) from the 48.22
percent for enrollees without valid contact information. The percentage climbs nearly 12 points
to 76.72 percent when measured for the responding enrollees. Overall, 4.06 percent of enrollees
receive treatment, but this percentage drops slightly to 3.81 percent for enrollees who are frame
eligible, and further to 3.71 percent for enrollees who have valid contact information. The
enrollees without valid contact information are statistically different (p-value=0.0017) from
those with information, 4.08 to 3.71 percent. Enrollees not eligible for the frame also show a
higher percentage, 4.74 percent. There is no significant difference between respondents and nonrespondents (p-value=0.81).
As with the inpatient mental health and substance abuse treatment, when a significant difference
exists (four VISNs, two priority groups, and post-enrollee type), the percentage of enrollees with
invalid contact information is systematically higher than those with valid information.
This pattern, where the frame contact information causes overestimation of utilization, is
consistent for outpatient and inpatient care not related to health or substance abuse and across
years.
For Priority Groups 1 and 4, the strata with the highest percentages (11.52 and 9.83 percent), we
see two different patterns. In Priority Group 1, the percentage drops slightly at each stage with
the largest drop coming at the response stage—11.25 percent for frame eligible, 11.06 percent
for valid contact information, and 10.32 percent for responding enrollees. The same pattern holds
for Priority Group 4, but the drop at the response stage is much more exaggerated—9.63 percent
for frame eligible, further to 9.33 percent for valid contacts, and still further to 7.36 for
responding enrollees, which is significantly different from the 60.76 percent for non-respondents
(p-value<0.0001).
This pattern is similar for all strata. Percentages for enrollees with valid contact information are
significantly larger than those without valid contact information and percentages for responding
enrollees are significantly larger than non-responding enrollees. For all but one grouping, more
bias was introduced at the response stage than the eligibility stage.
24
Figure 5. Percentage of Enrollees Receiving Outpatient Treatment
(a) For Mental Health or Substance Abuse
Population
In Frame
4.06%
Yes
3.81%
No
In Sample
Yes
Eligible
Yes
4.74%
3.82% [3.72%, 3.93%]
3.71% [3.58%, 3.83%]
No
Respond
4.08% [3.88%, 4.28%]
Yes
3.73% [3.48%, 3.98%]
No
3.70% [3.55%, 3.84%]
0
2
4
6
8
10
Percent
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
(b) Not for Mental Health nor Substance Abuse
Population
In Frame
58.85%
Yes
59.86%
No
In Sample
Yes
Eligible
Yes
56.11%
59.71% [59.43%, 59.98%]
65.02% [64.70%, 65.34%]
No
Respond
48.22% [47.72%, 48.72%]
Yes
76.72% [76.15%, 77.28%]
No
60.76% [60.38%, 61.14%]
0
20
40
60
Percent
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
25
80
100
Table 8. Percentage of Enrollees Receiving Outpatient Treatment
(a) For Mental Health or Substance Abuse
In Frame
Population
Total
VISN
Priority
No
Yes
Eligible
Yes
No
Respond
Pvalue
Yes
No
Pvalue
4.06
3.81
4.74
3.82
3.71
4.08 0.0017
3.73
3.70 0.8057
1
4.95
4.66
6.45
4.49
4.16
5.46 0.0160
4.27
4.12 0.8121
2
3.26
3.16
3.60
3.22
2.79
4.35 0.0014
2.60
2.85 0.6099
3
3.91
3.88
4.01
3.69
3.74
3.58 0.6819
3.74
3.75 0.9876
4
3.58
3.37
4.75
3.64
3.24
4.85 0.0031
3.06
3.30 0.6562
5
3.46
3.29
4.18
3.49
3.50
3.46 0.9295
2.97
3.63 0.2987
6
4.25
4.12
4.70
3.89
3.86
3.95 0.8724
3.58
3.96 0.5546
7
4.32
3.91
4.95
3.61
3.53
3.72 0.6124
3.75
3.48 0.6462
8
3.28
2.96
3.91
2.81
2.90
2.63 0.5195
3.25
2.79 0.4314
9
4.28
4.17
4.64
4.71
4.41
5.39 0.1068
3.56
4.73 0.0940
10
5.18
4.71
6.64
4.79
4.77
4.83 0.9231
4.09
5.04 0.2364
11
4.15
3.92
4.85
4.02
3.62
4.93 0.0372
4.60
3.16 0.0288
12
3.96
3.43
5.44
3.47
3.47
3.47 0.9822
3.24
3.55 0.5907
15
4.22
3.96
5.30
3.96
3.91
4.07 0.8020
3.90
3.92 0.9858
16
4.29
4.07
4.89
4.30
4.41
4.07 0.5633
4.33
4.45 0.8856
17
4.22
3.91
5.14
4.00
3.89
4.19 0.5746
4.54
3.66 0.2341
18
3.45
3.26
3.72
3.25
3.08
3.54 0.3370
3.95
2.74 0.0532
19
4.29
3.90
5.39
3.82
3.96
3.55 0.4187
3.58
4.11 0.4065
20
4.45
4.22
5.03
4.48
4.30
4.80 0.4143
3.79
4.55 0.3089
21
4.37
4.02
5.47
4.12
3.97
4.44 0.4248
4.11
3.91 0.7736
22
4.04
3.83
4.42
3.48
3.34
3.73 0.3656
3.99
3.14 0.1675
23
3.63
3.28
5.21
3.23
3.10
3.60 0.3513
3.04
3.13 0.8752
1
11.52
11.25
12.16
11.05
11.06
11.04 0.9812
10.32
11.36 0.1078
2
5.16
4.92
5.75
5.06
5.09
5.00 0.7979
4.88
5.17 0.4830
3
3.00
2.82
3.45
2.66
2.63
2.72 0.6564
3.25
2.42 0.0053
4
9.83
9.63
10.32
9.80
9.33
10.56 0.0040
7.36
10.10 <.0001
5
3.57
3.38
4.01
3.48
3.23
3.95 0.0082
3.02
3.30 0.4306
6
3.89
3.67
4.36
3.69
3.75
3.58 0.5869
3.95
3.69 0.5462
7/8
1.28
1.20
1.58
1.26
1.25
1.29 0.6705
1.35
1.21 0.2861
3.20
2.90
4.06
2.96
2.80
3.33 0.0002
2.86
2.78 0.6350
5.77
5.71
5.94
5.63
5.70
5.51 0.4058
5.65
5.71 0.8546
Enrollee POST
Type
PRE
Note:
Yes
Sampled
Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
26
Table 9. Percentage of Enrollees Receiving Outpatient Treatment
(b) Not for Mental Health or Substance Abuse
In Frame
Population
Priority
No
Yes
Yes
No
58.85
59.86
56.11
59.71
65.02
1
60.75
62.49
51.78
61.85
2
53.69
56.27
45.02
3
44.88
47.25
4
60.34
5
Respond
Pvalue
Pvalue
Yes
No
48.22 <.0001
76.72
60.76 <.0001
67.43
45.45 <.0001
78.66
63.74 <.0001
56.21
60.90
43.79 <.0001
74.36
56.25 <.0001
36.17
48.20
53.10
37.27 <.0001
69.20
49.40 <.0001
61.36
54.57
59.90
64.23
46.56 <.0001
76.30
60.05 <.0001
52.33
53.50
47.47
51.98
57.42
41.92 <.0001
72.74
53.65 <.0001
6
58.99
61.07
51.81
61.31
66.39
50.07 <.0001
78.20
62.09 <.0001
7
56.18
50.76
64.43
51.55
55.28
45.65 <.0001
68.31
52.18 <.0001
8
66.85
67.96
64.64
67.20
72.29
55.99 <.0001
85.40
68.40 <.0001
9
62.98
64.93
56.96
64.10
68.33
54.46 <.0001
79.59
64.05 <.0001
10
56.92
55.63
60.89
55.51
61.92
42.31 <.0001
75.78
56.43 <.0001
11
60.31
63.00
52.00
63.46
69.76
49.19 <.0001
79.29
65.34 <.0001
12
60.70
59.79
63.30
59.14
64.57
46.75 <.0001
75.84
60.92 <.0001
15
64.39
66.23
56.90
65.80
71.47
52.66 <.0001
81.73
66.62 <.0001
16
62.01
63.29
58.50
62.83
68.29
51.86 <.0001
76.45
64.79 <.0001
17
59.03
59.49
57.64
60.22
65.22
51.59 <.0001
74.43
62.06 <.0001
18
60.71
61.01
60.27
61.91
67.49
51.88 <.0001
76.69
63.88 <.0001
19
57.76
58.29
56.30
58.21
64.36
45.88 <.0001
76.32
59.60 <.0001
20
55.00
55.80
53.01
55.79
61.85
45.15 <.0001
72.61
56.45 <.0001
21
54.74
55.47
52.47
56.06
60.84
45.86 <.0001
73.62
55.82 <.0001
22
51.82
51.77
51.92
51.43
56.96
42.00 <.0001
69.84
52.93 <.0001
23
65.38
66.91
58.55
67.19
71.88
54.26 <.0001
79.35
68.08 <.0001
1
72.58
73.14
71.22
73.37
75.49
68.05 <.0001
79.25
73.92 <.0001
2
60.02
60.82
58.07
60.90
65.58
51.74 <.0001
73.85
62.44 <.0001
3
53.75
55.55
49.34
55.78
62.05
44.56 <.0001
72.20
58.63 <.0001
4
67.59
68.61
65.05
68.49
73.89
59.59 <.0001
82.18
70.65 <.0001
5
60.19
61.11
57.98
60.67
67.75
47.81 <.0001
80.78
62.74 <.0001
6
46.36
47.16
44.63
47.20
52.07
38.71 <.0001
62.20
49.15 <.0001
7/8
54.59
55.98
49.45
55.73
60.58
41.96 <.0001
74.90
55.68 <.0001
56.60
57.51
54.00
57.29
62.36
45.76 <.0001
74.86
57.81 <.0001
63.38
64.82
59.87
64.79
70.88
52.88 <.0001
80.80
67.26 <.0001
Enrollee POST
Type
PRE
Note:
Eligible
Yes
Total
VISN
Sampled
Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
27
4. VHA Pharmacy Benefits
The percentage of enrollees receiving VHA pharmacy benefit follows very closely to the
observed patterns for outpatient treatment unrelated to mental health or substance abuse. The
percentage of enrollees receiving the benefit is 57.18 percent and increases to 58.04 percent for
frame eligible enrollees. There is a five point increase to 63.01 percent when limited to sampled
enrollees with valid contact information and another increase to 74.92 percent when measured
for responding enrollees. This pattern is consistent across all strata—a slight increase in the
percentage from population to frame eligible and significant increases in the percentage for
enrollees with valid contact information and responding enrollees. All comparisons between
enrollees with valid information to those without are significant. Further, all comparisons of
responding to non-responding enrollees are significant.
Figure 6. Percentage of Enrollees Receiving Prescription Drug Benefits
Population
In Frame
57.18%
Yes
No
In Sample
Yes
Eligible
Yes
54.86%
58.03% [57.75%, 58.30%]
63.01% [62.69%, 63.33%]
No
Respond
47.25% [46.74%, 47.75%]
Yes
74.92% [74.35%, 75.50%]
No
58.67% [58.29%, 59.06%]
0
20
40
60
Percent
Source: 2007 Survey of Veteran Enrollees' Health and Reliance Upon VA
28
80
100
Table 10. Percentage of Enrollees Receiving Prescription Drug Benefits
In Frame
Population
Priority
Enrollee
Type
Note:
Eligible
Yes
No
Yes
Yes
No
57.18
58.04
54.86
58.03
63.01
1
59.42
61.01
51.21
60.45
2
51.43
53.88
43.23
3
43.26
45.71
4
57.84
5
Respond
Pvalue
Pvalue
Yes
No
47.25 <.0001
74.92
58.67 <.0001
65.66
45.13 <.0001
76.91
61.96 <.0001
53.70
57.69
43.15 <.0001
71.90
52.77 <.0001
34.21
46.06
50.85
35.36 <.0001
67.05
47.12 <.0001
58.71
52.86
57.22
61.27
44.76 <.0001
73.74
56.95 <.0001
48.37
49.68
42.96
48.72
53.69
39.50 <.0001
67.92
50.20 <.0001
6
58.36
60.38
51.39
60.57
65.40
49.88 <.0001
77.61
60.96 <.0001
7
55.64
49.64
64.77
50.31
53.52
45.20 <.0001
65.86
50.58 <.0001
8
64.17
64.92
62.67
64.31
69.09
53.80 <.0001
81.34
65.45 <.0001
9
62.08
63.93
56.38
64.09
67.83
55.55 <.0001
78.12
63.92 <.0001
10
57.21
55.43
62.67
55.05
60.95
42.89 <.0001
73.88
55.82 <.0001
11
59.56
62.09
51.76
62.87
68.51
50.11 <.0001
79.54
63.40 <.0001
12
59.23
58.03
62.64
57.77
63.13
45.54 <.0001
75.08
59.25 <.0001
15
63.74
65.43
56.88
65.93
71.32
53.48 <.0001
81.80
66.36 <.0001
16
61.84
63.00
58.68
63.24
68.49
52.69 <.0001
76.80
64.93 <.0001
17
57.64
57.99
56.58
58.60
63.27
50.56 <.0001
73.59
59.74 <.0001
18
57.90
58.43
57.13
58.85
64.13
49.35 <.0001
74.52
60.05 <.0001
19
56.13
56.51
55.09
55.87
61.96
43.67 <.0001
74.53
56.96 <.0001
20
52.97
53.67
51.24
53.90
59.15
44.69 <.0001
68.59
54.41 <.0001
21
51.96
52.41
50.57
53.50
57.81
44.32 <.0001
71.05
52.61 <.0001
22
48.32
48.22
48.49
47.65
52.67
39.10 <.0001
65.41
48.69 <.0001
23
62.48
63.82
56.46
64.19
69.24
50.27 <.0001
77.63
64.98 <.0001
1
78.16
78.27
77.91
78.34
80.31
73.38 <.0001
83.90
78.81 <.0001
2
55.64
56.03
54.69
56.14
60.24
48.14 <.0001
67.20
57.59 <.0001
3
47.69
49.10
44.23
49.33
54.77
39.61 <.0001
64.82
51.38 <.0001
4
74.73
75.24
73.45
75.22
80.63
66.29 <.0001
88.09
77.72 <.0001
5
59.62
60.61
57.25
60.39
67.03
48.35 <.0001
80.49
61.84 <.0001
6
37.66
38.47
35.88
38.29
42.36
31.22 <.0001
52.91
39.31 <.0001
7/8
51.42
52.83
46.18
52.88
57.62
39.43 <.0001
72.00
52.69 <.0001
POST 53.26 54.07 50.95 54.06
58.77
43.35 <.0001
71.33
54.20 <.0001
PRE 65.05 66.38 61.82 66.36
72.35
54.65 <.0001
82.85
68.53 <.0001
Total
VISN
Sampled
Statistical tests for independence are based on the Rao-Scott Chi Square statistic.
29
Survey Weighting
The weighting methodology for the 2007 Survey of Enrollees includes a base weight as the
inverse of the probability of selection in each stratum (enrollee type, VISN, and priority group)
with a non-response adjustment by age group (under 45, 45-64, and 65+). This non-response
adjustment was semi-successful in reducing bias:
• Overall, the non-response weighting tends to reduce bias in measuring the health
estimates—four of six estimates are closer to the population.
• The percentage of enrollees receiving outpatient care unrelated to mental health or
substance abuse and the percentage receiving pharmacy benefits still overestimate the
population by about 12 points, though this is better than the base-weighted estimate.
• For the percentage of enrollees receiving home healthcare—most likely correlated to
age—the age-based non-response adjustment essentially removes the bias completely.
• The bias is considerably reduced for the percentage of enrollees receiving inpatient
treatment related to mental health and substance abuse.
• The bias increased for the percentage of enrollees receiving inpatient treatment unrelated
to mental health or substance abuse and the percentage of enrollees receiving outpatient
treatment related to mental health or substance abuse.
A recommendation stemming from the 2006 analysis was to add utilization statistics to the nonresponse adjustment. The details of the non-response modeling and weighing adjustment are
forthcoming in a later section. The preceding bias analysis is based on weighted data that
accounts for the differential sampling probabilities for each stratum and does not adjust for nonresponse. We ran the bias analysis using the weights used for the Survey of Enrollees to analyze
if the non-response adjustment reduces the biases observed for the health estimates.
Table 11. Survey Estimates and Bias for Weighted and Weighted and Adjusted Data
Base weight and nonresponse
adjustment
Base weight only
Population
1. Home Healthcare
Est
Bias
L95
U95
Est
Bias
L95
U95
0.11
0.14
0.03
-0.02
0.09
0.11
0.00
-0.04
0.03
(a) Related to MH/SA
0.85
0.46
-0.39
-0.47
-0.30
0.89
0.04
-0.12
0.21
(b) Unrelated to MH/ SA
4.42
4.98
0.56
0.25
0.86
4.51
0.09
-0.21
0.38
4.06
3.73
-0.32
-0.57
-0.07
4.07
0.01
-0.29
0.32
(b) Unrelated to MH/ SA
58.85
76.72
17.86
17.30
18.43
58.85
0.00
-0.86
0.86
4. VHA Pharmacy benefit
57.18
74.92
17.74
17.17
18.31
57.30
0.11
-0.73
0.96
2. Inpatient treatment
3. Outpatient treatment
(a) Related to MH/SA
The new weighting procedure has eliminated the bias for each of the six health measures. This is
expected since these health measures contribute to the propensity score estimates that are used to
make the adjustment. The weighting adjustment will succeed in reducing bias when survey
30
responses are correlated with the probability to respond and with one of the six health measures
in the model.
Weighting Adjustments
In 2006, Macro recommended a weighting adjustment that corrects for the differential nonresponse by health utilization and demographic information. After adjusting for stratum level
disproportionate sampling with base weights (or design weights) we recommended a propensity
score weighting adjustment, in which individual propensities to respond are measured with a
probability model. The estimated probabilities are then used to group the enrollees in to classes
with similar probabilities. Typically five classes (or quintiles) are formed. Within each class, the
respondents are weighted up to account for the non-respondents. These weighting adjustments
reduce potential bias to the extent that the non-respondents and respondents with similar
response probabilities are also similar with respect to the survey statistics of interest.
For the 2007 Survey of Enrollees, Macro used this weighting methodology with two
adjustments, one for survey non-response and the second for frame coverage.
The first adjustment is for survey non-response (including ineligible contact information). We
classify each sampled respondent into a non-response category (y) based on whether the
interview was a complete or an incomplete interview:
⎧0 if interview is an incomplete interview
y=⎨
⎩1 if interview is a complete interview
Using logistic regression, we estimate the probability that an enrollee completes the interview
e x′β
given their characteristics, Pr( y = 1 | x) =
, where x is a matrix of sampled enrollees each
1 + e x′β
enrollee has a set of p covariates, x ′i = (1, x1i ,...x pi ) for enrollee i, used as explanatory (or predictor)
variables, and β = ( β 0 , β 1 ,..., β p ) is a set of regression coefficients, or parameters. The predictor
variables include the sample design variables, the six administrative health measures, and
demographic variables. For this modeling, we use design weights equal to the ratio of the frame
total to the sample total in each stratum. The model results are presented in Table 14 of the
appendix.
After estimating each sampled enrollees probability of being a completed interview based on the
predictor variables, we group the respondents and non-respondents into quintiles based on their
propensity score. Within each quintile, the respondents are ratio adjusted to account for the nonrespondents.
31
0-20th percentile
20-40th percentile
40-60th percentile
60-80th percentile
80-100th percentile
Response
59251
124621
182024
265225
326065
Nonresponse
991065
925821
868391
785013
724524
Non-response
Adjustment
17.73
8.43
5.77
3.96
3.22
Each respondent’s design weight is multiplied by the adjustment factor from the quartile where
they fall to calculate the non-response adjusted weights.
The second adjustment accounted for frame ineligibility due to missing phone numbers. For this
adjustment, each enrollee in the universe is categorized based on whether they were eligible for
the frame or not (z):
⎧0 if the enrollee is not eligible for the frame
z=⎨
⎩1 if the enrollee is eligible for the frame
Similarly, we use a logistic regression model to estimate the probability that an enrollee
completes the interview given their characteristics. The model results are presented in Table 14
of the appendix.
After estimating an enrollee’s probability of being eligible for the frame based on the predictor
variables, we group the respondents and non-respondents into quintiles based on their propensity
score. Within each quintile, the respondents are ratio adjusted to account for the nonrespondents.
0-20th percentile
20-40th percentile
40-60th percentile
60-80th percentile
80-100th percentile
Eligible
794812
936100
1046483
1168110
1306494
Not eligible
627779
486517
375725
254698
116306
Frame
Adjustment
1.79
1.52
1.36
1.22
1.09
Each respondent’s non-response adjusted weight is multiplied by the adjustment factor from the
quartile where they fall to calculate the frame adjusted weights.
Finally, to account for the enrollees who are not included in the modeling due to missing
stratification variables, a final adjustment calibrates the weighted respondents to the total number
of enrollees, 7,186,950.
Bias reduction for Self Reported Utilization
As illustrated in the previous section, the non-response and coverage adjustments correct for the
biases observed for the six health measures, but this is expected since this information was used
32
to make the adjustments. These adjustments will also mitigate bias when the survey data is
correlated with the probability of response. To evaluate this, we’ve calculated VA reliance
estimates based on the survey responses with the base weight (adjusted only for disproportionate
sampling) and the non-response adjusted weight. Note, that survey response data may the survey
estimates. Note that survey responses about healthcare are subject to other forms of bias
including recall and response bias. Differences between the self-reported reliance and
administratively measured reliance should not be attributed only to non-response.
The percentage of enrollees who received outpatient care unrelated to substance abuse or mental
health and the percentage who receive VHA pharmacy benefits both decrease about 10
percentage points with the non-response adjustment. This is consistent with large drops in the
administratively measured estimates. The percentage of enrollees with inpatient visits–both
related to substance abuse or mental health and unrelated follow the same pattern as the
corrections in the administratively measured estimates. The percentage of enrollees who received
outpatient care unrelated to substance abuse or mental health is roughly the same before and after
the non-response adjustments—consistent with very little change in the administrative data.
Finally, the self-reported estimate for the percentage of enrollees who receive home health care
seems to be inflicted by extreme recall bias or response bias. The self-reported estimate is
roughly 2.5 percent, whereas the administratively measured estimate is only 0.1 percent.
Table 12. Self-reported Utilization with and without Non-response Adjustments
Base weight and non-response
adjustment
Base weight only
Est
1. Home Health Care
L95
U95
Est
L95
U95
2.91
2.67
3.15
2.48
2.25
2.71
(a) Related to MH/SA
0.46
0.46
0.46
0.82
0.68
0.96
(b) Unrelated to MH/ SA
5.66
5.33
5.99
5.18
4.85
5.50
8.20
7.83
8.57
8.14
7.72
8.55
(b) Unrelated to MH/ SA
57.14
56.44
57.84
47.01
46.18
47.83
4. VHA Pharmacy benefit
78.54
77.99
79.10
67.72
66.89
68.55
2. Inpatient treatment
3. Outpatient treatment
(a) Related to MH/SA
33
Appendix
Table 13. Frequency and Percent of Enrollees with Valid Addresses and Telephone
Numbers
Phone
Total
VISN
Priority
Group
Enrollee
Type
Full
Partial
Address
None
Full
Partial
None
1
316,460
84.18
12.63
3.19
99.84
0.16
0.00
2
197,528
77.39
19.30
3.32
99.91
0.09
0.00
3
317,488
79.33
17.40
3.26
99.85
0.15
0.00
4
418,972
85.43
12.15
2.43
99.58
0.42
0.00
5
175,200
81.12
14.19
4.68
98.93
1.07
0.00
6
381,461
77.87
18.87
3.26
99.89
0.11
0.00
7
431,230
60.68
36.12
3.20
99.93
0.07
0.00
8
603,388
67.07
30.65
2.29
99.75
0.25
0.00
9
327,825
75.90
20.70
3.39
99.78
0.22
0.00
10
262,048
75.85
20.10
4.04
99.93
0.07
0.00
11
311,908
75.82
21.14
3.04
99.82
0.18
0.00
12
307,158
74.44
22.63
2.93
99.50
0.50
0.00
15
291,310
80.56
15.99
3.45
99.92
0.08
0.00
16
584,123
73.70
23.15
3.15
99.81
0.19
0.00
17
325,362
75.45
21.45
3.10
99.94
0.06
0.00
18
300,407
59.18
37.73
3.09
99.91
0.09
0.00
19
209,591
73.88
22.59
3.53
99.51
0.49
0.00
20
313,982
71.68
23.85
4.47
99.19
0.81
0.00
21
298,688
76.05
20.69
3.27
99.49
0.51
0.00
22
384,319
64.05
31.98
3.97
99.41
0.59
0.00
23
354,576
81.95
14.96
3.09
99.88
0.12
0.00
1
870,018
71.65
24.75
3.60
99.31
0.66
0.03
2
496,102
71.38
24.51
4.12
99.30
0.65
0.06
3
939,191
71.64
23.59
4.78
99.26
0.64
0.10
4
196,791
72.80
23.44
3.76
99.55
0.41
0.04
5
2,252,477
71.39
24.87
3.74
99.51
0.42
0.07
6
253,289
69.19
28.32
2.49
99.29
0.62
0.09
7/8
2,179,059
79.54
18.16
2.30
99.48
0.43
0.09
POST
4,796,634
74.91
22.41
2.67
99.36
0.53
0.11
PRE
2,390,316
71.82
23.31
4.86
99.55
0.45
0.01
34
Appendix
Table 14. Model Results for Non-response and Frame Coverage Adjustments
Parameter
Intercept
priostrat1
priostrat2
priostrat3
priostrat4
priostrat5
priostrat6
visn_r1
visn_r2
visn_r3
visn_r4
visn_r5
visn_r6
visn_r7
visn_r8
visn_r9
visn_r10
visn_r11
visn_r12
visn_r15
visn_r16
visn_r17
visn_r18
visn_r19
visn_r20
visn_r21
visn_r22
enroll_pr
gender_m
age
dupflag_y
hhuse_y
ip_mhsause_y
ip_nonmhsause_y
op_mhsause_y
op_nonmhsause_y
rxuse_y
Non-response adjustment
Std.
Wald chiPEst Error
square
value
-3.41
0.01
131092.72 <.0001
0.18
0.00
1958.03 <.0001
0.18
0.01
1259.68 <.0001
0.04
0.00
93.38 <.0001
-0.13
0.01
288.48 <.0001
0.00
0.00
0.98 0.3211
0.11
0.01
199.54 <.0001
-0.35
0.01
2639.10 <.0001
-0.23
0.01
797.06 <.0001
-0.67
0.01
7938.74 <.0001
-0.26
0.01
1753.34 <.0001
-0.59
0.01
4040.95 <.0001
-0.23
0.01
1229.62 <.0001
-0.61
0.01
6267.25 <.0001
-0.60
0.01
9068.04 <.0001
-0.22
0.01
997.85 <.0001
-0.17
0.01
529.52 <.0001
-0.09
0.01
158.86 <.0001
-0.40
0.01
3013.26 <.0001
-0.08
0.01
153.34 <.0001
-0.17
0.01
783.66 <.0001
-0.39
0.01
2926.35 <.0001
-0.29
0.01
1348.91 <.0001
-0.20
0.01
593.20 <.0001
0.01
0.01
0.53 0.4684
-0.18
0.01
668.56 <.0001
-0.39
0.01
2855.74 <.0001
-0.15
0.00
3105.93 <.0001
-0.05
0.01
68.34 <.0001
0.03
0.00
76720.01 <.0001
-0.65
0.01
14905.55 <.0001
0.57
0.03
309.31 <.0001
-0.51
0.02
992.23 <.0001
-0.04
0.01
64.82 <.0001
0.61
0.01
6215.22 <.0001
0.68
0.00
19883.86 <.0001
0.28
0.00
3757.01 <.0001
35
Frame Coverage Adjustment
Std.
Wald chiPEst Error
square
value
-0.16
0.01
560.13 <.0001
0.03
0.00
71.47 <.0001
0.03
0.00
51.35 <.0001
-0.01
0.00
17.03 <.0001
-0.28
0.01
2317.73 <.0001
-0.29
0.00
14164.14 <.0001
-0.08
0.00
277.57 <.0001
0.11
0.01
296.45 <.0001
-0.33
0.01
2120.28 <.0001
-0.27
0.01
1791.74 <.0001
0.20
0.01
1010.62 <.0001
-0.08
0.01
116.68 <.0001
-0.23
0.01
1492.99 <.0001
-1.07
0.01
37958.58 <.0001
-0.90
0.01
28967.39 <.0001
-0.32
0.01
2701.58 <.0001
-0.38
0.01
3527.37 <.0001
-0.37
0.01
3629.45 <.0001
-0.46
0.01
5735.48 <.0001
-0.14
0.01
453.51 <.0001
-0.46
0.01
7320.54 <.0001
-0.39
0.01
4093.36 <.0001
-1.14
0.01
37743.90 <.0001
-0.43
0.01
4114.95 <.0001
-0.53
0.01
7653.25 <.0001
-0.37
0.01
3411.77 <.0001
-0.90
0.01
25249.28 <.0001
-0.20
0.00
10614.69 <.0001
0.08
0.00
358.93 <.0001
0.03
0.00
171011.18 <.0001
15.69
5.54
8.03 0.0046
0.15
0.03
26.77 <.0001
-0.02
0.01
5.31 0.0212
-0.17
0.00
1582.58 <.0001
-0.04
0.01
56.43 <.0001
0.07
0.00
375.65 <.0001
0.00
0.00
0.47 0.4948
| File Type | application/pdf |
| File Title | Supplementary Analysis and Technical Assistance for the 2007 SoE Final Report |
| Subject | 2007 Survey of Enrollees |
| Author | Department of Veterans Affairs |
| File Modified | 2008-06-10 |
| File Created | 2008-06-10 |