Methodological Experiments

ANALYSIS OF METHODOLOGICAL
EXPERIMENTS AND NON-RESPONSE BIAS
FOR THE 2012 VHA SURVEY OF VETERAN
ENROLLEES’ HEALTH AND RELIANCE
UPON VA
FINAL REPORT

--- Not For Distribution ---

Submitted to:
Office of the Assistant Deputy Under Secretary for Health for Policy and Planning

Prepared by:

126 College Street
Burlington, Vermont 05401

November 30, 2012

TABLE OF CONTENTS
Background ____________________________________________________________ 1
History of Survey of Enrollees Bias Assessments _____________________________ 1
Summary of Methodological Experiments, 2006–2011 ________________________ 2
2012 Experiments _______________________________________________________ 4
Enrollee Records without a Valid Telephone Number _________________________
Increased Coverage__________________________________________________
Alternate Response Modes ____________________________________________
Mail Survey Requests ________________________________________________
Non-Response Follow-Up _____________________________________________
Nonworking Telephone Numbers _______________________________________

4
6
7
7
7
7

Impact of Mail Option on the Survey Estimates ______________________________ 7
Adding a Web Survey _________________________________________________ 11
Sample Design ________________________________________________________ 14
Sampling Frame _____________________________________________________ 14
Sampling Design and Interview Outcomes _________________________________ 15
Frame Eligibility ______________________________________________________ 15
Valid Contact ________________________________________________________ 16
Non-Response _______________________________________________________ 16
Bias Analysis __________________________________________________________ 18
1. Long-Term Care Benefits ____________________________________________ 22
2. Inpatient Treatment ________________________________________________ 26
Reasons Related to Mental Health or Substance Abuse (MHSA) ______________ 26
Reasons Unrelated to MHSA__________________________________________ 26
3. Outpatient Treatment ______________________________________________ 30
Outpatient Treatment Unrelated to MHSA ______________________________ 30
Outpatient Treatment Related to MHSA ________________________________ 30
4. VHA Pharmacy Services _____________________________________________ 35
Survey Weighting ______________________________________________________ 37
Design Weights ______________________________________________________ 37
Non-Response Adjustment _____________________________________________ 38
Discussion and Recommendations ________________________________________ 41

BACKGROUND
The Department of Veterans Affairs (VA) serves American Veterans by providing primary and
specialized health care as well as related medical and social support services. VA also
administers the country’s largest, most comprehensive, integrated health care system. Veterans
Health Administration (VHA) enrollment files show that the number of Veterans turning to
Veterans Health Administration (VHA) for health care increases every year. Enrollment in VHA
will likely continue to increase due to factors such as the nation’s economy, shifts in Veteran
population demographics, and rising health care costs.
VHA’s ability to enroll Veterans is regulated by the Veteran’s Health Care Eligibility Reform Act
of 1996 (Public Law 104-262). This law instituted a priority-based enrollment system designed
to balance service to those Veterans most in need with the need to control health care costs and
system burden. Under this law, the number of priority levels to which VHA can deliver care is a
function of annual funding levels and utilization patterns.
To meet enrollees’ health care needs, VHA also must understand fully the reliance of enrolled
Veterans on VHA health care services and programs compared to their use of non-VA services
and programs (known as “VA reliance”). Data gathered by the VHA Survey of Veteran Enrollees’
Health and Reliance Upon VA (Survey of Enrollees) is a major contributor to VA’s understanding
of enrollee reliance. The Survey of Enrollees was developed to gather a variety of information
used to determine the relationship between utilization patterns and demographic and
socioeconomic characteristics of VHA enrollees.
The Survey of Enrollees data inform health care budgets, assist VA with its annual enrollment
decisions, and inform the VA Enrollee Health Care Projection Model (EHCPM). Forecasts
developed from this model have a number of purposes, such as budgeting, and scenario-based
policy and planning analyses.
VHA has conducted ten cycles of the Survey of Enrollees (1999, 2000, 2002, 2003, 2005, 2007,
2008, 2010, 2011, and 2012). Through 2011, the survey methodology could be summarized as
an English-only, 15- to 20-minute survey available via Computer-Assisted Telephone
Interviewing (CATI), using a stratified sampling design with the objective of obtaining 42,000
interviews. In 2012, VHA added mail and Computer-Assisted Web Interviewing (CAWI) modes
as part of VHA’s ongoing experiments to reduce survey response bias.
ICF International, Inc. (ICF) has provided technical and data collection services to VHA in support
of the Survey of Enrollees since 2005. This analysis of methodological experiments and nonresponse bias pertains to the 2012 data collection period from March 28 through June 21, 2012.

History of Survey of Enrollees Bias Assessments
Any information collection from the general public and conducted or sponsored by a Federal
agency requires periodic Office of Management and Budget (OMB) clearance. As part of the
Fiscal Year (FY) 2006 OMB clearance package, VHA was tasked with both conducting a nonresponse bias assessment, as well as examining the quality of the sampling frame. In 2006, VHA
and ICF met with OMB to discuss the non-response analysis and agreed to develop methods to
improve the survey program. OMB granted clearance to VHA but required that VHA improve
the design, starting with the 2007 survey. Since then, the Survey of Enrollees has:

1

 Added a pre-survey notification letter sent from the Under Secretary for Health. The
letter described the survey’s purpose, explained that ICF is conducting the study on
VHA’s behalf, and provided a number to call with questions or to complete the survey;
 For Veterans with missing phone numbers, added a customized letter with an inbound
phone number to call to complete the survey;
 Experimented with reverse-phone number look-up based on address information;
 Increased the maximum number of call attempts from six to seven; and
 Improved the weighting methodology to use a propensity score adjustment based on
demographics and health care utilization administrative records.
Bias in the Survey of Enrollees exists in two forms, 1) differences between enrollees with and
without a valid phone number (coverage bias), and 2) differences between those who respond
to the telephone survey and those who do not (non-response bias). Thus, in 2012, VHA
introduced a mail survey to offer participation to enrollees without a phone number or with a
nonworking number. VA also experimented with the use of a mail survey for telephone nonresponders. Finally, as an alternative to mail or telephone modes, VHA offered a Web survey.
This report provides an overview of the methodological experiments conducted in previous
survey cycles, an analysis of the 2012 introduction of a multi-mode survey format, and an
overall bias analysis.

Summary of Methodological Experiments, 2006–2011
For the past five years, we have conducted a bias assessment and have evaluated the results of
methodological experiments designed to reduce bias. In 2006, ICF used the 2005 data to
examine the survey process and potential biases resulting from missing or outdated contact
information and survey non-response—including both the inability to make contact and the
effects of respondent refusals. The report, submitted to the Office of Management and Budget
(OMB), included several recommendations to improve the research design.
The 2007 Survey of Enrollees included several methodological experiments to gauge the impact
of design enhancements. These experiments included sending pre-survey notification letters to
potential respondents by the Under Secretary for Health; and, extending the maximum number
of call attempts from 6 to 10. The results of these experiments are documented in the 2007
report, “Supplementary Analysis and Technical Assistance for the 2007 Annual Survey of
Veteran Enrollees Health and Reliance on VA,” dated February 14, 2008. The response rate
among the experimental treatment group (pre-survey notification letter and 10 call attempts)
doubled that of the control group (no pre-survey notification letters and 6 call attempts)—43.3
percent vs. 21.4 percent, respectively. Based on the experimental evidence, ICF recommended
that VHA adopt both of these design enhancements for the 2008 Survey of Enrollees. VHA
approved sending pre-survey notification letters and increasing the maximum call attempts to
seven (concern for increased respondent burden prevented an increase to 10).
In 2008, VHA approved a methodological experiment to improve sample frame coverage:
reverse telephone look-up directories that used respondent addresses to obtain valid telephone
numbers from a sample of 62,516 enrollees. This process resulted in 59,426 potential
respondents (95 percent coverage), and this group yielded 12,765 completed surveys.

2

The 2010 Survey of Enrollees followed the same methodology as 2008—including the reverse
phone number look-up from a sample of 62,515 enrollees. Again, the results indicated that
address matching improved contact information quality. This process resulted in 61,376
potential respondents (98 percent coverage), and this experimental group yielded 16,851
completed surveys.
For 2011, the plan for the Survey of Enrollees also included reverse telephone look-ups.
Unfortunately, this service was not implemented in 2011 because the address matching vendor
was not able to comply with the project’s security requirements. A 2011 experiment included a
tailored pre-survey notification letter sent to enrollees with a known address but unknown
telephone number, as listed in the database. This letter asked the enrollee to call ICF to conduct
the survey. This experiment marked the first time that Veterans with unknown telephone
numbers were included in the frame. This test yielded 244 interviews from 15,339 total
enrollees without phone numbers. While relatively few, these respondents represent Veterans
who would not otherwise be included in the survey results.

3

2012 EXPERIMENTS
Enrollee Records without a Valid Telephone Number
The Survey of Enrollees has been conducted as a telephone interview since its inception in 1999.
Enrollees with invalid telephone numbers (e.g., missing or incorrect area code) or no telephone
were not included, which was a source of bias. In 2012, VHA addressed this design bias by
developing an experimental mail survey that was sent to all enrollees without a valid telephone
number. In addition to adding a mail survey, VHA offered an experimental Web option for the
first time. Enrollees were sent a pre-survey notification letter informing them of the survey and
offering the opportunity to complete the survey online. Figure 1, below, presents the 2012
allocation of sample.

4

Figure 1. Sample Allocation

Enrollee
Population
9,554,202

Enrollee Population
Eligible
8,695,824

In frame
8,013,308

VA sampled
Sent to ICF
419,991

Response
38,389

VA not sampled
7,593,317

Not in sample

159,577

260,414

Address only

Successful Lookup
622

145,348

Eligible contact
112,331

Not in frame
682,516

In sample

Address and phone

Enrollee Population
Ineligible
858,378

14,229

CATI survey

Mail survey

145,970

13,607

Nonworking
number
22,085

Ineligible contact
11,554

Request mail
4,769

Non-response
69,173

Sent mail survey
1,586

Sent mail survey
15,078

Response
4,007

Non-response
9,600

Sent mail survey
15,761

Response
3,388

Response
626

Response
2,705

Non-response
960

Non-response
12,373

Non-response
12,373

5

The shaded boxes in the figure above represent design
features added in 2012. The lighter boxes represent the
design as conducted in past iterations. The addition of mail
and Web versions introduces many potential benefits:
1. Increases coverage to include enrollees unavailable to
participate by phone;
2. Allows an additional response mode for those who
prefer to take a self-administered survey online or on
paper;
3. Offers a non-response follow-up option for those who
did not respond to the phone survey; and,
4. Provides a second contact option for enrollees whose
telephone number is no longer working.

Note that throughout this report, we refer
to the “telephone survey” (CATI) and the
“mail survey.” The former term
designates the traditional telephone
design, as indicated by the lighter boxes in
Figure 1; and the latter term designates
features introduced by the mail survey, as
indicated by the shaded boxes in Figure 1.
The terms “telephone survey” and “mail
survey” do not indicate whether the
respondent answered the survey by
telephone or mail. For example, some
Veterans who were sent a mail survey
called ICF and either completed an
inbound CATI survey or provided
telephone information that resulted in an
outbound CATI survey.

We examine each of these benefits below.

Increased Coverage
One of the main concerns about the Survey of Enrollees design was coverage bias. Coverage
bias can occur when population sub-groupsin this case, enrolleesare systematically
excluded from the sample frame due a variety of factors. Among them, for telephone surveys, is
the lack of a telephone number through which to contact the respondent. Coverage bias is the
deviation of observed data values from the actual values due to differences between covered
and non-covered cases. Telephone surveys suffer from coverage bias because they cannot
include enrollees without telephones. In 2010, VHA experimented with reverse telephone
number look-ups based on the enrollee’s name and address (via LexisNexis, a business research
service). This look-up found a telephone number for 5,731 of 6,870 enrollees (83 percent)
without a telephone, and resulted in 870 interviews.
The telephone look-ups improved the sample frame and expanded coverage; however, the
process was discontinued in 2011 because LexisNexis was unable to comply with VA data
security requirements. Instead, ICF mailed a pre-survey notification letter to all sampled
Veterans without a phone number. The pre-survey notification letter explained that we wanted
to include them in the study, but we had no telephone number to do so. The letter was nearly
identical to the pre-survey notification letter sent to Veterans for whom we had a telephone
number; and, it contained the following additional text: We do not have a current telephone
number for you but would like to include you in our study. To participate please call ICF
Macro at 1-888-871-0345. The letter was sent to 15,339 enrollees. A total of 244 of these
enrollees called in to complete the survey.
As part of the 2012 survey, ICF administered an experimental mail survey to all sampled
enrollees without a phone number. The mail protocol included a pre-survey notification
advance letter, followed by a cover letter and survey packet, a postcard reminder, and a second
cover letter and survey packet. As expected, the mail mode resulted in many more completed
surveys than were attained in previous years. In total, 14,229 (8.92 percent) enrollees did not
have a valid telephone number. We were able to determine a phone number via area code
6

look-ups for 622 cases (4.37 percent), leaving 13,607 selected enrollees in the mail survey.
From the mail sample, we obtained 4,007 completed surveys across all response modes, a 29.4
percent completion rate. The majority of these surveys were completed via mail (3,632). In
addition, 327 of these enrollees completed the survey via Web, and 48 via telephone.

Alternate Response Modes
The addition of a mail survey component also allowed for alternative response options. In
previous iterations of the Survey of Enrollees, some enrollees had requested a mail version of
the survey. This was not possible in prior years, since no mail version existed. The addition of a
mail survey component allowed us to honor these requests, and provided a means to follow up
on non-response and non-working numbers in the CATI study.

Mail Survey Requests
In 2012, 1,586 enrollees requested and received mailed surveys, resulting in 556 completed
returns by mail, a 35.1 percent completion rate for this group. Another 19 surveys from this
group were completed by Web, and 51 surveys were completed by telephone, for a total of 626
responses, a 39.5 percent response rate.

Non-Response Follow-Up
Non-response occurs when an enrollee refuses, does not have time, or cannot be reached to
conduct the telephone interview. In 2012, there were 69,173 non-responding enrollees. A
subsample of 15,078 telephone non-responders was sent a mail survey. This resulted in 2,705
surveys completed across all response modes, a 17.9 percent response rate.

Nonworking Telephone Numbers
A final benefit of the mail survey is that it can be used to reach enrollees with non-working
numbers. About 15 percent of the telephone numbers listed in the frame was nonworking.
Since we had addresses for these enrollees, we could send a mail survey to seek a response. We
mailed surveys to 15,761 enrollees with non-working numbers and received 3,388 responses
across all response modes, a 21.5 percent response rate.

Impact of Mail Option on the Survey Estimates
To understand how the mail survey affects the survey response options, we divided the
completed interviews into four groups based on the data collection protocol. The first group
represents a control, in that the treatment of enrollees with valid telephone numbers is very
similar to the historical telephone-only design. The other three groups represent treatment
groups; here, we implemented data collection protocols that were made possible by the
inclusion of a mail response channel. It is important to note that enrollees in each group could
respond to the survey via any one of the four response channels—outbound CATI, inbound CATI,
Web, or mail.
For example, an enrollee with a valid phone number (Group 1) could complete a survey via
outbound CATI when contacted by an ICF agent; via inbound CATI by calling the ICF number
printed on the pre-survey notification letter and displayed on caller ID; via Web using
7

instructions printed on the pre-survey notification letter; or, via mail if the enrollee requested a
mail survey.
Table 1 provides counts of interviews completed by each of these groups broken out by
response channel. The four groups of enrollees were:
Group 1 – Valid Phone; Enrollees with valid telephone numbers—this group most closely
represents the historical telephone-only design used through 2011. It is slightly different than
previous administrations of the survey in that enrollees could respond via Web.
Group 2 – Non-Working Phone; Enrollees with a non-working telephone number—this group
started in the telephone survey, but was sent a mail survey once it became clear that the
telephone number was nonworking.
Group 3 – Mail Request and Non-Response; Enrollees with a valid telephone number who
requested a mail survey or who were telephone non-responders—this group was sent a mail
survey upon request or when ICF satisfied dialing protocol without obtaining a completed
interview.
Group 4 – Invalid Phone; Enrollees with an invalid telephone number listed on the frame—this
group was administered a mail survey at the outset.
Table 1. Treatment Group by Response Channel
Description
CATI Eligible Response
CATI Eligible Non-Response,
Mail Response
CATI Non-Working Number,
Mail Response
CATI Eligible Requested Mail,
Mail Response

Mail Survey, Response
Total Responses

Group
Group 1

Outbound Inbound
Phone
Phone
30,691
1,724

Web

Mail

5,974

0

Group 2

1

13

13

2,678

Group 3

0

0

198

3,190

Group 3

12

39

19

556

12

39

217

3,746

4
30,708

44
1,820

327
6,531

3,632
10,056

Group 3
Total
Group 4

We computed separate estimates for these four groups to demonstrate how they change based
on each of the new features available with the multi modes. These characteristics were chosen
since they are both important characteristics of enrollees and of interest to VHA.

8

Table 2. Comparison of Select Estimates of Coverage, Access, and Health Behaviors for Each Response Group
New Mail-based
CATI Study Protocol
Protocol

All Enrollees
58.4
(57.8, 59.0)
Medicaid coverage for some health
6.6
care (%)
(6.3, 7.0)
Coverage by another individual or
26.4
group health plan (%)
(25.8, 26.9)
Use VA services to meet….(%)
01 All of my health care
31.8
needs
(31.2, 32.4)
02 Most of my health care
17.9
needs
(17.4, 18.4)
03 Some of my health care
29.8
needs
(29.2, 30.4)
04 None of my health care
17.0
needs
(16.5, 17.5)
05 I have no health care
3.5
needs
(3.3, 3.7)
Self-Reported General Health (%)
10.7
01 Excellent
(10.3, 11.1)
25.2
02 Very good
(24.7, 25,8)
30.8
03 Good
(30.2, 31.4)
22.0
04 Fair, or
(21.5, 22,6)
11.1
05 Poor
((10.7, 11.5)
Employment Status (%)
17.7
01 Employed Full-time
(17.3, 18.1)
2.8
02 Self-employed full-time
(2.6, 3.0)
5.0
03 Employed part-time
(4.7, 5.3)
3.0
04 Self-employed part-time
(2.8, 3.3)
05 Unemployed, looking for
6.2
work, or laid off
(5.9, 6.4)
06 Currently not employed
65.2
– either retired, a
(64.6, 65.8)
homemaker, student, etc.
17.7
Current Smokers (%)
(17.2, 18.2)
Medicare coverage (%)

Valid Phone
(Group 1)
59.5
(58.8, 60.2)
6.9
(6.5, 7.3)
25.1
(24.4, 25.7)

Non-working
Phone
(Group 2)
51.0
(48.6, 53.5)
6.4
(5.2, 7.6)
31.4
(29.1, 33.7)

Mail Request
& Nonresponse
(Group 3)
59.9
(57.5, 62.3)
4.8
(3.7, 5.8)
26.3
(24.2, 28.4)

Invalid Phone
(Group 4)
51.5
(49.2, 53.8)
6.0
(4.9, 7.1)
38.0
(35.7, 40.2)

33.1
(32.4, 33.7)
19.2
(18.6, 19.7)
31.1
(30.5, 31.8)
13.5
(13.0, 14.0)
3.2
(2.9, 3.4)

27.0
(24.8, 29.2)
10.9
(9.4, 12.4)
25.1
(22.9, 27.3)
31.5
(29.2, 33.8)
5.5
(4.4, 6.7)

34.9
(32.5, 37.2)
18.3
(16.4, 20.3)
29.3
(27, 31.6)
14.8
(13.1, 16.5)
2.7
(1.9, 3.6)

17.6
(15.7, 19.5)
9.3
(7.8, 10.8)
17.8
(16.0, 19.6)
49.0
(46.6, 51.3)
6.4
(5.2, 7.5)

11.1
(10.6, 11.5)
24.8
(24.1, 25.4)
30.1
(29.4, 30.8)
22.1
(21.5, 22.7)
12.0
(11.5, 12.4)

9.0
(7.6, 10.4)
27.6
(25.4, 29.8)
32.5
(30.2, 34.8)
22.4
(20.3, 24.4)
8.6
(7.2, 10.0)

8.6
(7.2, 10.0)
25.7
(23.6, 27.9)
35.5
(33.1, 37.8)
22.7
(20.7, 24.7)
7.5
(6.2, 8.8)

11.1
(9.5, 12.7)
28.4
(26.3, 30.5)
33.5
(31.3, 35.7)
20.2
(18.3, 22.1)
6.8
(5.6, 8.0)

16.0
(15.5, 16.4)
2.7
(2.5, 3.0)
5.0
(4.7, 5.3)
3.1
(2.8, 3.4)
6.4
(6.0, 6.7)
66.8
(66.2, 67.5)

27.2
(25.0, 29.4)
2.9
(2.0, 3.8)
4.5
(3.5, 5.5)
2.4
(1.7, 3.2)
5.4
(4.3, 6.5)
57.6
(55.1, 60.1)

19.1
(17.3, 21.0)
3.5
(2.5, 4.5)
4.8
(3.7, 5.8)
3.3
(2.4, 4.2)
5.4
(4.2, 6.6)
63.9
(61.5, 66.3)

28.8
(26.8, 30.9)
2.9
(2.1, 3.7)
6.7
(5.3, 8.0)
2.8
(2.0, 3.6)
5.1
(4.1, 6.1)
53.8
(51.4, 56.2)

17.4
(16.9, 18.0)

21.7
(19.6, 23.7)

16.9
(15.1, 18.8)

17.9
(16.1, 19.7)

Note: 95% Confidence Intervals
Note: Rao-Scott Chi-square tests of association were run on the measures reported in Table 1. In all cases these tests showed a highly significant
association between the response group and the enrollee characteristic.

9

In most respects, respondents with valid telephone numbers (Group 1) and respondents who requested
a mail survey and CATI non-responders (Group 3) have similar estimates. Less than 20 percent of these
two groups are employed full-time, and approximately 65 percent of these two groups are not currently
in the labor force. The other two groups, respondents who received a mail survey because of nonworking numbers (Group 2) and respondents who received a mail survey because we had no telephone
number (Group 4), have a higher percentage of respondents employed full-time, and a lower percentage
not in the labor force. Compared to Groups 2 and 4, Groups 1 and 3 also have a higher percentage of
enrollees covered by Medicare, and a smaller percentage covered by private health care. Over 50
percent of enrollees in Groups 1 and 3 receive all or most of their health care needs from the VA. Nearly
50 percent of the respondents in Group 4 do not use VA for their health care needs. This group has the
highest percentage of enrollees, 38 percent, who are covered by private health insurance. Group 2
reported the highest smoking rates, with a 21.7 percent incidence of current smoking.
Although general patterns of self-reported health were similar across groups, the exception was
enrollees self-reporting poor health, the highest being Group 1 enrollees at 12 percent.
We also examined a selected number of Key Driver questions in which the respondents were read a
series of statements, and then asked if they: 1 completely agree, 2 agree, 3 neither agree nor disagree, 4
disagree, or 5 completely disagree. We present average scores in Table 3, below. Lower scores indicate
higher agreement; higher scores indicate lower agreement.
Table 3. Comparison of Select Key Driver Responses for Each Response Group
CATI Study Protocol

d11c: VA offers Veterans like me
the best value for our health care
dollar
d12b: Veterans like me who use
VA are satisfied with the health
care they receive
d13b: Veterans like me can get in
and out of an appointment at VA
in a reasonable time
d14d: I understand how my VA
health benefits works
d15f: It is easy to get to my local
VA facility
d16c: I would only use VA if I did
not have access to any other
source of health care

New Mailbased
Protocol

Valid Phone
(Group 1)

Non-working
Phone
(Group 2)

Mail Request &
Non-response
(Group 3)

Invalid
Phone
(Group 4)

2.08
(2.06, 2.09)

2.14
(2.09, 2.19)

1.87
(1.82, 1.91)

2.35
(2.30, 2.40)

2.13
(1.12, 2.14)

2.22
(2.18, 2.27)

1.95
(1.90, 1.99)

2.39
(2.34, 2.43)

2.22
(2.21, 2.24)

2.40
(2.35, 2.45)

2.06
(2.10, 2.11)

2.59
(2.53, 2.64)

2.34
(2.33, 2.36)
2.17
(1.16, 2.19)

2.75
(2.70, 2.80)
2.27
(2.21, 2.31)

2.41
(2.36, 2.46)
1.97
(1.92, 2.02)

2.95
(2.90, 3.00)
2.39
(2.33, 2.44)

2.86
(2.84, 2.88)

2.77
(2.70, 2.83)

3.11
(3.04, 3.17)

2.66
(2.60, 2.73)

Note: 95% Confidence Intervals

Groups 1 and 3 provided the most favorable ratings for VA. The least favorable ratings came from the
enrollees in Group 4. This group reported less use of VA for their health care needs than did the
enrollees in the other three groups.

10

Adding a Web Survey
Another addition to the 2012 survey was a Web option. The Web survey offers enrollees the
opportunity to respond online, instead of by mail or by telephone. All enrollees were sent a pre-survey
notification letter that stated the following:
ICF International, a respected research and management consulting company, is partnering with
VHA to conduct the survey. Our goal is to make it as easy as possible for you to participate in
this survey. Within the next few days, ICF International will contact you by telephone to
complete the survey. Alternatively, you may contact ICF International directly to arrange to
participate in the survey at 1-866-784-7219 referencing your passcode noted below. You may
also complete the survey online by going to the Web site (http://www.vhasurvey.com),
entering your unique passcode (Passcode: < MASTER ID>), and following the instructions.
Enrollees have always had the opportunity to call in to complete the survey, but 2012 was the first time
they could complete it online. In 2011, telephone respondents completed 43,633 surveys, of which
3,085 (7.1 percent) were completed via inbound calls. Inbound calling was also an option in 2012, but it
was utilized at about half the 2011 rate—possibly due to the availability of alternate response channels
in 2012. Of the 38,389 completes from enrollees in the telephone-only portion of the survey, only
1,724 (4.5 percent) completed the survey via inbound calling.
Another 5,974 (15.6 percent) enrollees chose to respond via Web; this was more than three times the
percentage of those responding via inbound telephone. The combined Web and inbound telephone
response for 2012 (7,698) totaled 20 percent of the 38,389 interviews obtained from Group 1.
Including the enrollees who were sent a mail survey, 6,531 enrollees completed the survey online. This
total represents 13.3 percent of all completed interviews and demonstrates an important new channel
to engage this population. Web surveys can reduce call center labor hours and, potentially, data
collection timelines.
Table 4, on page 12, shows the same enrollee characteristics depicted in Table 2, split out by response
channel, or survey mode. While we can see differences in characteristics across modes, they cannot
necessarily be attributed to mode effects.
Due to the study design, mode effects are confounded with the effects of reaching different subpopulations. For example, among enrollees with data collected via an outbound telephone interview,
the full time employment rate was 15%, compared to 23% and 25% for enrollees responding via mail
and Web, respectively. It is possible that this is due to population effects—those employed full time
may find it easier to respond via web and mail, and are less likely to be found at home by an outbound
call. On the other hand, differences observed in self report health status may be due to mode effects—
speaking with a person may prompt different responses to these items than filling out a survey in
private.
As we design the next cycle of this study, we can consider the possibility of embedding controlled
experiments in the survey design that would allow us to tease apart the mode and population effects.

11

Table 4. Mode Analysis
Outbound Phone
Percent

Inbound Phone

65.26

Lower 95%
Confidence
Limit
62.13

Upper 95%
Confidence
Limit
68.38

Upper 95%
Confidence
Limit
61.22

Percent

7.23

8.11

7.97

6.20

23.70

25.07

23.43

01 All of my
34.18
health care
needs
02 Most of my
19.21
health care
needs
03 Some of my
30.65
health care
needs
04 None of my
12.58
health care
needs
05 I have no
3.38
health care
needs
Self-Reported General Health (%)

33.43

34.93

18.57

01 Excellent

Mail

52.76

Lower 95%
Confidence
Limit
51.07

Upper 95%
Confidence
Limit
54.44

54.76

Lower 95%
Confidence
Limit
53.34

Upper 95%
Confidence
Limit
56.18

9.74

3.15

2.56

3.73

5.81

5.14

6.47

20.67

26.18

29.71

28.19

31.24

31.05

29.73

32.36

35.05

31.92

38.18

26.50

25.01

27.98

27.38

26.08

28.68

19.86

18.41

15.97

20.85

18.43

17.13

19.74

13.18

12.18

14.18

29.89

31.41

30.72

27.61

33.83

32.98

31.36

34.59

24.21

22.96

25.47

12.03

13.13

12.93

10.70

15.16

19.63

18.28

20.98

30.34

29.04

31.65

3.08

3.67

2.89

1.71

4.07

2.47

1.93

3.01

4.88

4.23

5.52

11.06

10.53

11.59

12.87

10.53

15.20

10.40

9.34

11.46

9.59

8.70

10.47

02 Very good

23.52

22.82

24.23

25.39

22.48

28.31

30.76

29.17

32.34

26.70

25.41

27.99

03 Good

29.47

28.73

30.22

26.98

24.10

29.85

33.96

32.34

35.57

33.83

32.46

35.20

04 Fair, or

22.80

22.13

23.48

21.32

18.71

23.93

18.78

17.47

20.08

22.15

20.94

23.35

05 Poor

13.14

12.59

13.69

13.44

11.19

15.70

6.11

5.32

6.90

7.74

6.96

8.52

Medicare
60.45
coverage (%)
Medicaid
7.67
coverage for
some health care
(%)
Coverage by
24.39
another
individual or
group health
plan (%)
Use VA services to meet….(%)

Lower 95%
Confidence
Limit
59.68

Web
Percent

Percent

12

Outbound Phone
Percent

Inbound Phone

Web

Lower 95%
Confidence
Limit

Upper 95%
Confidence
Limit

Percent

Lower 95%
Confidence
Limit

Upper 95%
Confidence
Limit

15.03

14.51

15.56

9.81

7.99

11.63

2.65

2.38

2.91

2.20

1.24

4.71

4.35

5.07

4.79

2.91

2.62

3.20

6.78

6.38

67.92

18.58

Mail

Percent

Lower 95%
Confidence
Limit

Upper 95%
Confidence
Limit

Percent

Lower 95%
Confidence
Limit

Upper 95%
Confidence
Limit

22.83

21.51

24.15

24.49

23.28

25.70

3.15

3.48

2.83

4.14

2.98

2.45

3.52

3.36

6.22

6.22

5.38

7.07

5.25

4.57

5.92

2.41

1.47

3.36

4.25

3.50

5.00

2.78

2.29

3.27

7.17

6.12

4.63

7.62

4.83

4.10

5.57

5.15

4.51

5.79

67.18

68.67

74.67

71.89

77.44

58.38

56.72

60.04

59.35

57.92

60.79

17.96

19.20

17.66

15.16

20.16

11.92

10.84

12.99

19.25

18.10

20.41

Employment Status (%)
01 Employed
Full-time
02 Selfemployed fulltime
03 Employed
part-time
04 Selfemployed parttime
05 Unemployed,
looking for work,
or laid off
06 Currently not
employed –
either retired, a
homemaker,
student, etc.
Current Smokers
(%)

13

SAMPLE DESIGN
Sampling Frame
VHA provided a random stratified sample of 419,991 records from its enrollee database as follows:






VHA extracted the entire universe of enrollees who were listed as of September 30, 2011; this
list included both institutionalized and non-institutionalized Veterans enrolled in VA health care
and contained 9,554,202 records
After dropping deceased, enrollment cancelled/declined, and enrollment ineligible cases (known
as “current”) enrollees, the VHA enrollment file contained 8,695,824 records
VHA eliminated all records meeting the following criteria:
o Lacking a valid address;
o Not eligible for VA Healthcare;
o Not in the U.S. or Puerto Rico; and,
o Missing one of the stratification variables listed below.
This left a file of 8,013,308 enrollees to be stratified by OEF/OIF/OND status, pre/post-enrollee
status, priority group, and Veteran Integrated Service Network (VISN).

ICF then randomly selected a subsample of these records to meet the target sample sizes in each
stratum. ICF released records into the study as needed, using a random selection algorithm.
Operationally, this process was based on monitoring the number of completed interviews periodically
during fielding. We compared the estimated sample yield (that is, the number of completed interviews
we predicted we would obtain from the sample at a given point in the study) to the target number
required by the sampling plan. Enrollee records were drawn randomly from the set of records provided
by VA and released into the study for calling/mailing based on this analysis.
For cycles prior to 2008, the sampling frame had been stratified into 294 strata based on VISN (21),
1
enrollee type (2), and priority group (1–6 and 7/8). To increase the data utility for OEF/OIF/OND
enrollees, VHA added additional strata based on OEF/OIF/OND status in 2008 and repeated this in 2010
and 2011.
For 2012, there were two modifications to the stratification. First, VHA combined the enrollee type
stratum with the OEF/OIF/OND status stratum because very few enrollees were “pre” and
OEF/OIF/OND. The new stratification variable was:
1. Post, not OEF/OIF/OND;
2. Pre, not OEF/OIF/OND; and,
3. OEF/OIF/OND.
Second, priority group 8 was stratified separately from priority group 7. In previous years, priority
groups 7 and 8 were a combined stratum.

1

Pre-enrollees are defined as those Veterans who used the VA Health Care System during fiscal years 1996, 1997, or 1998 or
enrolled during the first six months of enrollment (October 1, 1998 to March 31, 1999). All other enrolled Veterans are
considered Post-enrollees.

14

The stratification and sample allocation were based on achieving target sample sizes for OEF/OIF/OND
enrollees, VISN, enrollee type, and priority. Each of the 21 VISN was allocated 2,000 interviews as
follows:
1. 875 allocated to pre-enrollees, with
a. 125 interviews each for priority groups 1 through 5, for a total of 625 interviews
b. An additional 250 interviews proportionally allocated across priority groups 6
through 8
2. 875 allocated to post-enrollees, with,
a. 125 interviews each for priority groups 1 through 5, for a total of 625 interviews
b. An additional 250 interviews proportionally allocated across priority groups 6
through 8
c. Within each priority group, the sample split proportionally between OEF/OIF/OND
and non-OEF/OIF/OND enrollees.
3. 250 oversample for OEF/OIF/OND proportionally allocated across priority groups 1 through
8
A total of 145,970 enrollees with a valid telephone number (including 622 telephone look-ups) were
sampled to meet the sample size requirements in all strata.
This was much higher than in 2011, which required about 137,000 records, but less than 2010, which
required about 167,000 records. The introduction of a mail survey—with an administration cycle time
nearly twice that of CATI—limited the number of sample waves in 2012 to just two. This outcome
meant that sample draw estimates were less precise than in previous years when sample was released
over several waves. In 2011, the sample yielded an average of one completed interview per 3.2 sampled
records. In 2012, the yield for the telephone frame was one per 3.8 records.
However, when the mail survey is factored in, the yield for 2012 was 49,115 interviews from 159,577
records, or one in every 3.2 records. If the mail survey were to be used for all non-responders and all
nonworking numbers, the yield could be as high as one in every 2.7 records.

Sampling Design and Interview Outcomes
The final sample for the Survey of Enrollees must pass through many stages—the sampling stages
described above, as well as the survey process. The record had to lead to the correct enrollee, the
enrollee had to be contacted, and the enrollee had to elect to respond to the survey.
The only stage that is a controlled random process—and, therefore, not subject to potential bias—is the
random sample selection. All other stages have the potential to introduce non-random, systematic bias
into enrollee estimates.

Frame Eligibility
Referring to Table 6, on page 20, 7,451,077 enrollees (92.9%) were eligible for the telephone sampling
frame, leaving 562,231 enrollees (7.1%) ineligible for the telephone sampling frame due to incomplete
telephone information.
At that time, according to administrative records, about 63 percent of enrollees received services (longterm, inpatient, or outpatient care) during the previous 12 months. Telephone frame eligibility was
15

higher for those who had received services compared to those who had not (97 versus 85 percent).
Similarly, telephone frame eligibility was higher for the 55 percent of enrollees receiving the prescription
drug service compared to the 45 percent who did not (97 versus 87 percent frame eligibility).
These percentages are similar to 2011; but in 2012, enrollees who were ineligible for the telephone
frame were administered a mail survey. This reduced the risk of coverage bias by including responses
from normally non-covered enrollees.

Valid Contact
All of the sampled enrollees were called at least once to initiate an interview, except for cases where no
telephone number was available. During data collection, many telephone numbers were classified as
invalid, including non-working numbers, wrong numbers at which the selected enrollee was not known,
out-of-service numbers, fax or modem telephone numbers, and business numbers at which the enrollee
was not known.
Enrollees with these invalid numbers were unable to complete the survey since the telephone number
did not lead to the selected enrollee. This loss of the sample population in prior years may have
introduced non-response bias in the survey estimates since these enrollees were part of the total
population, yet could not be reached for interview.
The only way to obtain an alternate telephone number for use in the CATI study was to collect one if it
was provided by an individual at the incorrect telephone number, or if the sampled enrollee called ICF in
response to the pre-survey notification letter and provided a number.
In 2012, alternative response channels included a mail survey or a Web survey if the enrollee accessed
the Web survey in response to pre-survey notification materials.
Enrollees with invalid contact information numbered 27 percent (33,639), with 20 percent (22,085)
classified as nonworking numbers. Of the 22,085 nonworking numbers, 15,761 were sent a mail survey,
which yielded 3,388 interviews, a 21.5 percent overall response rate.

Non-Response
After determining that the telephone contact information was accurate, the final stage of the process
was either a complete interview (response) or unsuccessful attempts. ICF classifies non-response into
two forms, enrollee refusal and enrollee non-contact.
Enrollee refusals result when an interviewer contacts an enrollee (or an enrollee agent), and
communicates the sponsor (VHA) and purpose of the survey, but the enrollee elects not to participate
by verbal refusal, hang-up, or another form of termination.
A non-contact means that an interviewer never reaches the enrollee (or an enrollee agent) directly; this
includes answering machines and other technological barriers, language barriers, and busy numbers, as
well as hang-ups and refusals before or during the survey introduction (where an enrollee’s presence is
not yet confirmed).

16

We sent a non-response mail survey to 15,078 out of the 69,173 non-responding enrollees in the CATI
study. The follow-up was effective in obtaining responses from 2,705 (17.9 percent) of these telephone
non-respondents.
In general, non-response is evaluated by examining a survey’s response rate (i.e., the proportion of
completed interviews relative to the selected sample, minus the identified ineligible sample elements).
For the 2012 Survey of Enrollees, the final response rate using American Association of Public Opinion
Research response rate calculations (AAPOR RR12) was 39 percent for both the sample selected from the
telephone frame as well as the sample selected from the mail-only frame. The overall response rate was
39 percent.
This rate is higher than the 35 percent obtained in 2010, but lower than the 42 percent in 2011.

2

AAPOR response rates are more complex than a simple ratio, in order to account for records with unknown eligibility status.
The RR1 is defined as
RR1 = I/(I + P) + (R + NC + O) + (UH + UO), where
I = Complete interview
P = Partial interview
R = Refusal and break-off
NC = Non-contact
O = Other
UH = Unknown if household/occupied HU
UO = Unknown, other
e = Estimated proportion of cases of unknown eligibility that are eligible

17

BIAS ANALYSIS
To facilitate bias analysis, the sample was divided into two groups:




No telephone records: those records with a missing or incomplete telephone number (less than
10 digits) or lacking a valid area code and exchange (prefix) combination.3
o

For telephone numbers with seven digits or an invalid area code, we attempted to look
up the correct number based on city and state. This resulted in 622 additional enrollees
with telephone numbers that were included with the telephone survey.

o

The remaining enrollees with no telephone records were administered a mail survey.

Telephone records: those records where the telephone number appeared valid—having 10 digits
and a valid area code and exchange (prefix) combination were administered a telephone survey.
o

A sample of enrollees with nonworking numbers was sent a mail survey.

o

A sample of non-responding enrollees (with valid telephone numbers) was sent a mail
survey.

o

A sample of enrollees (with valid telephone numbers) who requested a mail survey was
sent one.

These groups were further subdivided into the following sub-populations, tallied in Table 5, below. The
bias analysis then computes measures of enrollee characteristics, described below, for these
populations, using comparisons of estimates across groups to assess bias through various stages of the
survey process. Note that these population sub-groups may or may not correspond to treatment groups
described in the preceding section.
Table 5 presents a cross-walk between these definitions. Some of the counts in the tables below and
Figure 1 above may not match exactly due to definitional differences noted in the crosswalk.
Subpopulation 1:
The enrollee population (excluding those with non-valid addresses, living
outside the U.S. or Puerto Rico, or missing one of the stratification variables).
Subpopulation 2:

Frame: telephone and mail only (considered ineligible in past years)

Subpopulation 3:

The telephone sample

Subpopulation 4:

Compares invalid versus valid telephone contact information

Subpopulation 5:

Compares telephone respondents versus telephone non-respondents

Subpopulation 6:

Compares all respondents (telephone and mail) versus all non-respondents

Subpopulations 1–5 represent the bias analyses as presented since 2007. Subpopulation 6 is the change
in bias that resulted from including the mail survey. Table 6 presents the counts for each stage.

3

85 percent of invalid phone numbers were missing completely. Another 5.5 percent of the invalid phone numbers contained
only seven digits. The remaining 10 percent of invalid phone numbers was due to invalid area code and exchange (prefix)
combinations.

18

Table 5. Sample Crosswalk
Subpopulation #/descriptor

Table 6
Column/Label
1-Enrollee
Population
2-Frame

Figure 1 Label

Bias Figure Bar Label

In frame

Population

-na-

Comparison of subgroups, below, estimates bias
due to exclusion from telephone study due to
missing phone data
Frame - Phone
In Frame Telephone
Frame - Mail
In Frame - Mail

CATI survey. Does not
match exactly, bias
analysis does not
include additional
records resulting from
phone lookup.
Eligible contact. Does
not match exactly, bias
analysis includes
records that were
initially flagged as
ineligible and
subsequently returned a
survey.

In Sample - Yes

1.

Enrollee population

2.

Frame: telephone and mail only
(considered ineligible in past years)

3.

Telephone Sample

3-Enrollees
Selected

4.

Valid Telephone Contacts

4-Correct
Contact

Eligible - No
Telephone Respondents

5-Survey
Responses

Response. Does not
match exactly,
differences in how
records moved from
one group to another
were handled

Sampled - Yes

Comparison of subgroups, below, estimates bias
due to ability to contact enrollee

Eligible - Yes

5.

Bias Table
Comparison Group
Population

Valid Telephone Yes
Valid Telephone - No

Comparison of subgroups, below, assesses bias
due to non-response in the CATI study

Phone Resp – Yes

6.

All responses

6-All Survey
Responses

Response

Telephone Survey
Response - Yes
Phone Resp – No
Telephone Survey
Response - No
Comparison of subgroups, below, assesses bias
due to non-response overall
All Resp - Yes
Telephone and Mail
Survey Responses yes
All Resp - No
Telephone and Mail
Survey Responses No

19

Table 6. Sample Stages and Enrollee Totals for the 2012 Survey of Enrollees
Enrollees with a Telephone Number

Total

Enrollee
Population
8,013,308

Frame
7,451,077

Enrollees
Selected
146,022

Correct
Contact
112,426

Survey
Responses
38,427

All Survey
Responses
49,115

OEF/OIF/OND

N

7,194,207

6,673,799

107,070

83,095

31,053

40,288

OEF/OIF/OND

Y

819,101

777,278

38,952

29,331

7,374

8,827

VISN

1

339,473

317,182

6,906

5,533

1,835

2,294

2

196,387

178,961

7,331

5,839

1,881

2,373

3

298,276

267,786

9,207

7,124

1,980

2,609

4

439,281

413,518

6,855

5,414

1,843

2,357

5

206,534

188,133

8,242

6,140

1,822

2,409

6

465,342

429,940

7,175

5,420

1,854

2,461

7

526,440

496,906

6,961

5,197

1,781

2,314

8

613,824

581,393

7,314

5,603

1,940

2,439

9

377,434

354,916

6,186

4,721

1,832

2,265

10

298,493

280,134

6,827

5,305

1,801

2,237

11

370,877

342994

6,657

5,116

1,815

2,345

12

333,194

311,806

6,734

5,304

1,817

2,284

15

320,090

299,831

6,341

4,878

1,772

2,281

16

658,673

615,087

7,247

5,454

1,892

2,410

17

396,760

370,303

7,124

5,324

1,795

2,297

18

342,704

305,050

6,404

4,873

1,753

2,281

19

255,645

238,436

5,943

4,645

1,786

2,256

20

374,293

344,598

5,991

4,573

1,789

2,266

21

346,471

323,663

7,093

5,465

1,827

2,318

22

444,999

404,794

7,660

5,849

1,823

2,353

23

408,118

385,646

5,824

4,649

1,789

2,266

1

1,300,071

1,252,275

19,481

16,392

6,135

7,209

2

628,637

586,300

20,643

16,578

5,725

7,218

3

1,103,284

1,003,474

21,656

16,653

5,647

7,442

4

199,831

188,482

19,193

13,738

4,856

5,851

5

2,060,842

1,891,161

23,382

17,194

5,553

7,136

6

553,138

517,792

10,051

7,308

1,993

2,575

7

179,525

174,200

2,234

1,915

764

914

8

1,987,980

1,837,393

29,382

22,648

7,754

10,770

Enrollee Type

POST

6,134,364

5,804,414

86,822

66,854

21,977

26,857

Enrollee Type

PRE

1,878,944

1,646,663

59,200

45,572

16,450

22,258

Priority

20

With the exception of the controlled random sampling process, all sample stages described in the
previous section have the potential to introduce bias into the survey estimates. The impact of coverage
(or frame) bias and non-response bias are difficult to assess since data are not available for those who
do not participate in the survey. Therefore, there is no way to compare the groups (those who respond
and those who do not) and draw inferences about the survey data. In lieu of responses from individuals
who do not participate in the survey, we rely on secondary information available for both survey
respondents and non-respondents. This information generally comes from the sampling frame and/or
the population. In most cases, this information is limited; but in the case of VHA, there is considerable
administrative data available about the enrollee population. This information allows review of frame
coverage and non-response biases with respect to enrollees’ use of various VHA services.
To allow ICF to conduct this bias analysis, VHA provided a file based on administrative records that
indicated whether an enrollee had utilized any of the following services in the previous year. The file did
not indicate the frequency of use or amount paid for any of these benefits:
1. Received long-term care services4
a. Institutional
b. Non-institutional
2. Received Inpatient treatment
a. Mental Health or Substance Abuse (MHSA)
b. Non-MHSA
3. Received Outpatient treatment
a. MHSA
b. Non-MHSA
4. Received VHA pharmacy services
Since 2007, VHA has evaluated the impact of non-response bias on the utilization indicators above. For
the 2010 study, VHA evaluated differences between two administrative sources of service utilization:
 “Original” indicators: Service utilization sourced from VHA workload files based on bed section
and clinic stop. This categorization indicates where a Veteran received care.
 “HSC” indicators: Service utilization based on Health Service Categories (HSCs). This
categorization indicates what type of care a Veteran received.
While the indicators were generally consistent, several comparisons stood out:
 For pharmacy services (RX), the two indicators are identical.
 For outpatient MHSA services, the HSC indicator identifies an additional nine percent of
enrollees.
 For outpatient non-MHSA services, the HSC indicator identifies an additional five percent of
enrollees.
 For home health care (now, non-institutional long-term care), the HSC measures usage at 1.44
percent. Although this is a very small percentage, it represents a 650 percent increase relative to
the original indicator of 0.19 percent.

4

In previous years, we evaluated the percentage of enrollees receiving home health service. We expanded this utilization
statistic to include long-term care services in both institutional and non-institutional settings.

21

Considering that the HSC indicators are potentially more reflective of actual utilization, VHA determined
to use these for the 2011 bias analysis and beyond. Thus, the results of the non-response analysis below
are based on the HSC indicators. Differences in magnitude for estimates in reports prior to 2011 reflect
the change in the administrative source and not necessarily a change in utilization.

1. Long-Term Care Benefits
A small proportion of enrollees receive long-term care, 0.54 and 2.73 percent for institutional and noninstitutional care, respectively. This percentage is slightly higher for enrollees eligible for the telephone
frame for both institutional and non-institutional care.
For institutional care, those with valid contact information are significantly less likely to have received
institutionalized long-term care than those with invalid contact information (p<0.0001). This difference
is most evident for pre-enrollee utilization rates comparing those with and without valid contact
information (1.29 vs. 2.37 percent, respectively; p<0.0001) and Priority Group 1 (1.40 vs. 3.34 percent,
respectively; p<0.0001).
Overall, the utilization rate is lower among enrollees that did respond to the survey than those that did
not. This difference in utilization rates based on response status is highest for the Pre-enrollees and
Priority Group 1 enrollees. Pre-enrollees and Priority Group 1 enrollees also were significantly different
in terms of response rates in 2011.
Overall, enrollees with valid phone information have a higher percentage of non-institutionalized longterm care than those with invalid phone information (3.00 percent vs. 2.31 percent, p<0.0001).
Compounding this bias, the percentage is higher for respondents versus non-respondents (3.42 percent
vs. 2.74 percent, p<0.0001). When the responses gained from the mail survey are added (3.27 percent),
the bias is lower, but the final estimate still overestimates the population value (2.73 percent).

22

Figure 2. Percentage of Enrollees Receiving Long-Term Care (LTC)
(a) Institutional
I nst i t ut i onal LTC
0. 54%

Popul at i on
Fr ame

0. 55%

Phone
0. 35%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

0. 60% [ 0. 54%, 0. 65%]
0. 53% [ 0. 48%, 0. 59%]
0. 82% [ 0. 70%, 0. 94%]

No
Phone Resp

0. 42% [ 0. 34%, 0. 51%]

Yes

0. 60% [ 0. 53%, 0. 68%]

No
Al l Resp

0. 44% [ 0. 36%, 0. 52%]

Yes

0. 68% [ 0. 61%, 0. 75%]

No
0. 0

0. 4

0. 8

1. 2

1. 6

2. 0

Per cent
Sc al e: 2. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

Figure 2. Percentage of Enrollees Receiving Long-Term Care (LTC)
(b) Non-Institutional
Non- i nst i t ut i onal LTC
2. 73%

Popul at i on
Fr ame

2. 88%

Phone
0. 77%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

2. 85% [ 2. 74%, 2. 97%]
3. 00% [ 2. 87%, 3. 13%]
2. 31% [ 2. 10%, 2. 52%]

No
Phone Resp

3. 42% [ 3. 18%, 3. 66%]

Yes

2. 74% [ 2. 59%, 2. 90%]

No
Al l Resp

3. 27% [ 3. 06%, 3. 49%]

Yes

2. 62% [ 2. 49%, 2. 75%]

No
0

2

4

6

8

10

Per cent
Sc al e: 10. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

23

Table 7. Percentage of Enrollees Receiving Long-Term Care
(a) Institutional
In Frame
Sampled
Valid Telephone
TelePopulation phone
Mail
Yes
Yes
No
P-value
Total
0.54
0.55
0.35
0.60 0.53
0.82
0.0000
OEF/OIF/OND N
0.60
0.61
0.37
0.66 0.59
0.93
0.0000
OEF/OIF/OND Y
0.04
0.04
0.03
0.03 0.04
0.01
0.0758
VISN
1
0.57
0.58
0.43
0.57 0.53
0.71
0.5937
2
0.54
0.57
0.30
0.46 0.32
1.03
0.0148
3
0.50
0.53
0.30
0.53 0.41
0.97
0.0079
4
0.64
0.66
0.35
0.80 0.71
1.17
0.1755
5
0.73
0.76
0.43
0.80 0.67
1.22
0.0664
6
0.37
0.38
0.23
0.31 0.26
0.49
0.3117
7
0.33
0.33
0.26
0.26 0.27
0.22
0.6716
8
0.43
0.44
0.29
0.72 0.69
0.83
0.5998
9
0.45
0.46
0.27
0.41 0.34
0.64
0.3911
10
0.60
0.60
0.69
0.61 0.59
0.70
0.7059
11
0.63
0.65
0.35
0.80 0.63
1.44
0.0487
12
0.76
0.78
0.46
0.72 0.62
1.07
0.1299
15
0.52
0.54
0.27
0.56 0.37
1.23
0.0480
16
0.40
0.41
0.27
0.48 0.48
0.50
0.9397
17
0.45
0.46
0.24
0.36 0.35
0.39
0.7974
18
0.65
0.70
0.28
0.81 0.81
0.82
0.9715
19
0.56
0.56
0.45
0.62 0.59
0.75
0.6013
20
0.52
0.54
0.28
0.64 0.66
0.59
0.8195
21
0.72
0.74
0.56
0.82 0.67
1.38
0.0928
22
0.56
0.58
0.36
0.47 0.46
0.51
0.7419
23
0.79
0.81
0.53
1.01 0.83
1.78
0.0555
Priority
1
1.59
1.58
2.00
1.70 1.40
3.34
0.0000
2
0.27
0.28
0.14
0.32 0.30
0.40
0.4254
3
0.24
0.26
0.10
0.24 0.28
0.11
0.0266
4
3.13
3.16
2.62
3.33 3.05
4.06
0.0015
5
0.44
0.46
0.26
0.49 0.42
0.68
0.0703
6
0.05
0.05
0.02
0.08 0.10
0.01
0.1576
7
0.38
0.38
0.41
0.32 0.28
0.56
0.5335
8
0.09
0.10
0.03
0.13 0.14
0.10
0.5079
Enrollee Type POST
0.31
0.32
0.22
0.33 0.32
0.38
0.3486
PRE
1.28
1.38
0.52
1.52 1.29
2.37
0.0000
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone Survey
Response
Yes
0.42
0.45
0.03
0.15
0.31
0.23
0.58
0.51
0.29
0.05
0.63
0.21
0.62
0.51
0.47
0.18
0.49
0.34
0.77
0.71
0.51
0.55
0.21
0.54
0.90
0.36
0.27
2.66
0.40
0.13
0.18
0.07
0.28
0.93

No P-value
0.60 0.0019
0.68 0.0003
0.04 0.8026
0.75 0.0029
0.33 0.9373
0.49 0.1355
0.78 0.5090
0.74 0.3719
0.24 0.8180
0.41 0.0064
0.72 0.7653
0.44 0.1754
0.57 0.8112
0.70 0.4949
0.72 0.2317
0.50 0.0466
0.47 0.9393
0.35 0.9378
0.83 0.8571
0.50 0.4240
0.77 0.3653
0.73 0.5040
0.58 0.0364
1.06 0.1288
1.74 0.0001
0.26 0.3777
0.29 0.9102
3.27 0.0603
0.43 0.8339
0.09 0.7906
0.36 0.5207
0.18 0.1270
0.34 0.3176
1.51 0.0001

Telephone and Mail
Survey Response
PYes
No
value
0.44
0.68 0.0000
0.47
0.77 0.0000
0.03
0.03 0.8997
0.28
0.71 0.0299
0.39
0.49 0.5734
0.19
0.66 0.0016
0.54
0.94 0.1422
0.57
0.90 0.1660
0.27
0.34 0.6775
0.04
0.38 0.0011
0.74
0.70 0.9000
0.18
0.55 0.0350
0.55
0.64 0.6695
0.57
0.94 0.1892
0.45
0.86 0.0348
0.17
0.78 0.0012
0.58
0.43 0.5873
0.32
0.38 0.7077
0.75
0.85 0.7636
0.73
0.55 0.4584
0.44
0.77 0.1605
0.57
0.95 0.1664
0.25
0.56 0.0431
0.53
1.35 0.0093
1.01
2.14 0.0000
0.31
0.33 0.8477
0.25
0.24 0.9519
2.72
3.60 0.0023
0.45
0.51 0.6606
0.11
0.07 0.7151
0.16
0.44 0.2940
0.06
0.17 0.0593
0.30
0.35 0.4348
0.92
1.84 0.0000

24

Table 7. Percentage of Enrollees Receiving Long-Term Care
(b) Non-Institutional
In Frame
Sampled
Valid Telephone
Populatio Telen
phone
Mail
Yes
Yes
No
P-value
Total
2.73
2.88
0.77
2.85 3.00
2.31
0.0000
OEF/OIF/OND N
2.98
3.15
0.82
3.12 3.26
2.58
0.0000
OEF/OIF/OND Y
0.52
0.54
0.21
0.55 0.59
0.44
0.0845
VISN
1
2.93
3.08
0.65
3.07 3.32
2.00
0.0050
2
3.47
3.74
0.75
4.06 3.96
4.49
0.5008
3
2.87
3.13
0.59
2.90 3.16
1.99
0.0040
4
2.97
3.11
0.73
3.09 3.27
2.34
0.0974
5
2.44
2.64
0.47
2.90 3.32
1.61
0.0001
6
2.55
2.69
0.84
2.90 2.91
2.87
0.9495
7
2.22
2.31
0.69
2.21 2.37
1.72
0.1554
8
2.65
2.76
0.76
2.50 2.77
1.51
0.0039
9
2.74
2.87
0.67
2.70 2.96
1.75
0.0191
10
3.96
4.13
1.35
4.29 4.39
3.93
0.5208
11
2.64
2.78
0.93
2.85 2.91
2.61
0.6442
12
3.00
3.14
0.91
2.94 3.05
2.49
0.2427
15
2.91
3.05
0.90
2.62 2.50
3.09
0.3642
16
2.62
2.75
0.82
2.86 2.96
2.53
0.4607
17
2.60
2.74
0.67
2.64 2.96
1.62
0.0048
18
2.67
2.92
0.62
3.14 3.12
3.23
0.8729
19
3.09
3.24
0.93
3.71 3.89
3.00
0.2512
20
2.42
2.57
0.71
2.51 2.71
1.81
0.0835
21
2.48
2.61
0.62
2.39 2.65
1.43
0.0029
22
2.02
2.15
0.72
1.97 2.27
0.93
0.0001
23
3.15
3.27
1.11
3.35 3.14
4.28
0.1976
Priority
1
5.57
5.68
2.64
6.07 6.10
5.87
0.6819
2
2.04
2.15
0.56
2.04 2.07
1.92
0.6176
3
1.74
1.87
0.39
2.00 2.11
1.66
0.1187
4
14.52
15.05
5.79
14.68 15.56 12.41
0.0000
5
2.79
2.97
0.80
2.59 2.84
1.87
0.0001
6
0.44
0.47
0.13
0.49 0.56
0.24
0.1433
7
2.66
2.70
1.52
2.18 2.11
2.63
0.6069
8
1.03
1.10
0.20
1.17 1.26
0.82
0.0312
Enrollee Type POST
1.98
2.05
0.67
2.04 2.17
1.59
0.0000
PRE
5.17
5.77
0.91
5.67 5.89
4.84
0.0003
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone Survey
Response
Yes
No P-value
3.42 2.74 0.0000
3.60 3.05 0.0005
0.84 0.51 0.0091
3.20 3.39 0.7299
4.23 3.82 0.5493
3.84 2.89 0.1100
4.59 2.51 0.0025
3.22 3.37 0.8096
2.97 2.87 0.8762
2.26 2.44 0.7613
3.20 2.51 0.2522
3.28 2.73 0.3967
5.18 3.92 0.0911
2.95 2.88 0.9153
3.48 2.80 0.2880
2.82 2.29 0.3434
2.99 2.94 0.9389
4.19 2.28 0.0057
4.56 2.19 0.0015
3.39 4.25 0.2388
3.45 2.12 0.0498
3.12 2.38 0.2067
2.76 2.03 0.1899
3.74 2.69 0.1179
6.29 5.98 0.5015
2.20 1.99 0.4254
2.54 1.85 0.0279
15.58 15.55 0.9563
3.49 2.47 0.0029
1.07 0.30 0.0666
1.81 2.35 0.4999
1.65 1.02 0.0089
2.53 1.94 0.0003
6.49 5.53 0.0022

Telephone and Mail
Survey Response
Yes
3.27
3.44
0.84
3.23
3.99
3.43
4.31
3.08
2.72
2.23
3.00
2.95
4.81
3.57
3.14
2.68
3.05
3.85
3.89
3.91
3.26
2.73
2.70
3.62
6.20
2.15
2.45
15.48
3.28
0.94
1.79
1.55
2.43
6.22

No
P-value
2.62 0.0000
2.93 0.0003
0.48 0.0015
2.98 0.6075
4.10 0.8566
2.71 0.1480
2.43 0.0021
2.83 0.6430
2.99 0.6350
2.21 0.9587
2.22 0.1248
2.54 0.4626
4.02 0.2334
2.42 0.0586
2.83 0.5683
2.59 0.8556
2.77 0.6197
2.06 0.0027
2.70 0.0668
3.59 0.6343
2.02 0.0348
2.22 0.3014
1.65 0.0375
3.17 0.4760
5.98 0.6136
1.98 0.4804
1.77 0.0159
14.32 0.0614
2.27 0.0006
0.30 0.0809
2.49 0.3417
0.95 0.0039
1.83 0.0000
5.37 0.0024

25

2. Inpatient Treatment
Reasons Related to Mental Health or Substance Abuse (MHSA)
Overall, 1.18 percent of enrollees have been admitted to a hospital or medical facility for MHSA.
Similar to previous years, there is a considerable difference in inpatient MHSA utilization between
enrollees with valid contact information and enrollees without valid contact information, with rates of
1.07 to 1.70 percent (p<0.0001) respectively. This underestimation—the difference between these two
rates—is further compounded by the fact that non-respondents were more likely to have received
inpatient treatment (1.21 percent vs. 0.84 percent, p<0.0001).
Both of these effects combine to result in a survey-based estimate that underestimates the true
utilization based on the population rate (1.18) by 33 percentage points, or 38 percent of the population
value. This is similar to 2011 and earlierunderestimation of utilization has been a consistent pattern
for enrollees receiving inpatient care for substance abuse or mental health. The addition of mail
respondents does not have a measurable impact on the final rate estimate (0.85 percent); there is still
severe underestimation of the population. Moving to a full administration of the mail protocol could
address some of this bias.

Reasons Unrelated to MHSA
For enrollees admitted to a hospital or medical facility for reasons unrelated to MHSA, the telephone
sample-based estimate of the percentage of enrollees receiving inpatient treatment unrelated to MHSA
issues is 5.14 percent, higher than the actual percentage of 4.25 percent. This is consistent with data
from 2011 and 2010. The percentage of inpatient utilization unrelated to MHSA for enrollees eligible for
the frame (4.43 percent) is considerably higher than utilization for enrollees not eligible for the frame
(1.84 percent). This is compounded by both a higher rate of utilization among enrollees with eligible
contact information and by significantly higher utilization among respondents (5.14 percent) relative to
non-respondents (4.47 percent) (p=0.0004). The addition of the mail survey reduces this bias, but the
estimated utilization rate of 5.04 percent still overestimates the true population rate of 4.25 percent.

26

Figure 3. Percentage of Enrollees Receiving Inpatient Treatment
(a) For Mental Health or Substance Abuse (MHSA)
I npat i ent Car e: Ment al Heal t h Rel at ed
1. 18%

Popul at i on
Fr ame

1. 19%

Phone

1. 09%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

1. 20% [ 1. 13%, 1. 27%]
1. 07% [ 0. 99%, 1. 14%]
1. 70% [ 1. 52%, 1. 88%]

No
Phone Resp

0. 84% [ 0. 72%, 0. 96%]

Yes

1. 21% [ 1. 11%, 1. 31%]

No
Al l Resp

0. 85% [ 0. 74%, 0. 96%]

Yes

1. 39% [ 1. 30%, 1. 49%]

No
0

2

4

6

8

10

Per cent
Sc al e: 10. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

Figure 3. Percentage of Enrollees Receiving Inpatient Treatment
(b) For Neither Mental Health nor Substance Abuse
I npat i ent Car e: Not Ment al Heal t h Rel at ed
4. 25%

Popul at i on
Fr ame

4. 43%

Phone
1. 84%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

4. 55% [ 4. 39%, 4. 70%]
4. 72% [ 4. 54%, 4. 90%]
3. 90% [ 3. 60%, 4. 20%]

No
Phone Resp

5. 14% [ 4. 83%, 5. 44%]

Yes

4. 47% [ 4. 25%, 4. 68%]

No
Al l Resp

5. 04% [ 4. 76%, 5. 32%]

Yes

4. 28% [ 4. 10%, 4. 46%]

No
0

2

4

6

8

10

Per cent
Sc al e: 10. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

27

Table 8. Percentage of Enrollees Receiving Inpatient Treatment
(a) For Mental Health or Substance Abuse (MHSA)
Samp
In Frame
led
Valid Telephone
Population Telephone Mail
Yes
Yes
No
P-value
Total
1.18
1.19
1.09
1.20
1.07 1.70
0.0000
OEF/OIF/OND N
1.15
1.16
1.07
1.17
1.04 1.68
0.0000
OEF/OIF/OND Y
1.46
1.47
1.39
1.48
1.37 1.83
0.0033
VISN
1
1.44
1.45
1.23
1.38
1.38 1.41
0.9230
2
1.20
1.25
0.72
1.25
1.16 1.66
0.2477
3
1.15
1.18
0.89
1.24
1.05 1.89
0.0333
4
1.19
1.20
1.13
1.18
1.00 1.95
0.0406
5
1.31
1.28
1.54
1.52
1.55 1.43
0.7083
6
1.20
1.21
1.13
1.08
0.94 1.57
0.0835
7
1.16
1.17
1.10
1.46
1.49 1.38
0.8033
8
1.05
1.05
1.07
0.86
0.71 1.40
0.0291
9
1.32
1.34
1.02
1.65
1.47 2.28
0.1828
10
1.39
1.39
1.51
1.19
1.01 1.89
0.0611
11
1.09
1.10
1.04
1.09
0.98 1.52
0.2202
12
1.32
1.29
1.69
1.25
1.15 1.63
0.2246
15
1.39
1.38
1.49
1.74
1.40 2.99
0.0142
16
1.22
1.24
0.92
1.57
1.37 2.26
0.0708
17
1.25
1.26
1.13
1.24
1.26 1.20
0.8720
18
1.05
1.09
0.76
1.45
1.18 2.42
0.0455
19
1.15
1.15
1.09
1.09
0.95 1.61
0.1686
20
1.16
1.18
0.83
0.81
0.57 1.65
0.0060
21
0.95
0.94
1.02
0.91
0.80 1.32
0.2540
22
1.02
1.00
1.13
0.66
0.41 1.51
0.0024
23
1.04
1.03
1.17
0.86
0.88 0.78
0.7626
Priority
1
2.26
2.24
2.80
2.15
1.85 3.78
0.0000
2
0.98
0.99
0.80
1.06
0.97 1.43
0.0601
3
0.80
0.82
0.54
0.85
0.79 1.06
0.1619
4
6.90
6.68 10.61
7.01
6.24 8.99
0.0000
5
1.50
1.50
1.41
1.56
1.40 2.01
0.0147
6
0.30
0.30
0.29
0.35
0.38 0.25
0.2773
7
0.56
0.54
1.22
0.45
0.44 0.58
0.6798
8
0.15
0.16
0.10
0.14
0.13 0.21
0.2659
Enrollee Type POST
0.92
0.91
1.05
0.90
0.80 1.24
0.0001
PRE
2.03
2.15
1.16
2.27
1.98 3.31
0.0000
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone Survey
Response
Yes
No
P-value
0.84 1.21
0.0000
0.81 1.18
0.0000
1.27 1.40
0.4522
0.97 1.61
0.0560
0.90 1.29
0.1981
1.38 0.92
0.2379
0.67 1.19
0.0516
1.09 1.77
0.0740
1.01 0.90
0.7593
1.42 1.53
0.8262
0.35 0.93
0.0067
0.65 2.09
0.0006
0.63 1.24
0.0244
0.95 1.00
0.9128
1.05 1.22
0.6106
0.70 1.85
0.0020
1.38 1.37
0.9939
1.29 1.24
0.9033
0.75 1.46
0.0695
0.84 1.04
0.5289
0.67 0.49
0.4705
0.45 1.00
0.0423
0.29 0.47
0.1631
0.40 1.23
0.0039
1.65 1.99
0.1659
0.83 1.05
0.2065
0.40 1.03
0.0000
4.44 7.29
0.0000
1.14 1.55
0.0829
0.28 0.44
0.3789
0.27 0.57
0.3888
0.05 0.17
0.0088
0.66 0.89
0.0083
1.47 2.30
0.0000

Telephone and Mail
Survey Response
Yes
No
P-value
0.85 1.39
0.0000
0.82 1.37
0.0000
1.24 1.55
0.0401
1.00 1.58
0.0536
0.89 1.43
0.0544
1.28 1.22
0.8746
0.71 1.44
0.0052
0.96 1.74
0.0125
0.96 1.14
0.5779
1.26 1.58
0.4487
0.38 1.12
0.0002
0.87 2.15
0.0011
0.64 1.48
0.0013
0.99 1.15
0.6711
1.01 1.39
0.1896
0.76 2.30
0.0000
1.61 1.55
0.8925
1.13 1.29
0.6764
0.67 1.91
0.0007
0.78 1.28
0.0949
0.63 0.93
0.2247
0.48 1.13
0.0114
0.48 0.74
0.2298
0.36 1.22
0.0004
1.57 2.53
0.0000
0.85 1.18
0.0472
0.51 1.03
0.0002
4.50 8.12
0.0000
1.17 1.74
0.0073
0.26 0.39
0.3902
0.26 0.61
0.2551
0.10 0.17
0.1835
0.65 1.03
0.0000
1.54 2.66
0.0000

28

Table 8. Percentage of Enrollees Receiving Inpatient Treatment
(b) For Neither Mental Health nor Substance Abuse
SamIn Frame
pled
Valid Telephone
Popula
Tele-tion
phone
Mail
Yes
Yes
No
P-value
Total
4.25
4.43
1.84
4.55
4.72
3.90
0.0000
OEF/OIF/OND N
4.60
4.81
1.93
4.93
5.09
4.33
0.0001
OEF/OIF/OND Y
1.17
1.20
0.76
1.22
1.32
0.91
0.0014
VISN
1
3.56
3.73
1.18
3.49
3.61
2.95
0.2909
2
3.64
3.87
1.30
4.07
3.87
4.91
0.1954
3
3.18
3.41
1.18
3.35
3.37
3.28
0.8625
4
3.42
3.53
1.71
3.94
3.90
4.14
0.7562
5
3.97
4.16
2.02
4.38
4.65
3.58
0.0914
6
4.16
4.34
1.91
3.86
3.80
4.05
0.7256
7
3.90
4.02
1.86
4.64
5.16
2.96
0.0044
8
5.04
5.19
2.20
5.06
5.47
3.57
0.0080
9
5.02
5.21
1.99
5.51
5.91
4.05
0.0324
10
4.35
4.50
2.11
4.43
4.46
4.30
0.8504
11
3.84
3.99
2.01
3.78
4.06
2.73
0.0363
12
4.50
4.68
1.83
4.30
4.45
3.74
0.3241
15
5.09
5.26
2.52
5.81
6.16
4.52
0.0509
16
4.72
4.91
2.02
5.38
5.36
5.45
0.9256
17
4.29
4.48
1.58
4.51
4.85
3.42
0.0336
18
4.74
5.14
1.53
4.99
4.97
5.06
0.9201
19
4.30
4.46
1.98
4.24
4.43
3.51
0.2206
20
4.19
4.41
1.61
4.76
4.98
3.99
0.2675
21
4.34
4.51
1.98
4.66
4.60
4.90
0.7316
22
4.17
4.38
2.13
4.33
4.76
2.89
0.0046
23
3.84
3.95
2.04
4.62
4.77
3.95
0.3225
Priority
1
7.75
7.85
5.17
7.78
7.66
8.49
0.2168
2
3.30
3.43
1.45
3.31
3.57
2.23
0.0000
3
2.87
3.06
0.94
3.23
3.53
2.21
0.0000
4
14.71
14.96 10.47
15.33 15.74 14.27
0.0181
5
5.70
5.99
2.46
6.17
6.56
5.07
0.0009
6
1.09
1.13
0.47
1.13
1.15
1.05
0.7780
7
4.06
4.09
2.89
5.33
5.22
6.14
0.6553
8
1.36
1.44
0.43
1.56
1.72
0.95
0.0001
Enrollee Type POST
3.22
3.30
1.75
3.43
3.62
2.71
0.0000
PRE
7.60
8.40
1.97
8.43
8.53
8.07
0.2195
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone Survey
Response
Yes
5.14
5.38
1.66
3.57
4.68
3.50
4.89
5.50
4.20
5.73
5.40
5.86
5.05
3.77
4.92
6.47
5.96
5.39
5.86
4.78
5.32
3.74
6.42
5.10
7.50
4.32
3.94
15.72
7.39
1.32
5.79
2.05
4.16
8.56

No
4.47
4.91
1.20
3.64
3.44
3.31
3.32
4.25
3.56
4.80
5.51
5.95
4.11
4.26
4.16
5.95
5.01
4.56
4.40
4.18
4.71
5.08
3.92
4.52
7.76
3.14
3.29
15.75
6.09
1.06
4.76
1.51
3.30
8.51

P-value
0.0004
0.0226
0.0111
0.9168
0.0916
0.7126
0.0341
0.1090
0.3497
0.3375
0.8978
0.9237
0.2421
0.5199
0.3173
0.5890
0.2888
0.3120
0.1032
0.4467
0.4946
0.0997
0.0051
0.4998
0.6187
0.0017
0.1077
0.9647
0.0168
0.5519
0.4661
0.0379
0.0001
0.9014

Telephone and Mail
Survey Response
Yes
5.04
5.27
1.61
3.83
4.40
3.51
5.07
5.13
4.08
5.49
5.33
5.64
5.01
3.68
4.55
6.25
6.44
5.06
5.56
4.74
5.09
3.85
6.20
4.73
7.55
4.03
3.76
15.35
7.41
1.19
5.86
1.94
4.06
8.48

No
4.28
4.73
1.11
3.31
3.91
3.29
3.34
4.09
3.74
4.18
4.90
5.43
4.12
3.84
4.17
5.55
4.85
4.25
4.66
3.93
4.54
5.08
3.51
4.53
7.93
2.92
2.95
15.32
5.59
1.10
4.92
1.34
3.09
8.40

P-value
0.0000
0.0036
0.0018
0.3908
0.4652
0.6348
0.0113
0.1226
0.5955
0.1202
0.5517
0.7971
0.2225
0.8086
0.5686
0.4027
0.0579
0.2580
0.2609
0.2466
0.4947
0.0939
0.0006
0.7919
0.4303
0.0008
0.0231
0.9670
0.0001
0.8057
0.4832
0.0076
0.0000
0.8136

29

3. Outpatient Treatment
Outpatient Treatment Unrelated to MHSA
As in all years of this study, there is evidence of extreme systematic bias in the data describing
outpatient treatment unrelated to MHSA. Overall, the population percentage of enrollees receiving
non-MHSA outpatient treatment is 62.34 percent. Consistent with prior years, the percentage increases
at each stage of the sampling process:


65.34 percent for frame eligible enrollees; 22.59 percent for frame ineligible enrollees;



71.59 percent for those with valid telephone numbers; 42.31 for those without (p<0.0001); and,



79.71 percent for telephone responders; 66.63 percent for telephone non-responders
(p<0.0001).

All of these stages result in a telephone survey estimate that overestimates the population percentage
by 17.34 percentage points, or 28 percent of the population value. The addition of the mail survey
moves the estimated utilization of 77.19 percent closer to the population value, but it is still significantly
higher.
This pattern of overestimation is consistent across VISNs, enrollee types, OEF/OIF/OND status, and
priority groups. However, the pattern is not as extreme in Priority Group 1 and 7, the two priorities with
the highest utilization, where the survey estimates overestimate the population by 5–7 percentage
points. The bias is greatest in priority groups with lower utilization. Table 9 below presents the priority
groups ordered from lowest to highest levels of utilization. The table includes the population utilization,
the survey estimate (based on all completes), and the percent overestimation.
Table 9. Priority groups ordered from lowest to highest levels of utilization

1

Priority
Group
6

Population
Utilization
44.38

Survey
Estimate
39.20

Percent
Overestimation
12%

2

8

51.03

46.45

9%

3

3

58.21

56.07

4%

4

5

62.90

59.13

6%

5

2

65.80

63.59

3%

6

4

78.13

75.49

3%

7

7

81.05

80.39

1%

8

1

83.21

81.47

2%

Order

Outpatient Treatment Related to MHSA
Overall, 15.14 percent of enrollees receive outpatient treatment for MHSA, and this percentage is higher
when restricted to telephone frame-eligible enrollees (15.78 percent). The percentage is significantly
higher for enrollees with valid contact information (16.61 percent) relative to those without (13.11
percent; p<0.0000). Both of these biases are similar to 2011. In 2012, there is no difference between
respondents and non-respondents, a departure from 2011. As with the other utilization measures, the
30

addition of mail mode reduces bias by allowing a more diverse and representative group of enrollees to
respond to the survey.
Priority Groups 1 and 4 have the highest percentage of enrollees receiving outpatient care for MHSA
(35.91 percent and 29.59 percent, respectively). In Priority Group 1, the percentage of enrollees with
valid contact information is significantly higher than those with invalid information (p=0.0002), but there
is no significant difference between respondents and non-respondents (p=0.9692 for telephone, and
p=0.6969 when including mail survey). Ultimately, the estimate, 36.46 percent, is close to the
population percentage, 35.91 percent.
For Priority Group 4, there are significant differences in the MHSA outpatient utilization percentage
between enrollees with valid contact information and those without valid information (p=0.3966), as
well as respondents and non-respondents (p<0.0012). However, the differences are in opposite
directions, and the estimate based on the respondents in this group is 28.48 percent, which is
reasonably close to the population percentage of 29.59 percent.
Figure 4. Percentage of Enrollees Receiving Outpatient Treatment
(a) For Mental Health or Substance Abuse (MHSA)
Out pat i ent Car e: Ment al Heal t h Rel at ed
15. 14%

Popul at i on
Fr ame

15. 78%

Phone

6. 74%

Mai l
I n Sampl e

Yes

15. 86% [ 15. 61%, 16. 11%]

El i gi bl e

Yes

16. 61% [ 16. 32%, 16. 90%]
13. 11% [ 12. 61%, 13. 62%]

No
Phone Resp

Al l Resp

Yes

16. 67% [ 16. 19%, 17. 15%]

No

16. 57% [ 16. 21%, 16. 93%]

Yes

16. 22% [ 15. 78%, 16. 66%]

No

15. 66% [ 15. 36%, 15. 97%]
0

20

40

60

80

100

Per cent
Sc al e: 100. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

31

Figure 4. Percentage of Enrollees Receiving Outpatient Treatment
(b) For Neither Mental Health nor Substance Abuse
Out pat i ent Car e: Not Ment al Heal t h Rel at ed
62. 34%

Popul at i on
Fr ame

65. 34%

Phone
22. 59%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

65. 30% [ 64. 95%, 65. 66%]
71. 59% [ 71. 20%, 71. 97%]
42. 31% [ 41. 53%, 43. 10%]

No
Phone Resp

79. 71% [ 79. 14%, 80. 29%]

Yes

66. 63% [ 66. 13%, 67. 13%]

No
Al l Resp

77. 19% [ 76. 64%, 77. 74%]

Yes

58. 88% [ 58. 43%, 59. 33%]

No
0

20

40

60

80

100

Per cent
Sc al e: 100. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

Table 10. Percentage of Enrollees Receiving Outpatient Treatment
(a) For Mental Health or Substance Abuse (MHSA)

Total
OEF/OIF/OND
OEF/OIF/OND
VISN

N
Y
1
2
3
4
5
6
7
8
9
10
11
12
15
16
17
18
19
20
21
22
23

Population
15.14
14.43
21.45
15.32
12.62
13.12
12.98
13.56
15.23
16.90
16.03
16.14
17.15
14.18
14.24
14.86
16.66
16.42
15.36
15.50
15.23
15.20
15.25
12.22

In Frame
Telephone
Mail
15.78
6.74
15.06
6.26
21.93 12.61
15.98
6.00
13.42
4.43
14.09
4.64
13.41
6.09
14.22
6.79
15.89
7.25
17.49
7.11
16.52
7.17
16.77
6.23
17.78
7.56
14.70
7.79
14.71
7.28
15.35
7.59
17.35
6.97
17.13
6.44
16.57
5.57
16.07
7.72
15.99
6.39
15.75
7.47
16.01
7.57
12.50
7.44

Sampled
Yes
15.86
15.14
22.00
15.67
13.83
13.57
14.23
14.87
15.61
18.39
16.43
16.80
18.37
14.10
14.83
16.57
17.02
16.57
16.36
16.33
15.30
16.82
16.38
11.92

Valid Telephone
PYes
No
value
16.61 13.11 0.0000
15.80 12.65 0.0000
23.94 16.37 0.0000
16.65 11.51 0.0000
14.43 11.30 0.0049
14.20 11.29 0.0012
14.47 13.23 0.3405
16.08 11.17 0.0000
16.29 13.17 0.0120
19.68 14.17 0.0001
17.11 13.98 0.0112
17.85 12.97 0.0006
19.18 15.27 0.0061
14.75 11.66 0.0149
15.80 11.06 0.0001
17.32 13.76 0.0104
17.37 15.81 0.2311
18.08 11.72 0.0000
17.11 13.70 0.0115
16.78 14.55 0.1203
16.03 12.71 0.0132
17.79 13.20 0.0006
17.08 14.01 0.0176
12.37
9.93 0.0502

Telephone Survey
Response
PYes
No
value
16.67 16.57 0.7400
15.87 15.76 0.7543
28.17 22.53 0.0000
15.72 17.18 0.2259
14.90 14.18 0.5299
15.02 13.88 0.2836
14.60 14.40 0.8674
15.25 16.47 0.3306
16.16 16.38 0.8647
19.28 19.94 0.6616
17.82 16.67 0.3839
16.77 18.66 0.1997
20.18 18.58 0.2618
13.80 15.38 0.2262
15.78 15.81 0.9797
17.49 17.21 0.8419
17.10 17.53 0.7482
19.68 17.20 0.0865
18.35 16.32 0.1631
17.49 16.27 0.3860
15.87 16.16 0.8353
17.78 17.80 0.9927
18.23 16.51 0.2157
11.52 13.01 0.2103

Telephone and Mail
Survey Response
PYes
No
value
16.22 15.66 0.0435
15.48 14.95 0.0688
27.03 20.64 0.0000
15.47 15.78 0.7783
14.71 13.41 0.2030
14.04 13.39 0.4758
14.01 14.36 0.7451
14.94 14.83 0.9215
15.76 15.52 0.8364
18.52 18.32 0.8768
17.28 15.96 0.2566
16.59 16.94 0.7901
20.02 17.51 0.0514
12.86 14.83 0.0849
15.25 14.60 0.5632
17.21 16.19 0.4221
17.32 16.87 0.7185
18.75 15.53 0.0105
16.97 16.01 0.4499
17.08 15.85 0.3257
15.57 15.12 0.7158
16.89 16.79 0.9393
18.43 15.49 0.0194
11.20 12.43 0.2486

32

SamIn Frame
pled
Valid Telephone
TelePPopulation
phone
Mail
Yes
Yes
No
value
Priority
1
35.91
36.49 20.73 36.25 36.91 32.63 0.0002
2
17.52
18.27
7.10 18.36 19.27 14.71 0.0000
3
10.95
11.66
3.83 11.72 12.42
9.35 0.0000
4
29.59
29.77 26.60 29.61 30.33 27.76 0.0012
5
15.20
15.81
8.37 15.97 16.58 14.20 0.0006
6
8.74
9.06
4.10 9.12
9.98
6.21 0.0000
7
9.31
9.32
8.96 9.02
8.40 13.38 0.0825
8
3.93
4.15
1.32 4.45
4.62
3.80 0.0396
Enrollee Type POST
13.09
13.42
7.25 13.43 14.10 10.96 0.0000
PRE
21.86
24.10
6.00 24.31 25.30 20.67 0.0000
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone Survey
Response
PYes
No
value
36.93 36.89 0.9692
19.11 19.35 0.7553
12.15 12.58 0.5281
28.88 31.16 0.0109
16.76 16.48 0.7217
9.28 10.34 0.2973
6.93
9.58 0.0886
4.37
4.78 0.3178
14.05 14.13 0.8188
25.78 25.00 0.1698

Telephone and Mail
Survey Response
PYes
No
value
36.46 36.12 0.6969
18.32 18.39 0.9252
11.71 11.72 0.9808
28.48 30.10 0.0390
16.47 15.73 0.2866
9.12
9.11 0.9948
7.65 10.11 0.1070
4.34
4.52 0.6137
13.69 13.29 0.2198
25.05 23.91 0.0259

33

Table 10. Percentage of Enrollees Receiving Outpatient Treatment
(b) For Neither Mental Health nor Substance Abuse
SamIn Frame
pled
Valid Telephone
Telephone Survey Response
Popula- Teletion
phone Mail
Yes
Yes
No
P-value
Yes
No
P-value
Total
62.34 65.34 22.59 65.30 71.59
42.31 0.0000
79.71
66.63 0.0000
OEF/OIF/OND N
63.66 66.91 21.86 66.86 73.16
43.17 0.0000
80.61
68.33 0.0000
OEF/OIF/OND Y
50.79 51.82 31.58 51.86 57.22
36.25 0.0000
66.81
54.02 0.0000
VISN
1
63.28 66.49 17.55 65.68 71.78
39.74 0.0000
80.27
66.96 0.0000
2
57.64 61.65 16.49 61.85 67.64
37.35 0.0000
76.09
63.15 0.0000
3
49.53 53.59 13.91 52.14 57.70
32.23 0.0000
68.32
53.43 0.0000
4
62.64 65.45 17.54 65.55 71.29
41.64 0.0000
82.76
64.73 0.0000
5
51.64 54.83 18.95 54.60 61.12
34.77 0.0000
67.64
58.07 0.0000
6
61.81 64.86 24.71 64.60 70.94
42.18 0.0000
78.33
66.39 0.0000
7
62.46 64.75 24.07 65.44 71.76
44.84 0.0000
76.63
68.69 0.0000
8
68.44 70.88 24.77 70.70 77.08
47.70 0.0000
84.56
72.41 0.0000
9
65.94 68.76 21.41 69.12 76.05
43.78 0.0000
83.34
70.60 0.0000
10
63.79 66.55 21.71 66.25 72.44
42.47 0.0000
79.64
68.13 0.0000
11
63.80 66.89 25.88 67.06 73.49
43.08 0.0000
81.51
68.16 0.0000
12
64.33 67.27 21.45 67.76 73.85
44.02 0.0000
79.79
70.20 0.0000
15
64.81 67.46 25.59 66.51 73.13
41.88 0.0000
82.23
67.18 0.0000
16
64.43 67.13 26.23 67.26 72.97
47.46 0.0000
82.79
67.19 0.0000
17
60.87 63.62 22.50 63.64 70.74
40.89 0.0000
78.58
66.40 0.0000
18
62.80 67.39 25.61 67.29 73.01
46.97 0.0000
82.31
67.05 0.0000
19
61.56 64.30 23.64 64.79 71.32
39.18 0.0000
77.37
66.99 0.0000
20
61.25 64.66 21.73 64.82 72.59
37.19 0.0000
78.80
67.74 0.0000
21
60.13 62.81 22.06 61.79 67.59
40.15 0.0000
74.87
63.45 0.0000
22
56.69 60.04 22.92 60.98 67.52
38.71 0.0000
77.60
62.47 0.0000
23
67.48 69.74 28.80 69.67 75.17
45.39 0.0000
81.17
70.64 0.0000
Priority
1
83.21 84.53 48.44 84.39 87.37
67.94 0.0000
90.22
85.40 0.0000
2
65.80 68.78 24.44 68.35 74.02
45.34 0.0000
79.85
70.61 0.0000
3
58.21 62.37 16.46 62.31 69.60
37.71 0.0000
76.57
65.42 0.0000
4
78.13 79.86 49.27 79.99 86.12
64.28 0.0000
91.68
82.88 0.0000
5
62.90 66.22 25.92 66.23 73.66
44.86 0.0000
83.91
67.87 0.0000
6
44.38 46.23 17.38 46.02 52.78
23.16 0.0000
63.95
46.93 0.0000
7
81.05 81.89 53.75 82.51 83.49
75.73 0.0294
85.45
81.90 0.1353
8
51.03 54.21 12.28 54.32 60.95
28.99 0.0000
71.45
54.41 0.0000
Enrollee Type POST
59.85 61.75 26.31 61.70 68.11
38.32 0.0000
76.99
62.68 0.0000
PRE
70.48 77.98 17.29 77.83 83.66
56.31 0.0000
89.19
80.30 0.0000
Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

Telephone and Mail
Survey Response
Yes
77.19
78.04
64.57
77.96
75.02
65.31
79.74
65.04
76.30
73.95
82.15
79.84
77.60
78.51
77.78
79.87
80.38
74.90
77.94
75.73
76.37
73.66
75.18
79.44
88.87
77.18
74.09
90.23
81.32
61.70
85.18
68.04
74.41
86.88

No
P-value
58.88
0.0000
60.39
0.0000
48.40
0.0000
59.39
0.0000
55.45
0.0000
47.13
0.0000
57.93
0.0000
50.44
0.0000
58.48
0.0000
60.78
0.0000
64.33
0.0000
62.24
0.0000
60.32
0.0000
60.34
0.0000
62.20
0.0000
58.71
0.0000
60.66
0.0000
58.26
0.0000
61.13
0.0000
57.86
0.0000
57.26
0.0000
55.63
0.0000
54.76
0.0000
62.71
0.0000
81.47
0.0000
63.59
0.0000
56.07
0.0000
75.49
0.0000
59.13
0.0000
39.20
0.0000
80.39
0.0326
46.45
0.0000
54.83
0.0000
72.96
0.0000

34

4. VHA Pharmacy Services
As in previous cycles, the pattern of enrollees reporting participation in the VHA pharmacy service very
closely follows the observed patterns for outpatient treatment unrelated to mental health or substance
abuse. The proportion of enrollees participating in this service is 55.25 percent; this increases to 58
percent for frame-eligible enrollees. There is an increase (p<0.0001) to 63.80 percent when limiting to
sampled enrollees with valid contact information, and another increase to 71.21 percent when
measuring telephone-responding enrollees. The addition of the mail survey reduces the bias to a
difference of 11.6 between the population figure and the survey estimate, but the final estimates are
still considerably higher than the population. This pattern is consistent across all strata: an increase in
the percentage from population to frame-eligible, with further increases in the percentage for enrollees
with valid contact information, and then more in responding enrollees. All comparisons between
enrollees with valid information and those without are significant. Further, all comparisons of
responding to non-responding enrollees are significant. This is a pattern similar to those in 2008, 2010,
and 2011.
Figure 5. Percentage of Enrollees Receiving Prescription Drug Services
Pr escr i pt i on Dr ug Benef i t
55. 25%

Popul at i on
Fr ame

58. 00%

Phone
18. 86%

Mai l
I n Sampl e

Yes

El i gi bl e

Yes

58. 08% [ 57. 71%, 58. 44%]
63. 80% [ 63. 39%, 64. 20%]
37. 15% [ 36. 38%, 37. 92%]

No
Phone Resp

71. 21% [ 70. 57%, 71. 85%]

Yes

59. 28% [ 58. 76%, 59. 79%]

No
Al l Resp

68. 88% [ 68. 28%, 69. 49%]

Yes

52. 24% [ 51. 79%, 52. 70%]

No
0

20

40

60

80

100

Per cent
Sc al e: 100. 0%

Sour ce: 2012 Sur vey of Vet er an Enr ol l ees' Heal t h and Rel i ance Upon VA

35

Table 11. Percentage of Enrollees Receiving Prescription Drug Services
Samp
led

In Frame
Populat
ion
Total

Telephone

Mail

Yes

Valid Telephone

Telephone Survey
Response

Yes

No

P-value

Yes

No

P-value

Telephone and Mail
Survey Response
Yes

No

P-value

55.25

58.00

18.86 58.08

63.80

37.15

0.0000

71.21

59.28

0.0000

68.88

52.24

0.0000

OEF/OIF/OND

N

57.06

60.07

18.57 60.16

65.90

38.56

0.0000

72.55

61.60

0.0000

70.14

54.39

0.0000

OEF/OIF/OND

Y

39.34

40.25

22.42 40.12

44.56

27.18

0.0000

52.09

42.04

0.0000

50.37

37.33

0.0000

VISN

1

54.64

57.51

13.80 57.41

62.60

35.33

0.0000

69.82

58.50

0.0000

67.86

52.06

0.0000

2

50.80

54.44

13.43 54.40

59.44

33.12

0.0000

66.05

55.92

0.0000

65.05

49.23

0.0000

3

42.71

46.29

11.29 45.22

50.12

27.70

0.0000

59.62

46.30

0.0000

57.04

40.73

0.0000

4

54.08

56.56

14.16 57.44

63.08

33.97

0.0000

73.36

57.18

0.0000

70.07

50.66

0.0000

5

44.70

47.56

15.46 47.10

52.60

30.37

0.0000

59.03

49.60

0.0000

56.60

43.32

0.0000

6

56.34

59.21

21.42 59.25

64.96

39.06

0.0000

71.99

60.63

0.0000

69.81

53.72

0.0000

7

56.16

58.28

20.52 59.38

65.33

39.98

0.0000

69.36

62.79

0.0006

67.02

55.20

0.0000

8

59.71

61.93

19.94 61.19

67.25

39.34

0.0000

73.81

63.14

0.0000

71.60

55.39

0.0000

9

59.28

61.93

17.56 62.44

69.08

38.13

0.0000

75.71

64.13

0.0000

72.48

56.00

0.0000

10

57.09

59.61

18.68 58.88

64.29

38.05

0.0000

71.19

60.17

0.0000

69.65

53.24

0.0000

11

57.46

60.30

22.52 60.03

65.63

39.15

0.0000

72.54

61.04

0.0000

69.80

54.30

0.0000

12

57.70

60.43

17.84 60.92

66.55

38.94

0.0000

72.35

63.01

0.0000

70.62

55.54

0.0000

15

58.77

61.26

21.90 60.94

67.01

38.35

0.0000

74.69

61.99

0.0000

73.17

53.80

0.0000

16

58.87

61.45

22.39 61.94

67.50

42.68

0.0000

76.52

62.19

0.0000

74.10

55.83

0.0000

17

54.89

57.48

18.69 57.33

64.12

35.55

0.0000

71.90

59.83

0.0000

68.87

51.82

0.0000

18

55.75

59.98

21.48 60.53

65.93

41.34

0.0000

74.00

60.77

0.0000

70.07

55.02

0.0000

19

53.54

55.98

19.74 57.33

63.04

34.90

0.0000

68.42

59.20

0.0000

66.61

51.44

0.0000

20

54.06

57.18

17.81 57.66

64.13

34.65

0.0000

71.12

58.68

0.0000

68.49

50.58

0.0000

21

52.43

54.85

17.98 52.97

57.87

34.66

0.0000

63.57

54.63

0.0000

62.61

47.97

0.0000

22

48.42

51.31

19.35 52.19

58.06

32.20

0.0000

67.02

53.56

0.0000

65.18

46.50

0.0000

23

58.83

60.89

23.46 60.73

65.62

39.19

0.0000

71.88

60.89

0.0000

70.02

54.13

0.0000

1

77.29

78.62

42.50 78.75

81.73

62.27

0.0000

84.60

79.75

0.0000

83.11

75.91

0.0000

2

55.44

58.06

19.09 57.67

62.45

38.31

0.0000

66.97

59.80

0.0000

64.88

53.79

0.0000

3

47.35

50.82

12.39 51.42

57.56

30.71

0.0000

62.88

54.37

0.0000

60.59

46.57

0.0000

4

74.58

76.30

46.06 76.64

82.65

61.23

0.0000

87.19

80.01

0.0000

85.91

72.57

0.0000

5

58.28

61.43

23.18 62.09

69.22

41.59

0.0000

79.28

63.53

0.0000

76.95

55.10

0.0000

6

32.93

34.38

11.70 33.95

39.34

15.74

0.0000

46.87

35.39

0.0000

44.95

29.17

0.0000

7

68.28

68.98

45.30 67.88

69.42

57.20

0.0050

72.59

66.85

0.0540

72.14

64.50

0.0063

8

45.12

48.04

9.39 47.60

53.64

24.52

0.0000

63.54

47.49

0.0000

60.39

40.27

0.0000

POST

51.77

53.51

21.26 53.62

59.33

32.75

0.0000

67.49

54.35

0.0000

65.12

47.40

0.0000

PRE

66.61

73.83

15.44 73.59

79.29

52.59

0.0000

84.14

76.33

0.0000

81.99

69.07

0.0000

Priority

Enrollee Type

Note: Statistical tests for independence are based on the Rao-Scott Chi Square statistic.

36

SURVEY WEIGHTING
In 2005, we conducted a non-response bias analysis for that year’s Survey of Enrollees. One of the
resulting recommendations was a propensity score weighting adjustment. This weighting adjustment,
also used in 2007 and 2008, corrects for the differential non-response by health utilization and
demographic information. To determine the adjustment, we:
 Used a probability model (described below) to estimate an enrollee’s individual propensity (or
probability) of being in the respondent sample;
 Grouped the estimated enrollees into five equal-sized classes (or quintiles) with similar
probabilities; and,
 Weighted the respondents up to account for the non-respondents, using an independent
adjustment for all classes.
The propensity score weighting adjustment reduces potential bias to the extent that non-respondents
and respondents with similar response probabilities are also similar with respect to the survey statistics
of interest.
During the 2007 Survey of Enrollees, enrollees were sampled only from a frame of enrollees with
telephone numbers. Enrollees without telephone numbers had no chance of selection—thereby
introducing coverage error. Therefore, the 2007 survey was susceptible to two forms of bias, coverage
of enrollees with no chance of selection and non-response bias among enrollees who did not respond.
For that reason, two separate propensity score adjustments were developed: one for frame coverage
and another for non-response.
Since the 2008 Survey of Enrollees, the survey sample has been selected from a frame of enrollees with
and without telephone numbers. Since the sample has been selected from this complete frame,
coverage bias has not been a concern. However, non-response from a variety of sources, including
invalid contact information, has remained a concern. Some of these sources have been addressed
through the addition of a mail survey and a Web response channel. However, some remain. Therefore,
a single propensity score adjustment has been used to focus on mitigating non-response bias.

Design Weights
Prior to calculating the non-response adjustment, we adjusted for differential selection probabilities.
The sample was selected from the survey frame independently in each of the strata defined by VISN,
priority, pre- and post- status and OEF/OIF/OED status; so the probability of selection is calculated in
each stratum as Pr 
-

n
, where:
N

Pr is the sampling probability, n=159,577 is the sample5 size of enrollees, and

5

The sample was selected in two stages. VHA provided a sample (n1) of 419,991 enrollees. From this sample, we selected the
final sample (n), a sub-sample of 159,577 enrollees, to meet all targets by OEF/OIF/OND, VISN, enrollee type, and priority
status. The two stages allowed flexibility to reach targets without the need for multiple data transfers between VHA and ICF.
The probability of selection in each strata is: Pr



n1 n
n
 
N n1 N
37

N=8,013,308 is the total number of enrollees.
The inverse of these selection probabilities is the design weight, w1=1/Pr. The design weights were
used in calculating the non-response adjustment.

Non-Response Adjustment
To calculate the non-response adjustment, each sampled respondent was classified into a non-response
category using the indicator variable y based on whether we obtained an interview:

0 if nonresponse; no interview was obtained
y
1 if response; an interview was obtained
Using logistic regression, we estimated the probability that an enrollee completed the interview given
his or her characteristics:

e xβ
Pr( y  1 | x) 
, where x is a matrix of sampled enrollees; each enrollee has a set of p
1  e xβ

covariates, xi  (1, x1i ,...x pi ) for enrollee i. This set was used as explanatory (or predictor) variables,
and β  (  0 , 1 ,...,  p ) was a set of regression coefficients, or parameters. The predictor variables

included the sample design variables (OEF/OIF/OND, VISN, priority status, and enrollee type), the seven
administrative health measures (see below), and demographic variables (age and gender).
VHA provides a file (based on administrative records) that indicates whether an enrollee had utilized any
of the following services in the previous year (the file does not indicate the frequency or amount paid):
1. Received long-term care benefits
a. Institutional
b. Non-institutional
2. Inpatient treatment
c. MHSA
d. Non-MHSA
3. Outpatient treatment
a. MHSA
b. Non-MHSA
4. VHA pharmacy services
The utilization indicators have been used in the weighting process since 2007. From 2007–2010, the
indicators were based on service utilization sourced from VHA workload files that were based on bed
section and clinic stop. This categorization indicates where a Veteran received care. For the 2011
survey, the indicators were based on service utilization from Health Service Categories (HSCs). The
categorization indicates what care a Veteran received. A second change in 2011 included long-term
care in institutions and non-institutions. From 2007–2010, the indicator was a single measure of home
health service.

38

For the modeling, in each stratum, we used design weights equal to the ratio of the frame total
compared to the sample total. The outcome of the logistic regression model is the propensity score, the
estimated probability that the enrollee is in the final sample of respondents, given their characteristics
(VISN, priority status, enrollee type, age, gender, and service utilization). In 2012, we added to the
model an indicator of whether the enrollee was eligible for the telephone frame.
After estimating each sampled enrollee’s probability of completing an interview based on the predictor
variables, respondents and non-respondents were grouped into quintiles based on their propensity
score. Within each quintile, the design weights were increased by the ratio of the total design weight
for both responders and non-responders to the total design weight for responders only. This resulted in
numbers that represented the total population of enrollees. The first quintile represents the enrollees
with the lowest propensity scores; this means that these enrollees are less likely to be in the final
sample of respondents; thus, they receive the largest weights. The last quintile represents the enrollees
with the highest propensity scores; this means that these enrollees are more likely to be in the final
sample of respondents; thus, they receive the smallest weights.
Table 12. Non-Response Adjustment
Non-Response
Response Non-Response
Adjustment
First quintile:
0–20th percentile
Second quintile:
20–40th percentile
Third quintile:
40–60th percentile
Fourth quintile:
60–80th percentile
Fifth quintile:
80–100th percentile

257,402

1,345,181

6.23

440,505

1,162,178

3.64

574,547

1,028,152

2.79

711,982

890,605

2.25

813,765

788,993

1.97

Each respondent’s design weight was multiplied by the adjustment factor (NR) from the quintile where
he or she fell to calculate the non-response adjusted weights, w2 = w1×NR.
The preceding bias analysis was based on weighted data that accounts for the differential sampling
probabilities for each stratum and does not adjust for non-response. We also performed the bias
analysis using the non-response weights to determine whether the non-response adjustment reduces
the biases observed for the health estimates. These results are listed in Table 13, below.

39

Table 13. Survey Estimates and Bias for Weighted, Unweighted, and Adjusted Data
Telephone
Base Weight

Esti
Population mate

Bias

Lowe
Boun
d

Upper
Bound

Telephone and Mail

Telephone and Mail Base
Weight

Base Weight

and Non-Response Adjustment

Esti
mate

Bias

Uppe
Lowe
r
Boun Boun Estimat
d
d
e

Bias

Uppe
r
Lowe Boun
Bound
d

1. Long-term care
(a) Institutional

0.54

0.42 -0.12 -0.20

0.44 -0.10 -0.18 -0.02

0.63

0.09

-0.03

0.21

(b) Non-institutional

2.73

3.42

0.45

0.93

3.27

0.76

2.95

0.22

0.03

0.42

(a) Related to MHSA

1.18

0.84 -0.34 -0.46

-0.23

0.85 -0.33 -0.44 -0.22

1.28

0.10

-0.06

0.25

(b) Unrelated to MHSA

4.25

5.14

0.89

0.58

1.20

5.04

0.79

0.51

1.07

4.60

0.35

0.10

0.61

(a) Related to MHSA

15.14 16.67

1.53

1.05

2.01 16.22

1.07

0.63

1.51

15.98

0.84

0.41

1.27

(b) Unrelated to MHSA

62.34 79.71 17.37 16.80

17.95 77.19 14.85 14.30 15.40

65.62

3.27

2.58

3.97

55.25 71.21 15.96 15.32

16.60 68.88 13.63 13.03 14.23

58.44

3.19

2.50

3.87

0.69

-0.03

0.55

0.33

2. Inpatient treatment

3. Outpatient treatment

4. VHA Pharmacy service

The table above presents the bias estimates based on a telephone-only design, bias estimates based on a telephone and
mail design (including responses in all channels), and bias estimates after conducting the non-response adjustment. In
each case, adding the mail survey reduces bias, in amounts ranging from 2.33 percentage points to a tenth of a
percentage point. Adding the non-response weighting then reduces the bias even further, in amounts ranging from 12
percentage points to a tenth of a percentage point. The large biases for outpatient treatment unrelated to MHSA and
pharmacy services seen in the last two sets of rows in Table 10 were reduced from 16–17 percentage points down to 3–
4 percentage points.

40

DISCUSSION AND RECOMMENDATIONS
This is the fifth report in the Experimental Methods Series. Recommendations that have stemmed from
the annual analyses are to:





Use propensity score weighting based on utilization of administrative records (Full adoption);
Send a pre-survey notification letter to Veterans prior to calling (Full adoption);
Increase the call attempts from 6 to 7 (Full adoption);
Use address information to locate and update telephone numbers via database look-ups (Mixed
adopton: full adoption based on experiments in 2008 and 2010; not implemented in 2011 due to
security and privacy concerns; implemented sparingly in 2012 for 7-digit telephone numbers and
invalid area codes);
 Add a mail survey (Partial adoption as described in the current report); and,
 Add a Web survey (Full adoption).
In 2012, thirteen percent of enrollees used the Web survey option instead of returning a mail survey or
conducting a telephone survey. Web instruments are an effective way to reduce cost for large surveys.
Programming the survey is a one-time expense and interviewer labor is removed. If 13 percent of
respondents use the Web for a large survey such as the Survey of Enrollees, this can result in
considerable cost savings on interviewer labor. Moreover, the Web survey provides a response channel
that allows respondents to participate at their convenience.
Recommendation: VHA continue using the Web survey option.
For the mail survey the response rate and bias reduction benefits are positive. Counting responses via
all four response channels (i.e., web, mail, inbound phone, and outbound phone) the addition of a mail
component (mail survey, allowing mail requests, and mail follow-up) added 10,056 interviews. We can
use the mail survey to improve results in a number of different ways: for those with no telephone
number listed; for those with a nonworking telephone number listed; for those who would prefer to
respond to a print survey rather than conduct a telephone interview; and, as a nonresponse follow-up.
In each case, the Survey of Enrollees benefited from increased response. The response rate to the mail
survey was the same as that for the telephone survey. Considering that the enrollees with telephone
numbers on the frame tend to be the most vested in VHA services (higher utilization), which is
associated with response rates, it is conceivable that a mail survey might result in higher response than
a telephone survey if conducted on a larger scale.
Recommendation: VHA conduct an experiment where a sample of enrollees with both an address and a
telephone number are randomly assigned to:
1. Mail-first, telephone follow-up treatment
2. Telephone-first, mail follow-up treatment (similar to 2012)
The benefits of such an experiment will be to compare overall response to these two designs, as well as
to compare response differences between the two modes. This will address a limitation to the 2012
design: despite receiving both mail responses and telephone responses, the response sets are from
different enrollee groups (those with a phone number and those without, those who refused the phone

41

survey and those who did not, etc.). Thus, we do not know if differences are due to mode or enrollee
group.
Of the 15,761 enrollees who were sent a mail survey as a non-response follow-up, 2,705, or 18 percent
completed a survey. We know that the telephone responders are different from telephone nonresponders in terms of the HSC utilization indicators. However, the survey responses are very similar
when comparing the telephone responders to the telephone non-responders who responded by mail.
Our 2012 bias analysis suggests that while we are reducing bias by adding the mail survey, we still have
differences between survey responders (mail and phone) and non-responders.
These telephone non-responders were sent one survey packet. They were not sent a postcard reminder
or a second survey packet. A more rigorous follow-up protocol might yield more responses from these
telephone non-responders, which should continue to reduce bias.
As part of our continuing research on improvements to the sampling and weighting methodologies, we
will explore mode effects, including the option of including a weighting adjustment that would support
trend analysis.
Recommendation: Add a postcard reminder and a second survey mailing for the telephone nonresponders.
Recommendation: Include all telephone non-responders in the mail follow-up.
The benefits of moving from partial to full adoption of the mail survey, both in terms of broadening the
application and the use of a more rigorous mail follow-up protocol, should be balanced against
increased costs and a lengthened fielding period.
Operational challenges included item level non-response on the mail surveys, the handling of duplicated
surveys, and the need to interpret hand written comments on mail surveys. We can begin assessing the
costs of addressing these issues by extrapolating mail component costs from the 2012 study to a full
application using estimated increases in mail volume.
Methodological components of this assessment include looking at statistical adjustments to ensure
comparability for cross-year analysis and the design of an embedded experiment to untangle the modeand group-effects present in the current sample design.

42