A-11 Survey Requirements Document

Request for Approval under the “Generic Clearance for Improving Customer
Experience (OMB Circular A-11, Section 280 Implementation)” (OMB
Control Number: 2900-0876)
TITLE OF INFORMATION COLLECTION: Community Care Customer Satisfaction
Survey
PURPOSE
The Community Care Survey is designed to measure customer experience with a number of
interactions they may have in navigating Community Care benefits. Currently, there is limited
feedback and understanding of the Veteran Community Care Customer Experience (CX) in VA and
VHA. Specifically, the VHA Office of Community Care (OCC) does not have real-time customer
experience feedback mechanisms and processes to collect and measure their customer experience. This
limits their ability to provide proactive service recovery and evaluate and deploy process improvements
to improve VHA Community Care customer experiences.
The lack of a centralized VHA Community Care program feedback loop with corresponding analytics,
trending and case management inhibits the Office of Community Care's ability to: respond rapidly to
emerging issues; gauge impacts from new processes and legislation at a Community Care enterprise
level; provide service recovery for negative experiences; and ultimately regain Veterans' trust in VA
and Community Care services.
MISSION Act implementation was implemented June 6, 2019 and the start of healthcare delivery for
the new Community Care Network contract was June 26, 2019. The Veteran Choice costs the VA
approximately $1 Billion/annually.
Veterans experience data is collected by using an online transactional survey disseminated via an
invitation email sent to randomly selected beneficiary. The data collection occurs once per week with
invitation being sent out within 8 days of calling the Community Care. The questionnaire is brief and
contains general Likert-scale (a scale of 1-5 from Strongly Disagree to Strongly Agree) questions to
assess customer satisfaction as well as questions assessing the knowledge, speed, and manner of the
interaction. After the survey has been distributed, recipients have two weeks to complete the survey and
will receive a reminder email after one week.
DESCRIPTION OF RESPONDENTS:
The target population of the Community Care Survey is defined
as any Veterans who has interacted with Community Care in the past weeks. The table below
summarizes
the qualifying interactions.

1

Table 1. Target Population for Each Survey
Survey
Survey 1: Choosing VA Community Care
Survey 2: Scheduling a VA Community Care
Appointment
Survey 3: Attending a VA Community Care
Appointment
Survey 4: Filling a Prescription Through the
VA Community Care
Survey 5: Understanding Financial
Responsibility with VA Community Care
Survey 6: Calling VA About Community Care
Billing Questions

Qualifying Interaction
Selected Community Care coverage
Made a Community Care appointment
Saw a Community Care provider
Filled a prescription through Community Care
Receive a bill for Community Care
Contacted VA about Community Care billing

The sample frame is prepared by extracting population information directly from VHA’s
Corporate Data Warehouse. These extracts are also used to obtain universe figures for the
sample weighting process. The Veteran is the primary sampling unit and is randomly selected
from the population according to a stratified design. The primary stratification will be the type of
contact which fall into 3 strata—nurse triage, pharmacy, and Licensed Independent Provider
(LIP). The survey will also utilize implicit stratification or balancing by age, gender, and
location.
TYPE OF COLLECTION: (Check one)
[ ] Customer Comment Card/Complaint Form
[ ] Usability Testing (e.g., Website or Software
[ ] Focus Group

[X] Customer Satisfaction Survey
[ ] Small Discussion Group
[ ] Other: ______________________

CERTIFICATION:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low-burden for respondents and low-cost for the Federal Government.
3. The collection is non-controversial and does not raise issues of concern to other federal
agencies.
4. Personally identifiable information (PII) is collected only to the extent necessary and is not
retained.
5. Information gathered is intended to be used for general service improvement and program
management purposes.
6. The collection is targeted to the solicitation of opinions from respondents who have
experience with the program or may have experience with the program in the future.
7. All or a subset of information may be released as part of A-11, Section 280 requirements on
performance.gov. Additionally, summaries of the data may be released to the public in
2

communications to Congress, the media and other releases disseminated by VEO, consistent
with the Information Quality Act.
Name: Evan Albert, Director of Measurement and Data Analytics (Acting), Veterans Experience
Office Evan.Albert@va.gov (202) 875-478
To assist review, please provide answers to the following question:
Personally Identifiable Information:
1. Will this survey use individualized links, through which VA can identify particular
respondents even if they do not provide their name or other personally identifiable
information on the survey? [ X ] Yes [] No
2. Is personally identifiable information (PII) collected? [ ] Yes [X] No
3. If Yes, will any information that is collected be included in records that are subject to the
Privacy Act of 1974? [ ] Yes [ ] No [N/A]
4. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No
[N/A]
Gifts or Payments:
Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to
participants? [ ] Yes [ X] No
BURDEN HOURS
Category of Respondent

Individuals & Households
VA Form (if applicable)
Totals

No. of
Respondents

Participation
Time
( X minutes =)

Burden
(÷ 60 =)

54,000 annually

3

2,700

54,000 annually

3

2,700

Please answer the following questions.
1. Are you conducting a focus group, a survey that does not employ random sampling,
user testing or any data collection method that does not employ statistical methods?
[ ] Yes [X]No
If Yes, please answer questions 1a-1c, 2 and 3.
If No, please answer or attach supporting documentation that answers questions 2-8.
a. Please provide a description of how you plan to identify your potential group of
respondents and how you will select them.

3

b. How will you collect the information? (Check all that apply)
[ ] Web-based or other forms of Social Media
[ ] Telephone
[ ] In-person
[ ] Mail
[X] Other- E-mail-based surveys
c. Will interviewers or facilitators be used? [ ] Yes [ X ] No
2. Please provide an estimated annual cost to the Federal government to conduct this data
collection: __$13,000______
3. Please make sure that all instruments, instructions, and scripts are submitted with the request.
This includes questionnaires, interviewer manuals (if using interviewers or facilitators), all
response options for questions that require respondents to select a response from a group of
options, invitations given to potential respondents, instructions for completing the data
collection or additional follow-up requests for the data collection.
-Done
4. Describe (including a numerical estimate) the potential respondent universe and any
sampling or other respondent selection methods to be used. Data on the number of entities
(e.g., establishments, State and local government units, households, or persons) in the
universe covered by the collection and in the corresponding sample are to be provided in
tabular form for the universe as a whole and for each of the strata in the proposed sample.
Indicate expected response rates for the collection as a whole. If the collection had been
conducted previously, include the actual response rate achieved during the last collection.
- Please see Statistical Sample Plan in the Appendix.
5. Describe the procedures for the collection of information, including:
a. Statistical methodology for stratification and sample selection.
b. Estimation procedure.
c. Degree of accuracy needed for the purpose described in the justification.
d. Unusual problems requiring specialized sampling procedures.
e. Any use of periodic (less frequent than annual) data collection cycles to reduce
burden.
- Please see Statistical Sample Plan in the Appendix.
6. Describe methods to maximize response rates and to deal with issues of nonresponse. The
accuracy and reliability of information collected must be shown to be adequate for intended
uses. For collections based on sampling, a special justification must be provided for any
collection that will not yield "reliable" data that can be generalized to the universe studied.
Please see Statistical Sample Plan in the Appendix.
4

7. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an
effective means of refining collections of information to minimize burden and improve
utility. Tests must be approved if they call for answers to identical questions from 10 or more
respondents. A proposed test or set of tests may be submitted for approval separately or in
combination with the main collection of information.
Please see Statistical Sample Plan in the Appendix.
8. Provide the name and telephone number of individuals consulted on statistical aspects of the
design and the name of the agency unit, contractors, grantees, or other person(s) who will
actually collect or analyze the information for the agency.
Statistical Aspects:
Mark Andrews, Statistician, Veterans Experience Office, VA. (703)
483-5305
Collection and Analysis: Evan Albert, Director of Measurement and Data Analytics,
Veterans Experience Office, VA (202) 875-9478
Lilian Cuadra, Director, Customer Experience, CSC, (303) 3317551
,

APPENDIXSTATISTICAL SAMPLE
PLAN

Service Level Measurements –
Community Care Survey
Sampling Methodology Report
Prepared by
Veteran Experience Office
Version 1
5

June 2020

6

Contents

Executive Summary....................................................................................................................................... 8
Part I – Introduction ...................................................................................................................................... 8
A. Background .......................................................................................................................................... 8
B. Basic Definitions ................................................................................................................................ 10
C. Application to Veterans Affairs ......................................................................................................... 10
Part II – Methodology ................................................................................................................................. 10
A. Target Population and Frame ............................................................................................................ 10
B. Sample Size Determination............................................................................................................. 11
C. Data Collection Methods ................................................................................................................ 13
D. Reporting......................................................................................................................................... 13
E. Quality Control ............................................................................................................................... 14
F. Sample Weighting, Coverage Bias, and Non-Response Bias ......................................................... 14
G. Quarantine Rules ............................................................................................................................. 16
Part III – Assumptions and Limitations ....................................................................................................... 16
A. Respondent Satisfaction Bias ............................................................................................................. 16
B. Coverage Bias..................................................................................................................................... 16
Appendix 1. List of Data Extraction Variables .................................................................................. 16
Appendix 2. Survey Questions ............................................................................................................. 16
Appendix 3. References ........................................................................................................................ 19

7

Executive Summary
The Community Care Survey is designed to measure customer experience with a number
of interactions they may have in navigating Community Care benefits.
Veterans experience data is collected by using an online transactional survey
disseminated via an invitation email sent to randomly selected beneficiary. The data collection
occurs once per week with invitation being sent out within 8 days of calling the Community
Care. The questionnaire is brief and contains general Likert-scale (a scale of 1-5 from Strongly
Disagree to Strongly Agree) questions to assess customer satisfaction as well as questions
assessing the knowledge, speed, and manner of the interaction. After the survey has been
distributed, recipients have two weeks to complete the survey and will receive a reminder email
after one week.
The overall sample size for the Community Care Survey population is selected to assure
that, for each of the interactions targeted where sufficient sample is available, the reliability of
monthly survey estimate is a +/-3% margin of error at a 95% confidence level. The survey will
be sent to a representative sample of Veterans. Once data collection is completed, the participant
responses in the online survey will be weighted.
This report describes the methodology used to conduct the Community Care Survey.
Information about quality assurance protocols, as well as limitations of the survey methodology,
is also included in this report.
Part I – Introduction
A. Background
The Enterprise Measurement and Design team (EMD) is part of the Insights and
Analytics (I&A) division within the Veterans Experience Office (VEO). The EMD team is
tasked with conducting transactional surveys of the Veteran population to measure their
satisfaction with the Department of Veterans Affairs (VA) numerous benefit services. Thus, their
mission is to empower Veterans by rapidly and discreetly collecting feedback on their
interactions with such VA entities as NCA, VHA, and VBA. VEO surveys generally entail
probability samples which only contact minimal numbers of Veterans necessary to obtain
reliable estimates. This information is subsequently used by internal stakeholders to monitor,
evaluate, and improve beneficiary processes. Veterans are always able to decline participation
and have the ability to opt out of future invitations. A quarantine protocol is maintained to limit
the number of times a Veteran may be contacted, in order to prevent survey fatigue, across all
VEO surveys.
The VEO team designed six questionnaires for the Community Care Survey for each of the
interaction targeted:
• Survey 1: Choosing VA Community Care
• Survey 2: Scheduling a VA Community Care Appointment
• Survey 3: Attending a VA Community Care Appointment
• Survey 4: Filling a Prescription Through the VA Community Care
• Survey 5: Understanding Financial Responsibility with VA Community Care
• Survey 6: Calling VA About Community Care Billing Questions
8

In order to continue to provide quality services to Veterans, VEO has been commissioned to
measure the satisfaction with Community Care. To complete this goal, VEO proposed to conduct
a brief transactional survey with selected Veterans who had interacted with Community Care.
The surveys consists of seven questions developed using human-centered design, focusing on
Veterans’ experience with regard to their recent encounter and centered on to the factors of
Trust, Ease, Effectiveness, Helpfulness, Quality and Emotion. These Likert-scale (a scale of 1-5)
questions are designed through extensive Veteran input and recommendations from subject
matter experts in the VA. Veterans also have an opportunity to provide a free-text response about
their experience.
Veterans are selected to participate in the survey via an invitation email. A link is
enclosed so the survey may be completed using an online interface, with customized participant
information. The data is collected on a weekly basis and the survey is reported on a monthly
basis. The purpose of this document is to outline the planned sample design and provide a
description of the data collection and sample sizes necessary for proper reporting.

9

B. Basic Definitions
Coverage

The percentage of the population of interest that is
included in the sampling frame.
The difference between the response coded and the true value
of the characteristic being studied for a respondent.
Failure of some respondents in the sample to provide
responses in the survey.
A transaction refers to the specific time a Veteran interacts
with the VA that impacts the Veteran’s journey and their
perception of VA’s effectiveness in caring for Veterans.
The ratio of participating persons to the number of contacted
persons. This is one of the basic indicators of survey quality.
In statistics, a data sample is a set of data collected and/or
selected from a statistical population by a defined procedure.
Error due to taking a particular sample instead of measuring
every unit in the population.
A list of units in the population from which a sample may be
selected.
The consistency or dependability of a measure. Also referred
to as standard error.

Measurement Error
Non-Response
Transaction
Response Rate
Sample
Sampling Error
Sampling Frame
Reliability

C. Application to Veterans Affairs
Customer experience and satisfaction are usually measured at three levels to: 1) provide
enterprises the ability to track, monitor, and incentivize service quality; 2) provide service level
monitoring and insights; and 3) give direct point-of-service feedback. This measurement may
bring insights and value to all stakeholders at VA. Front-line VA leaders can resolve individual
feedback from Veterans and take steps to improve the customer experience; meanwhile VA
executives can receive real-time updates on systematic trends that allow them to make changes.
1) To collect continuous customer experience data
2) To help field staff and the national office identify areas of improvement.
3) To understand emerging drivers and detractors of customer experience.
Part II – Methodology
A. Target Population, Frame, and Stratification
The target population of the Community Care Survey is defined as any Veterans who has
interacted with Community Care in the past weeks. The table below summarizes the qualifying
interactions.

10

Table 1. Target Population for Each Survey
Survey
Qualifying Interaction
Survey 1: Choosing VA Community Care
Selected Community Care coverage
Survey 2: Scheduling a VA Community Care
Made a Community Care appointment
Appointment
Survey 3: Attending a VA Community Care
Saw a Community Care provider
Appointment
Survey 4: Filling a Prescription Through the
Filled a prescription through Community Care
VA Community Care
Survey 5: Understanding Financial
Receive a bill for Community Care
Responsibility with VA Community Care
Survey 6: Calling VA About Community Care
Contacted VA about Community Care billing
Billing Questions
The sample frame is prepared by extracting population information directly from VHA’s
Corporate Data Warehouse. These extracts are also used to obtain universe figures for the
sample weighting process. The Veteran is the primary sampling unit and is randomly selected
from the population according to a stratified design. The primary stratification will be the type of
contact which fall into 3 strata—nurse triage, pharmacy, and Licensed Independent Provider
(LIP). The survey will also utilize implicit stratification or balancing by age, gender, and
location.

B. Sample Size Determination
To achieve a certain level of reliability, the sample size for a given level of reliability is
calculated below (Lohr, 1999):
For a population that is large, the equation below is used to yield a representative sample
for proportions:
2
𝑍𝑍𝛼𝛼/2
𝑝𝑝𝑝𝑝
𝑛𝑛0 =
𝑒𝑒 2
where
•
•

•

𝒁𝒁𝜶𝜶/𝟐𝟐 = is the critical Z score which is 1.96 under the normal distribution when using a 95%
confidence level (α = 0.05).
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
o Note that pq attains its maximum when value p=0.5 or 50%. This is what is typically
reported in surveys where multiple measures are of interest. When examining
measures closer to 100% or 0% less sample is needed to achieve the same margin
of error.
e = the desired level of precision or margin of error. For example, for the Community Care
Survey the targeted margin of error is e = 0.03, or +/-3.0%.

For a population that is relatively small, the finite population correction is used to yield a
representative sample for proportions:
𝑛𝑛0
𝑛𝑛 =
𝑛𝑛
1 + 𝑁𝑁0
11

Where
•
•

𝒏𝒏𝟎𝟎 = Representative sample for proportions when the population is large.
N = Population size.

The margin of error surrounding the baseline proportion is calculated as:
𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑜𝑜𝑜𝑜 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 = 𝑧𝑧𝛼𝛼/2 �

𝑁𝑁 − 𝑛𝑛 𝑝𝑝(1 − 𝑝𝑝)
�
𝑛𝑛
𝑁𝑁 − 1

Where
•
𝒁𝒁𝜶𝜶/𝟐𝟐 = 1.96, which is the critical Z score value under the normal distribution when using
a 95% confidence level (α = 0.05).
•
N = Population size.
•
n = Representative sample.
•
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
The proposed sample plan is designed to achieve an MOE of +/-3.0% at a 95%
confidence for all surveys where sufficient sample is available. If insufficient sample is available
given a maximum sample rate of 50%, the sample target accuracy will be set as per the table
below. Each survey will have a maximum sample of 1,850 drawn weekly and a sample rate of no
more than 50%
Table 2A indicates the amount of sample needed in a month to achieve various target
accuracies (MOE/ Confidence). As the sample is drawn, population estimates will be made to
determine the target accuracy appropriate for each of the six surveys. In this way, the sample will
adjust depending on the sample availability and
Table 2A. Target Population Figures, Sample Size, and Email Contacts

Average Monthly
Encounter w/
Email Addresses
> 14,228
> 10,453
> 8,003
> 6,324
> 5,122
> 3,608
> 2,185

Target MOE 1
3.00%
3.50%
4.00%
4.50%
5.00%
5.00%
5.00%

Confidence
95%
95%
95%
95%
95%
90%
80%

Minimum
Monthly Responses Needed
1,068
784
601
475
385
271
164

Response
Rates
15%
15%
15%
15%
15%
15%
15%

Minimum
Monthly Sample
Needed
7,115
5,227
4,002
3,162
2,562
1,805
1,093

Table 2B shows the estimated minimum and rounded sample size needed for each target
accuracy.

1

MOE measures assume that non-response to the survey is randomly distributed.

12

Table 2B shows the weekly sample availability and sample needs.
Target MOE

Minimum Weekly Sample
Needed
1,779
1,307
1,001
791
641
452
274

Confidence
3.00%
3.50%
4.00%
4.50%
5.00%
5.00%
5.00%

95%
95%
95%
95%
95%
90%
80%

Rounded Weekly Sample
Targets
1,850
1,360
1,050
830
680
480
300

The sample will be drawn using a systematic sampling methodology. This statistical valid
approach allows the team to balance the sample across several variables such as age, gender, and
region. These balancing variables are often referred to as implicit strata. This has been shown to
stabilize trends and improve accuracy of estimates.
Email addresses will be acquired by matching Veteran ID numbers to the VBA’s Enterprise Data
Warehouse (EDW) and the VHA’s Corporate Data Warehouse (CDW). The CDW will be
prioritized if the two sources produce different and valid email addresses. Each email address
encountered is validated in several ways:
•
•
•
•
•

Validation that the email address has a valid structure
Comparison with a database of bad domains
Correction of common domain misspellings
Comparison of a database of bad emails including
o Opt outs
o Email held by multiple veterans
Comparison to a database of valid TDLs (e.g. “.com”, “.edu”)

C. Data Collection Methods
Invitations will be sent out each week to assure that initial invites are sent within eight
days of their call to the Community Care. Caller information will be regularly extracted from
VHA database resource: the VHA’s Corporate Data Warehouse (CDW). The extraction process
will be executed and validated by the Office of Performance Improvement and Assessment
(PA&I). with the population extracts sent to VEO twice a week. Invitation will be sent on
Mondays. Invitees that have not completed the survey will receive a reminder after one week.
The survey will remain open for a total of two weeks. Survey responses are immediately
available within VSignals as soon as feedback is submitted.

D. Reporting
Researchers will be able to use the Veteran Signals (VSignals) system for interactive
reporting and data visualization. VA employees with a PIV card may access the system at
https://va.voice.medallia.com/sso/va/. The scores may be viewed by Age Group, Gender, and
Race/Ethnicity in various charts for different perspective. They are also depicted within time

13

series plots to investigate trends. Finally, filter options are available to assess scores at varying
time periods and within the context of other collected variable information.
Recruitment is continuous but the results should be combined into a monthly data file for
more precise estimates, at the call center level. Short interval estimates are less reliable for small
domains, (i.e., VAMC-level) and should only be considered for aggregated
populations. Monthly estimates will have larger sample sizes, and therefore higher reliability.
Estimates over longer periods are the most precise but will take the greatest amount of time to
obtain and are less dynamic in that trends and short-term fluctuation in service delivery may be
missed. Users examining subpopulation should be particularly diligent in assuring that insights
stem from analysis with sufficient sample in the subpopulations being examined or compared.

E. Quality Control
To ensure the prevention of errors and inconsistencies in the data and the analysis, quality
control procedures will be instituted in several steps of the survey process. Records will undergo
a cleaning during the population file creation. The quality control steps are as follows.
1. Records will be reviewed for missing sampling and weighting variable data. When
records with missing data are discovered, they will be either excluded from the
population file or put into separate strata upon discussion with subject matter experts.
2. Any duplicate records will be removed from the population file to both maintain the
probabilities of selection and prevent the double sampling of the same Veteran.
3. Invalid emails will be removed.
The survey sample loading and administration processes will have quality control
measures built into them.
1. The survey load process will be rigorously tested prior to the induction of the survey to
ensure that sampled customers is not inadvertently dropped or sent multiple emails.
2. The email delivery process is monitored to ensure that bounce-back records will not hold
up the email delivery process.
The weighting and data management quality control checks are as follows:
1. The sum of the weighted respondents will be compared to the overall population count to
confirm that the records are being properly weighted. When the sum does not match the
population count, weighting classes will be collapsed to correct this issue.
2. The unequal weighting effect will be used to identify potential issues in the weighting
process. Large unequal weighting effects indicate a problem with the weighting classes,
such as a record receiving a large weight to compensate for nonresponse or coverage
bias.

F. Sample Weighting, Coverage Bias, and Non-Response Bias
Weighting is commonly applied in surveys to adjust for nonresponse bias and/or
coverage bias. Nonresponse is defined as failure of selected persons in the sample to provide
responses. This is observed virtually in all surveys, in that some groups are more or less prone to
14

complete the survey. The nonresponse issue may cause some groups to be over- or underrepresented. Coverage bias is another common survey problem in which certain groups of
interest in the population are not included in the sampling frame. The reason that these Veterans
cannot participate is because they cannot be contacted (no email address available). In both
cases, the exclusion of these portions of Veterans from the survey contributes to the
measurement error. The extent that the final survey estimates are skewed depends on the nature
of the data collection processes within an individual line of business and the potential alignment
between veteran sentiment and their likelihood to respond.
Survey practitioners recommend the use of sample weighting to improve inference on the
population so that the final respondent sample more closely resembles the true population. It is
likely that differential response rates may be observed across different age and gender groups.
Weighting can help adjust for the demographic representation by assigning larger weights to
underrepresented group and smaller weights to overrepresented group. Stratification can also be
used to adjust for nonresponse by oversampling the subgroups with lower response rates. In both
ways of adjustments, weighting may result in substantial correction in the final survey estimates
when compared to direct estimates in the presence of non-negligible sample error.
Weights are updated live within the VSignals reporting platform 2. Proportions are set
based on the monthly distribution of the previous month. 3
If we let wij denote the sample weight for the ith person in group j (j=1, 2, and 3), then the
CW formula is:

𝑤𝑤𝑖𝑖𝑖𝑖 =

% 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 𝑖𝑖𝑖𝑖 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑖𝑖𝑖𝑖 𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔 𝑗𝑗
# 𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉𝑉 𝑖𝑖𝑖𝑖 𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔𝑔 𝑗𝑗 𝑖𝑖𝑖𝑖 𝑡𝑡ℎ𝑒𝑒 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠

As part of the weighting validation process, the weights of persons in an age and gender
group are summed and verified that they match the universe estimates (i.e., population
proportion). Additionally, we calculate the unequal weighting effect, or UWE (see Kish, 1992;
Liu et al., 2002). This statistic is an indication of the amount of variation that may be expected
due to the inclusion of weighting. The unequal weighting effect estimates the percent increase in
the variance of the final estimate due to the presence of weights and is calculated as:

where
•
•
•

𝑠𝑠
2
𝑈𝑈𝑈𝑈𝑈𝑈 = 1 + 𝑐𝑐𝑐𝑐𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤ℎ𝑡𝑡𝑡𝑡
= ( )2
𝑤𝑤
�
cv = coefficient of variation for all weights 𝑤𝑤𝑖𝑖𝑖𝑖 .
s = sample standard deviation of weights.
1
𝒘𝒘
� = sample mean of weights, 𝑤𝑤
� = 𝑛𝑛 ∑𝑖𝑖𝑖𝑖 𝑤𝑤 ij.

Realtime weighting may cause some distortions at the beginning of each cycle due to empty cells or random
variance in small sample distributions.
3
Using previous months data is a design option for handling the problem of setting targets prior to fielding each
month. An alternative design is to set targets off annualized estimates to create more stability month to month. If the
population is known to fluctuate from month to month, past month population estimates may not be the optimal
solution.
2

15

G. Quarantine Rules
VEO seeks to limit contact with Veterans as much as possible, and only as needed to
achieve measurement goals. These rules are enacted to prevent excessive recruitment attempts
upon Veterans. VEO also monitors Veteran participation within other surveys, to ensure
Veterans do not experience survey fatigue. All VEO surveys offer options for respondents to opt
out, and ensure they are no longer contacted for a specific survey.
Table 5. Proposed Quarantine Protocol
Quarantine Rule
Past waves
Active Waves
Anonymous
Opt Outs

Description
Number of days between completing online survey any
VEO survey and receiving another invitation.
Number of days between receiving an invitation to a VEO
survey and receiving another invitation.
Callers explicitly wishing to remain anonymous will not be
contacted.
Persons indicating their wish to opt out of either phone or
online survey will no longer be contacted.

Elapsed Time
30 Days
14 Days
N/A
N/A

Part III – Assumptions and Limitations
A) Population Estimation Error
The population estimates for this survey include some uncertainty due to 1) fluctuation in the call
volumes due to the current pandemic (Covid 19); 2) an increase over time in the use of telemedicine over
time; and 3) potential policy shift (e.g. shift to more reliance on contractor or LIP). Estimates tried to
account for these factors. None-the-less, a large amount of uncertainty exists. To address this risk, we
recommend evaluating the sample plan over time to determine how well the estimates hold up.

B) Coverage Bias due to Email-Only Data Collection
Since the Community Care Survey is email-only, there is a segment of the population of
Community Care callers that cannot be reached by the survey. This will correspond to persons
that lack access to the internet, and those who do not have an email address, or elect to not share
their email address with the VA. Such beneficiaries may have different levels of general
satisfaction with their service they received.
Appendix 1. List of Data Extraction Variables
Appendix 2. Survey Questions
Survey 1: Choosing VA Community Care

16

My VA primary care provider and/or VA coordinating team explained benefits
offered through VA community care in a way I could understand.
After I chose VA community care, I knew what to expect.
I clearly understood why I was referred out of my local VA for care.
My VA provider and/or VA coordinating team discussed the urgency of my health
care needs and means of transportation before referring me for VA community care.
I am confident VA will coordinate my care with my community provider.
I am satisfied with my VA provider and/or VA coordinating team’s decision to use
VA community care.
I trust VA community care to coordinate the best medical treatment for my health
care needs.
Survey 2: Scheduling a VA Community Care Appointment
The considered all of my needs when scheduling my VA
community care appointment.
Scheduling a VA community care appointment was easy.
I was contacted to schedule my community care appointment shortly after I chose
to use VA community care.
I was treated fairly and with respect when I was contacted to schedule my
appointment.
I understood whom to contact to cancel or change my appointment with .
The clearly explained what I should expect regarding my
scheduled VA community care appointment.
I trust VA community care to coordinate my care with my community provider.
Survey 3: Attending a VA Community Care Appointment
I was satisfied with the amount of time I waited between scheduling and being seen
by .
I knew what to expect during my appointment with .
I had no difficulties traveling to my last VA community care appointment.
VA community providers are well-versed in how to properly treat and care for
Veterans.
I am confident that my VA health information was provided to
to ensure I received quality care.
I am satisfied with the care I received from .
I trust VA community care to address my medical needs.
Survey 4: Filling a Prescription Through the VA Community Care
VA provided clear information on how to fill a prescription written by an authorized
community provider.

17

I understand whom to contact if I have questions or concerns regarding filling a
prescription written by an authorized community provider.
I received my prescriptions from my last VA community care appointment in a
timely manner.
It was easy to fill a prescription written by my last VA community provider.
VA considered my medical needs when obtaining medication and/or medical
equipment prescribed by a VA community provider.
VA pharmacists were helpful and informative when answering questions about
prescriptions written by an authorized community provider.
I trust VA community care to coordinate the best medical treatment for my health
care needs.

18

Survey 5: Understanding Financial Responsibility with VA Community Care
The financial documents I received from VA clearly explained my financial
responsibilities related to my visit with an authorized community provider.
I was able to easily find a VA representative who could help address a billing
question or concern regarding a community care appointment.
The VA representative addressed my financial concerns regarding my visit with with respect and dignity.
VA provided clear instructions on how to address community care financial
concerns.
I am satisfied with VA's ability to address billing concerns after being referred out to
an authorized community provider.
Using VA community care has positively impacted my health and quality of life.
I trust VA to honor payments to community providers for authorized care.
Survey 6: Calling VA About Community Care Billing Questions
I understood whom to contact if I had billing concerns or credit reporting issues
related to VA community care.
It was easy to contact the VA community care contact center to address a question
or concern about an outstanding bill from a VA community care visit.
The VA addressed my billing concerns and adverse credit reporting issues with
respect and dignity.
The VA quickly addressed my billing concerns and adverse credit reporting issues
from my use of VA community care.
I am satisfied with VA's response to my VA community care billing concerns.
Using VA community care has positively impacted my health and quality of life.
I trust VA to resolve any billing issues that might result from my use of an
authorized community provider.
Appendix 3. References
Choi, N.G. & Dinitto, D.M. (2013). Internet Use Among Older Adults: Association with Health
Needs, Psychological Capital, and Social Capital. Journal of Medical Internet Research,
15(5), e97
Kish, L. (1992). Weighting for unequal P. Journal of Official Statistics, 8(2), 183-200.
Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.
Liu, J., Iannacchione, V., & Byron, M. (2002). Decomposing design effects for stratified
sampling. Proceedings of the American Statistical Association’s Section on Survey
Research Methods.

19