Sample Plan

Service Level Measurements – ECCC
Community Care Survey
Sampling Methodology Report
Prepared by
Veteran Experience Office
Version 1
May 2020

1

Contents
Executive Summary .................................................................................................................................... 3
Part I – Introduction ................................................................................................................................... 4
A. Background .......................................................................................................................................... 4
B. Basic Definitions .................................................................................................................................. 5
C. Application to Veterans Affairs............................................................................................................ 5
Part II – Methodology ................................................................................................................................ 5
A. Target Population and Frame .............................................................................................................. 5
B.

Sample Size Determination ............................................................................................................... 6

C.

Data Collection Methods .................................................................................................................. 8

D.

Reporting........................................................................................................................................... 8

E.

Quality Control ................................................................................................................................. 8

F.

Sample Weighting, Coverage Bias, and Non-Response Bias ........................................................... 9

G.

Quarantine Rules ............................................................................................................................. 10

Part III – Assumptions and Limitations ................................................................................................. 10
A. Respondent Satisfaction Bias ............................................................................................................. 10
B. Coverage Bias..................................................................................................................................... 10
Appendix 1. List of Data Extraction Variables .................................................................................. 11
Appendix 2. Survey Questions ............................................................................................................. 12
Appendix 3. References ........................................................................................................................ 14

2

Executive Summary
The Community Care ECCC Survey is designed to measure customer experience after contacting
the Enterprise Clinical Call Center (ECCC) for Community Care.
Veterans experience data is collected by using an online transactional survey disseminated via an
invitation email sent to randomly selected beneficiary. The data collection occurs once per week with
invitation being sent out within 8 days of calling the Community Care ECCC. The questionnaire is brief
and contains general Likert-scale (a scale of 1-5 from Strongly Disagree to Strongly Agree) questions to
assess customer satisfaction as well as questions assessing the knowledge, speed, and manner of the
interaction. After the survey has been distributed, recipients have two weeks to complete the survey and
will receive a reminder email after one week.
The overall sample size for the ECCC Community Care Survey population is determined so that
the reliability of monthly survey estimate is at 3% Margin of Error at the 95% Confidence Level. The
survey will be sent to a representative sample of Veterans and beneficiaries. Once data collection is
completed, the participant responses in the online survey will be weighted so that the samples more
closely represent the actual call volume of each call center
This report describes the methodology used to conduct the Community Care ECCC Survey.
Information about quality assurance protocols, as well as limitations of the survey methodology, is also
included in this report.

3

Part I – Introduction
A. Background
The Enterprise Measurement and Design team (EMD) is part of the Insights and Analytics
(I&A) division within the Veterans Experience Office (VEO). The EMD team is tasked with conducting
transactional surveys of the Veteran population to measure their satisfaction with the Department of
Veterans Affairs (VA) numerous benefit services. Thus, their mission is to empower Veterans by rapidly
and discreetly collecting feedback on their interactions with such VA entities as NCA, VHA, and VBA.
VEO surveys generally entail probability samples which only contact minimal numbers of Veterans
necessary to obtain reliable estimates. This information is subsequently used by internal stakeholders to
monitor, evaluate, and improve beneficiary processes. Veterans are always able to decline participation
and have the ability to opt out of future invitations. A quarantine protocol is maintained to limit the
number of times a Veteran may be contacted, in order to prevent survey fatigue, across all VEO surveys.
The VHA runs several Clinical Call Centers to help Veterans and other stakeholders navigate the benefits
provided and manage the care of the beneficiary. The Community Care ECCC provides these services for
beneficiaries and stakeholders for Community Care program that covers the use of providers outside of
the VHA’s network of facilities. While a majority of the calls to the Community Care ECCC are from
providers managing payment for services and authorizing care delivery; the primary focus of this research
will be the experience of the Veteran who call for a variety of reasons including checking claim status,
referrals, scheduling, questions about benefits, preauthorization, etc.1.
In order to continue to provide quality services to Veterans, VEO has been commissioned to measure the
satisfaction of Veterans with the Community Care ECCC. To complete this goal, VEO proposed to
conduct a brief transactional survey on randomly selected Veterans who had called the Community Care
ECCC. The survey consists of eight questions revolving a human-centered design, focusing on Veterans’
experience with regard to their recent encounter pertaining to the A-11 Customer Experience Domains of
Quality, Satisfaction, Confidence/Trust, Ease/Simplicity, Efficiency/Speed, and Employee Helpfulness.
These Likert-scale (a scale of 1-5) questions are designed through extensive Veteran input and
recommendations from subject matter experts in the VA. Several months after the deployment of the
survey, Community Care plans to include a free text response question.
Veterans are randomly selected to participate in the survey via an invitation email. A link is
enclosed so the survey may be completed using an online interface, with customized participant
information. The data is collected on a weekly basis and the survey is reported on a monthly basis. The
purpose of this document is to outline the planned sample design and provide a description of the data
collection and sample sizes necessary for proper reporting.

1

The focus on Veterans for this study is due to the lack of email addresses available for non-veterans. The survey
will send invitations out to non-Veterans as the email addresses become available. If the Community Care ECCC
team can improve email address collection substantially, this plan will be updated to cover additional stakeholders.

4

B. Basic Definitions
Coverage
Measurement Error
Non-Response
Transaction

Response Rate
Sample
Sampling Error
Sampling Frame
Reliability

The percentage of the population of interest that is included in
the sampling frame.
The difference between the response coded and the true value of the
characteristic being studied for a respondent.
Failure of some respondents in the sample to provide responses in
the survey.
A transaction refers to the specific time a Veteran interacts with the
VA that impacts the Veteran’s journey and their perception of VA’s
effectiveness in caring for Veterans.
The ratio of participating persons to the number of contacted
persons. This is one of the basic indicators of survey quality.
In statistics, a data sample is a set of data collected and/or selected
from a statistical population by a defined procedure.
Error due to taking a particular sample instead of measuring every
unit in the population.
A list of units in the population from which a sample may be
selected.
The consistency or dependability of a measure. Also referred to as
standard error.

C. Application to Veterans Affairs
Customer experience and satisfaction are usually measured at three levels to: 1) provide
enterprises the ability to track, monitor, and incentivize service quality; 2) provide service level
monitoring and insights; and 3) give direct point-of-service feedback. This measurement may bring
insights and value to all stakeholders at VA. Front-line VA leaders can resolve individual feedback from
Veterans and take steps to improve the customer experience; meanwhile VA executives can receive realtime updates on systematic trends that allow them to make changes.
1) To collect continuous customer experience data
2) To help field staff and the national office identify areas of improvement.
3) To understand emerging drivers and detractors of customer experience.

Part II – Methodology
A. Target Population, Frame, and Stratification
The target population of the Community Care ECCC Survey is defined as any Veterans who have
contacted the Community Care ECCC in the past weeks are eligible for participation.
The sample frame is prepared by extracting population information directly from the Customer
Relationship Management database. These extracts are also used to obtain universe figures for the sample
weighting process. The Veteran is the primary sampling unit and is randomly selected from the
population according to a stratified design with allocation proportional to the true population. The strata
will be implicit strata rather than precise targets and will include age, gender, and interaction purpose.
Non-Veterans may be surveyed as a census, if desired, until the point that a sample survey is possible.

5

B. Sample Size Determination
To achieve a certain level of reliability, the sample size for a given level of reliability is
calculated below (Lohr, 1999):
For a population that is large, the equation below is used to yield a representative sample for
proportions:
𝑛0 =

2
𝑍𝛼/2
𝑝𝑞

𝑒2

where
-

-

𝒁𝜶/𝟐 = is the critical Z score which is 1.96 under the normal distribution when using a 95%
confidence level (α = 0.05).
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
o Note that pq attains its maximum when value p=0.5 or 50%. This is what is typically
reported in surveys where multiple measures are of interest. When examining measures
closer to 100% or 0% less sample is needed to achieve the same margin of error.
e = the desired level of precision or margin of error. For example, for the Community Care ECCC
Survey the targeted margin of error is e = 0.03, or +/-3%.

For a population that is relatively small, the finite population correction is used to yield a
representative sample for proportions:
𝑛=

𝑛0
𝑛
1+ 0
𝑁

Where
-

𝒏𝟎 = Representative sample for proportions when the population is large.
N = Population size.

The margin of error surrounding the baseline proportion is calculated as:
𝑀𝑎𝑟𝑔𝑖𝑛 𝑜𝑓 𝐸𝑟𝑟𝑜𝑟 = 𝑧𝛼/2 √

𝑁 − 𝑛 𝑝(1 − 𝑝)
√
𝑁−1
𝑛

Where
𝒁𝜶/𝟐 = 1.96, which is the critical Z score value under the normal distribution when using a 95%
confidence level (α = 0.05).
N = Population size.
n = Representative sample.
p = the estimated proportion of an attribute that is present in the population, with q=1-p.
Estimates from the population files drawn for the first 90 days of 2020 indicate that in the average
month 25,000 calls are made to the Community Care ECCC by Veterans. Table 1A indicates the
population figures based on numbers from that period, as well as estimated population with email
addresses on file and the proportion that is likely to be usable after removing duplicates and quarantine
rules across VEO surveys.

6

Table 1A. Target Population Figures, Sample Size, and Email Contacts

Estimated
Monthly
Callers

Community Care
ECCC

25,156

Estimated
Monthly
Callers w/
Email
Addresses

15,072

Estimated
Monthly Callers
w/ Email
Addresses
Available After
Exclusion Rules
and Deduplication

Target
MOE

12,057

3.00%

Confidence

Minimum
Monthly
Responses
Needed

95%

Minimum
Monthly
Sample
Needed

Response
Rates

1,024

25%

4,096

Table 1B shows the estimated sample frame and minimum target sample size on a weekly basis.
Minimum targets are rounded upward to assure the prescribed accuracy is achieved.
Table 1B shows the weekly sample availability and sample needs.

Community Care ECCC

Estimated Weekly
Callers w/ Email
Addresses
Available After
Exclusion Rules
and Deduplication
2,774

Minimum weekly
sample needed

236

Rounded weekly
sample targets

1,050

Sampling Rate

30.3%

The sample will be drawn using a systematic sampling methodology. This statistical valid approach
allows the team to balance the sample across several variables such as age, gender, and interaction
purpose. This balancing variable are often referred to as implicit strata. In the coming wave, the VEO
team will begin to leverage this capability because, though the effect on margin of error is difficult to
measure, this methodology has been proven to improve the accuracy of estimates, stabilize weights, and
reduce the variability that make trends difficult to interpret.
Email addresses will be acquired by matching Veteran ID numbers to the VBA’s Enterprise Data
Warehouse (EDW) and the VHA’s Corporate Data Warehouse (CDW). The CDW will be prioritized if
the two sources produce different and valid email addresses. Each email address encountered is validated
in several ways:
•
•
•
•
•

Validation that the email address has a valid structure
Comparison with a database of bad domains
Correction of common domain misspellings
Comparison of a database of bad emails including
o Opt outs
o Email held by multiple veterans
Comparison to a database of valid TDLs (e.g. “.com”, “.edu”)

7

C. Data Collection Methods
Invitations will be sent out each week to assure that initial invites are sent within eight days of
their call to the Community Care ECCC. Caller information will be regularly extracted from VHA
database resource: Community Care’s Customer Relationship Management database. The extraction
process will be executed and validated by the Office of Performance Improvement and Assessment
(PA&I). with the population extracts sent to VEO twice a week. Invitation will be sent on Mondays.
Invitees that have not completed the survey will receive a reminder after one week. The survey will
remain open for a total of two weeks. Survey responses are immediately available within VSignals as
soon as feedback is submitted.

D. Reporting
Researchers will be able to use the Veteran Signals (VSignals) system for interactive reporting
and data visualization. VA employees with a PIV card may access the system at
https://va.voice.medallia.com/sso/va/. The scores may be viewed by Age Group, Gender, and
Race/Ethnicity in various charts for different perspective. They are also depicted within time series plots
to investigate trends. Finally, filter options are available to assess scores at varying time periods and
within the context of other collected variable information.
Recruitment is continuous but the results should be combined into a monthly data file for more
precise estimates, at the call center level. Short interval estimates are less reliable for small domains, (i.e.,
VAMC-level) and should only be considered for aggregated populations. Monthly estimates will have
larger sample sizes, and therefore higher reliability. Estimates over longer periods are the most precise
but will take the greatest amount of time to obtain and are less dynamic in that trends and short-term
fluctuation in service delivery may be missed. Users examining subpopulation should be particularly
diligent in assuring that insights stem from analysis with sufficient sample in the subpopulations being
examined or compared.

E. Quality Control
To ensure the prevention of errors and inconsistencies in the data and the analysis, quality control
procedures will be instituted in several steps of the survey process. Records will undergo a cleaning
during the population file creation. The quality control steps are as follows.
1. Records will be reviewed for missing sampling and weighting variable data. When records with
missing data are discovered, they will be either excluded from the population file or put into
separate strata upon discussion with subject matter experts.
2. Any duplicate records will be removed from the population file to both maintain the probabilities
of selection and prevent the double sampling of the same Veteran.
3. Invalid emails will be removed.
The survey sample loading and administration processes will have quality control measures built
into them.
1. The survey load process will be rigorously tested prior to the induction of the survey to ensure
that sampled customers is not inadvertently dropped or sent multiple emails.
2. The email delivery process is monitored to ensure that bounce-back records will not hold up the
email delivery process.
8

The weighting and data management quality control checks are as follows:
1. The sum of the weighted respondents will be compared to the overall population count to confirm
that the records are being properly weighted. When the sum does not match the population count,
weighting classes will be collapsed to correct this issue.
2. The unequal weighting effect will be used to identify potential issues in the weighting process.
Large unequal weighting effects indicate a problem with the weighting classes, such as a record
receiving a large weight to compensate for nonresponse or coverage bias.

F. Sample Weighting, Coverage Bias, and Non-Response Bias
Weighting is commonly applied in surveys to adjust for nonresponse bias and/or coverage bias.
Nonresponse is defined as failure of selected persons in the sample to provide responses. This is observed
virtually in all surveys, in that some groups are more or less prone to complete the survey. The
nonresponse issue may cause some groups to be over- or under-represented. Coverage bias is another
common survey problem in which certain groups of interest in the population are not included in the
sampling frame. The reason that these Veterans cannot participate is because they cannot be contacted (no
email address available). In both cases, the exclusion of these portions of Veterans from the survey
contributes to the measurement error. The extent that the final survey estimates are skewed depends on
the nature of the data collection processes within an individual line of business and the potential
alignment between veteran sentiment and their likelihood to respond.
Survey practitioners recommend the use of sample weighting to improve inference on the
population so that the final respondent sample more closely resembles the true population. It is likely that
differential response rates may be observed across different age and gender groups. Weighting can help
adjust for the demographic representation by assigning larger weights to underrepresented group and
smaller weights to overrepresented group. Stratification can also be used to adjust for nonresponse by
oversampling the subgroups with lower response rates. In both ways of adjustments, weighting may result
in substantial correction in the final survey estimates when compared to direct estimates in the presence of
non-negligible sample error.
Weights are updated live within the VSignals reporting platform2. Proportions are set based on
the monthly distribution of the previous month.3
If we let wij denote the sample weight for the ith person in group j (j=1, 2, and 3), then the CW
formula is:

𝑤𝑖𝑗 =

% 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑝𝑜𝑝𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 𝑗
# 𝑉𝑒𝑡𝑒𝑟𝑎𝑛𝑠 𝑖𝑛 𝑔𝑟𝑜𝑢𝑝 𝑗 𝑖𝑛 𝑡ℎ𝑒 𝑠𝑎𝑚𝑝𝑙𝑒

As part of the weighting validation process, the weights of persons in an age and gender group are
summed and verified that they match the universe estimates (i.e., population proportion). Additionally, we
calculate the unequal weighting effect, or UWE (see Kish, 1992; Liu et al., 2002). This statistic is an
indication of the amount of variation that may be expected due to the inclusion of weighting. The unequal

2

Realtime weighting may cause some distortions at the beginning of each cycle due to empty cells or random
variance in small sample distributions.
3
Using previous months data is a design option for handling the problem of setting targets prior to fielding each
month. An alternative design is to set targets off annualized estimates to create more stability month to month. If the
population is known to fluctuate from month to month, past month population estimates may not be the optimal
solution.

9

weighting effect estimates the percent increase in the variance of the final estimate due to the presence of
weights and is calculated as:
𝑠
2
𝑈𝑊𝐸 = 1 + 𝑐𝑣𝑤𝑒𝑖𝑔ℎ𝑡𝑠
= ( )2
𝑤
̅
where
-

cv = coefficient of variation for all weights 𝑤𝑖𝑗 .
s = sample standard deviation of weights.
1
̅ = sample mean of weights, 𝑤
𝒘
̅ = ∑𝑖𝑗 𝑤 ij.
𝑛

G. Quarantine Rules
VEO seeks to limit contact with Veterans as much as possible, and only as needed to achieve
measurement goals. These rules are enacted to prevent excessive recruitment attempts upon Veterans.
VEO also monitors Veteran participation within other surveys, to ensure Veterans do not experience
survey fatigue. All VEO surveys offer options for respondents to opt out, and ensure they are no longer
contacted for a specific survey.
Table 5. Proposed Quarantine Protocol
Quarantine Rule
Past waves
Active Waves
Anonymous
Opt Outs

Description
Number of days between completing online survey any VEO
survey and receiving another invitation.
Number of days between receiving an invitation to a VEO survey
and receiving another invitation.
Callers explicitly wishing to remain anonymous will not be
contacted.
Persons indicating their wish to opt out of either phone or online
survey will no longer be contacted.

Elapsed Time
30 Days
14 Days
N/A
N/A

Part III – Assumptions and Limitations
A Veterans Only
At the onset of the Community Care ECCC Survey, email addresses are only available for
Veterans and not other stakeholders. Since Veteran attitudes may differ from those of non-Veterans, the
exclusion of non-Veterans from the survey may contribute bias to the survey estimates. VEO will
continue to work with VHA to acquire contact information for all callers to benefit services, and this
information will be used in future releases to address the entire target population.

B Coverage Bias due to Email-Only Data Collection
Since the Community Care ECCC Survey is email-only, there is a segment of the population
Community Care ECCC callers that cannot be reached by the survey. This will correspond to persons that
10

lack access to the internet, and those who do not have an email address, or elect to not share their email
address with the VA. Such beneficiaries may have different levels of general satisfaction with their
service they received.

C Call Characteristics: Length of Call & Number of Calls
There is a possibility that length of call to the Community Care ECCC Survey may have be a
predictor of customer satisfaction. Longer calls may produce higher or lower levels of satisfaction,
perhaps either due to long waiting times or because of the increased levels of assistance provided to the
call by the call center representative. The data extraction process at the time of this version of the
sampling documentation does not include call length. VEO will work with the Performance, Assessment,
and Improve office within VHA to obtain this possibly relevant information. At such time, consideration
will be taken into incorporating call length into the sampling and weighting procedures.

Appendix 1. List of Data Extraction Variables
Survey Variables
Survey Person ID
Agent ID
Date Time Call
Call Center
Phone Number
Coach
Full Name
Service Request Action
Caller Relation to Veteran
Has eBenefit Account
Credit Level
Call Type
Sub Type
NCC Start Date
Age
Gender
Period of Service
Veterans Email
Veteran ID # (MVI)

11

Appendix 2. Survey Questions

12

13

Appendix 3. References
Choi, N.G. & Dinitto, D.M. (2013). Internet Use Among Older Adults: Association with Health Needs,
Psychological Capital, and Social Capital. Journal of Medical Internet Research, 15(5), e97
Kish, L. (1992). Weighting for unequal P. Journal of Official Statistics, 8(2), 183-200.
Lohr, S. (1999). Sampling: Design and Analysis (Ed.). Boston, MA: Cengage Learning.
Liu, J., Iannacchione, V., & Byron, M. (2002). Decomposing design effects for stratified
sampling. Proceedings of the American Statistical Association’s Section on Survey Research
Methods.

14