WH Hotline Survey Methodology Design

WHITE HOUSE VETERAN HOTLINE SURVEY
METHODOLOGY BRIEF: SAMPLE DESIGN
Introduction
The White House Hotline (WHH) is administered by the Dept. of Veterans Affairs (VA). It was
launched in June 2017 as part of the President’s agenda for reform at the VA. The hotline staff is
comprised mostly of veterans and is available 24 hours a day. It is a direct outlet for Veterans to speak
with live agents about their needs and concerns. The call center staff may function in a variety of ways,
including responding to inquiries; providing directory assistance; and discussing Veteran benefits and
health services. Certain calls are categorized as complaints, and steps are undertaken to address these
specific cases within days.
The Veteran Experience Office has been commissioned to measure the satisfaction of WHH
callers regarding their interaction with call center staff and the ultimate resolution of their complaint (if
necessary). VEO proposed to conduct a brief transactional survey on persons who recently used the
hotline. The survey may be completed online or via an automated telephone design. It will consist of a
handful of questions revolving around a human-centered design, focusing on such elements as trust;
emotion; effectiveness; and ease with the service. The purpose of this document is to outline the
sample design and provide a description of the data collection and reporting.
Table 1. Measurement Goals and Survey Mode
Survey Stratum

Preferred Mode of
Data Collection

Recruitment
Method

Time After
Transaction

Recruitment

Initial Call to WH
Hotline
Complaint
Resolution

Automated Telephone
Questionnaire
Online Survey

Post Call or
Call-Back
Email
Recruitment

Same Day as Call or
within 24 Hours
Within 1 Week
after Resolution

3 Calls over 24
Hours
2 Emails over
2 Weeks

1. Initial Call to White House Hotline
The WHH management would like to measure each caller’s impression of the WH
hotline directly after their conversation with the WHH representative. This will serve as an
evaluation of the call reception format and protocol, the depth of the VA knowledge base
accessible to staff members, and the personal affability of the call representative while fielding
the complaint or other assistance.

It is perceived that the accuracy of veteran feedback regarding the initial telephone
conversation is improved when collected immediately following the transaction. As such, the
ideal data collection mode would be through a post-phone call survey. Callers would be queried
whether they’d like to participate in a survey at the end of their call and would immediately be
transferred to an automated set of questions. Alternatively, VEO could call these veterans with
the question selection. Call-backs would occur within 24 hours, and would not exceed three
contact attempts.
2. Complaint Resolution
The other major subpopulation of interest centers on persons that filed a complaint.
Over half of all calls to the WHH result in the opening of a complaint case. Each new case
undergoes a triage and is facilitated to the appropriate division or office within the VA for
further consideration. Every complaint is quickly assessed and appropriate steps are
undertaken, no matter if that corresponds to an explanation and/or apology; a modification to
VA programs or policy; or some other specific action. Once a remedy to the original complaint is
enacted, the resolution is communicated to the veteran. Reporting the satisfaction with the final
outcome from their interaction with the White House Hotline constitutes the other major goal
of the survey.
VEO will collect information on complaint resolution on a weekly basis. In this instance,
veterans are assessing an action or decision taken by an entity within the VA. The cognitive
recall for such a decision is thought to be stronger than the recollection of the characterization
of the demeanor and helpfulness of an individual call center representative. Unlike that initial
dialogue, there is perceived to be less necessity in garnering prompt reaction from the veteran.
On the contrary, it may be constructive to allow additional time for veterans to reflect on the
actions taken by the VA due to their contact with the WHH. This is especially true for
transactional surveys stemming from preexisting negative circumstances. For these reasons, we
deem any collection bias emanating from a portion veterans being contacted within a week of
being notified of their resolution negligible.
Persons receiving their complaint resolution will receive an email invitation to join the
survey within a week of receipt. They will have two weeks to complete the online survey, with a
reminder email after 7 days if it has not yet been completed. Persons are asked to provide their
email address by call center staff during the initial call when they provide a description of their
complaint. However, some persons choose not to share their email address information while
other veterans may not have an email address. VEO will monitor the email rate within case
population and will consider automated telephone surveys for those veterans that do not
provide an email address.
Target Population and Sample Size Determination
The target population of the WHH survey is all persons who ever placed a call. Secondary
contact is made for persons who received a complaint resolution. The time between the initial call and
the actual resolution will fluctuate, but complaints are usually closed within days. It should be noted
that every veteran who has a complaint resolution will also have had to have called the hotline at some
earlier date. As such, these are not mutually exclusive groups and do not fit within the proper definition

of survey strata. Careful consideration of this matter will be taken into account so that undue burden is
not placed on possible respondents.

Nevertheless, we would like to make very precise overall measurements of certain population
parameters (see question list). The reliability standard within the survey industry corresponds to a 95%
confidence level and a 3% margin of error (Lohr 1999). Chart 1, listed above, depicts the number of calls
received by the hotline staff for a three month period between Nov. 2017 and Jan. 2018. The average
monthly total calls and average number of complaints is listed in Table 2. We have made the
conservative assumption that cases are opened and closed at the same rate during the month (a portion
of cases may remain unclosed and become overdue). The standard sample size determination formula
indicates the number of responses from veterans required to attain the stated level of precision.
Table 2. Target Population Figures
Survey Stratum

Target Population

Approximate
Monthly
Population

Initial Call to WH
Hotline

Veterans calling
the WHH in the
past day
Veterans w/ closed
cases in the past
week

Complaint
Resolution

Precision
Sample:
95% CL,
4% MOE
526

Expected Yield at
100% Sampling
Rate

4,200

Precision
Sample:
95% CL,
3% MOE
850

2,500

750

500

500

840

It is good survey practice to only contact as many persons as necessary to obtain a reliable
estimate of the population parameters. This minimizes the burden placed on the population. For a
population of 100,000 persons, it is only necessary to collect about 1,100 responses to get a reliable
overall estimate. However, when dealing with smaller populations, the number of required responses
does not decrease proportionally. For a monthly population of 2,500, the sample size needed for reliable
overall estimates is still a sizable 750.

Based on other surveys implemented by VEO on the veteran population – both via phone and
email – the expected survey response rate is placed at 20%. Under this assumption, sending invitations
to every possible veteran would yield a reliable sample (95% CI, 3% MOE) for ALL CALLERS at 850, but
only a reliable sample at 4% MOE for RESOLUTIONS. Therefore, it is necessary to sample all available
veterans for this effort to produce quality results for both populations. This obviates the need for any
stratification or formal probability sampling.
Reporting and Quality Control
The sample sizes reported above adhere to monthly reporting. However, it is inadvisable to only
recruit once a month, as elapsed time may severely reduce the cognitive recall of participants.
Therefore, data collection is enacted on a daily or weekly basis, depending on the survey type.
All responses from a particular calendar month will be pooled together and reported as a single
cohesive unit. The default reporting period in the user interface will pertain to monthly estimates, as
this will have the most reliable estimates, with more frequent data points than quarterly reporting. At
the monthly level, subpopulation estimates will still be viable (e.g. satisfaction with regard to individual
agents, or responses based on time of call). For these reasons, monthly reporting is the recommended
time unit of analysis.
Many investigators are also interested in estimates at the weekly and even daily level. This
permits them to observe changes of satisfaction over shorter time intervals. These estimates are more
volatile, however, and are associated with larger standard errors and margins of errors. Tables 3A and
3B depict the expected sample sizes for different reporting periods along with the associated reliability.
Data users will be able to observe the survey responses at any level reporting level (Daily, Weekly,
Monthly or Quarterly), but results must be interpreted with care.
Table 3A. Reporting Sizes for Initial Calls to WH Hotline
Time Period

Quarterly
Monthly
Weekly
Daily

Approximate
Population Size
12,600
4,200
1,050
140

Expected Yield
at 100%
Sampling Rate
2,520
840
210
28

Observed
Margin of
Error
2%
3%
6%
17%

Expected Yield
at 100%
Sampling Rate
1,500
500
125
17

Observed
Margin of
Error
2%
4%
8%
21%

Table 3B. Reporting Sizes for Complaint Resolution
Time Period

Quarterly
Monthly
Weekly
Daily

Approximate
Population Size
7,500
2,500
625
83

Coverage and Non-Response Bias
Many survey practitioners recommend the use of sampling weighting to improve inference on
the population. Under this process, the respondent sample is made to more closely resemble the true
population. The process requires accurate information on the Age and Gender of callers. Such
demographic information is only available for veterans, and there are a substantial number of veterans
with unknown information. Therefore, any future weighting methodology will need to accommodate
missing information from non-veteran callers and acquire more complete demographic information for
all veterans. Upon receipt of this information – either through administration data or a refinement of
the interview process – VEO will conduct a weighting analysis to determine whether there are possible
inferential gains. For the time being, all survey estimates will remain unweighted.
Quarantine Rules
The Veterans Experience Office seeks to limit contact with Veterans as much as possible and
only as necessary to achieve measurement goals. After some consideration, we are proposing to elicit
feedback from Veterans after their initial call and again days or weeks later only for complaint calls.
These persons would be contacted after they have had a complaint resolved and the case closed. Due to
the limited sample sizes and expected response rates, the double sampling is essential to fulfill the
survey objectives. Only half of the callers will be invited to both surveys, and only approximately 4-5%
will actually complete both.
The veterans may not be aware of two phases as they will be offered through different media.
The dual nature of these surveys will serve to reduce the burden placed upon them. The first survey will
consist of a phone survey directly after the initial call, while the second will be an online survey using
email recruitment. The email survey will be sent some time later and may be completed at their leisure
over a 2-week period. Both surveys are brief in nature and may be completed within minutes.
Moreover, the rules proposed below will prevent excessive recruitment attempts upon users of the
hotline service.
VEO also monitors veteran participation within other surveys to ensure veterans do not
experience survey fatigue. Finally, all VEO surveys offer options for respondents to opt out and ensure
they are no longer contacted for a specific survey.
Table 4. Proposed Quarantine Protocol
Quarantine Rule

Description

Elapsed Time

Phone to Email
(Complaint Calls Only)

Number of days between receiving/completing phone survey,
prior to receiving email invitation to the online survey for the
same complaint.
Number of days between receiving/completing phone survey
regarding, and receiving another phone survey related after
calling the hotline for another issue.
Number of days between completing online survey, and
receiving another online survey related to another complaint.
Veterans engaged that have recently completed other VEO
surveys will not be selected for 30 days.

1 Week

Phone to Phone

Email to Email
(Multiple Complaints)
Other Surveys

3 Months or 90
Days
3 Months or 90
Days
1 Month or 30
Days

Anonymous
Opt Outs

Callers explicitly wishing to remain anonymous will not be
contacted.
Persons indicating their wish to opt out of either phone or online
survey will no longer be contacted.

N/A
N/A