Survey Sample Plan

Service Level Measurements: Outpatient

Sampling Methodology Report

June 2018

Contents

Part I: Introduction 3

A. Background 3

B. Moments that Matter Survey Series 4

C. Basic Definitions 5

Part II: Methodology 5

A. Survey Instrument 7

Part III: Sample Design 10

A. Target Population and Frame 10

B. Quality Control 13

C. Sampling Design 13

D. Sample Size 16

E. Survey Invitation File 19

Part IV: Survey Administration 19

Part V: Analysis and Dashboarding 19

Part VI: Limitations and Mitigations 20

Sequencing and Conditionality 20

Compound Transaction 20

Appendix A. Services by VA Administration 21

Appendix B. Eligible Stop Code List. 22

Part I: Introduction

A. Background

When veterans interact with VA, they expect the highest quality of care and service, feeling that they are honored, respected, and their needs are met. In 2016, the Enterprise Management and Performance Improvement (EM&PI) team at the VEO created a model for measurement to hold VA accountable to veterans to fulfill those goals.

As a program, VA’s service level measurements are based on the most impactful interactions veterans have with VA–the moments that matter. Veterans were asked to provide feedback using a “push and pull” method. Unsolicited feedback is pulled from veterans through the web and social media. Email surveys are pushed to veterans after they have had an interaction with the VA that’s been determined to be a moment that mattered. Surveys are pushed to veterans digitally to capture continuous and real-time information about a veteran’s experience. Moments that matter take the customer transactions and customer journey into consideration, setting customer experience metrics based on customer experience drivers and detractors. For the first time, VA is able to capture veteran feedback continuously, in real-time and to amplify the voice of the veteran to the right people.

To meaningfully measure a veteran’s experience across each transaction along the customer journey, a human-centered approach was used to design the surveys. The survey instrument was created based on eighty-four interviews in nine locations across the country with veterans to understand which moments mattered and what is measurable. The surveys were then vetted again by fourteen veterans to prove comprehension, relevance, and measurable action.

As a program, we measure customer experience at three different levels: the enterprise level, the service level patterns, and point-of-service feedback. The focus of this report is on the second level – service level patterns.

The goal of service level measurements is three-fold:

1. To collect continuous customer experience data at moments that make or break the service experience

2. To help field staff and the national office identify areas for improvement

3. To understand emerging drivers and detractors of customer experience.

This measurement model brings insights and value to all stakeholders at VA. After every moment that matters transaction, veterans are asked about their experience. Front-line VA leaders can resolve individual feedback from veterans and take steps to improve the customer experience for others; additionally, VA executives receive real-time updates on systematic trends that allow them to make changes. VEO intends to identify CX drivers and set measures for all lines of business across all of VA’s agencies.

To achieve this goal, we measure transactions occurring within a service line and, where appropriate, within a specific VA facility across veterans and across time. Service-level patterns complement our enterprise trust trend data by providing actionable service-level understanding of the issues that are driving the customer satisfaction and sentiment. Ultimately, they allow us to identify performance improvements.

The purpose of this document is to define VA’s sampling methodology for veteran’s interactions at the service-level in receiving VA care and/or services. This approach defines the overarching sampling principals applicable for all service-level measurement surveys. The sampling design aims to allow decisions makers at different levels to make inference per agency, across veterans, and across transactions per line of business (LOB). The document provides a highlight of the overall strategy along with the business rules to operationalize the process with focus on the outpatient LOB as a starting point.

B. Moments that Matter Survey Series

Each line of business consists of a customer journey which can be broken out into key customer transactions. These transactions consist of interactions that represent what customers perceive to be part of the journey; within these journeys, a set of selected experiences are considered for measurement—the moments that matter to the veterans.

The first line of business to be sampled is the Outpatient Journey with VHA. Within the Outpatient Journey, our measurable moments that matter were identified. Currently, these experiences are grouped into the following four points in time related to VHA Outpatient Service (as shown in Table 1). This is a starting point and will expand to cover key experiences in all services across all VA agencies (Appendix A. shows the veteran experience cycle).

Table 1. The Outpatient Moments that Matter

Moment that Matters	Description	What is covered	Exclusions
Appointment	Appointments made for a veteran’s outpatient healthcare visit to a VA facility	VA-initiated appointment or veteran-initiated appointments	All Stop Codes related to Telephone, Telehealth, Compensation & Pension, Home Based Care are excluded for the initial release of the Outpatient surveys. Refer to Appendix B for a comprehensive list of excluded Stop Codes.
		Same-day walk-in veterans (the veteran sets up the appointment at the same time as they visited the facility)
Visit	The actual veteran outpatient healthcare visit, which could involve multiple encounters during a single visit.	Encounters within the Visit survey are defined by a set of stop codes within the VHA databases. These codes and their descriptions are outlined in Appendix B.	Refer to Appendix B for a comprehensive list of excluded Stop Codes.
Lab/Imaging	The follow-up visits requested during the outpatient healthcare visit.	Encounters within the Lab/Imaging survey are defined by a set of stop codes within the VHA databases. These codes and their descriptions are outlined in Appendix A.	Refer to Appendix B for a comprehensive list of excluded Stop Codes.
Pharmacy	Prescriptions obtained from VA pharmacies	Mail-order and in-person VA pharmacy interactions.
		Initiated by the veteran or initiated by the VA.

C. Basic Definitions

In order to avoid confusion, we need to differentiate between a transaction and an encounter.

• A transaction. A transaction refers to the specific time a veteran interacts with the VA that both impact the veteran’s journey and the veteran’s perception of VA’s effectiveness in caring for veterans.

• An Encounter. The veteran can participate in multiple actions within a transaction. Each of these actions is defined as an encounter and is transaction-specific.

Therefore, multiple encounters are possible within a transaction and across transactions -- e.g. an endocrinologist visit is one of many encounters within the main healthcare visits for attending a veteran’s needs or a cardiologist visit, lab work, and pharmacy service.

Part II: Methodology

VA is working to transform into the number one customer service agency in the federal government through an in-depth understanding of veterans’ experiences from their point of view. To create a successful customer experience measurement program, we begin with the customer. For each line of business measured through this effort, we start by describing the customer segments to be measured, identifying the differences and similarities in customer journeys, and leveraging quantitative research from the existing data channels to understand customer pain points and business process. We went into the field and used human-centered design research techniques including 1-on-1 interviews, field observations, and contextual inquiry to understand the journey from a veteran’s point of view and identify what drivers the customer experiences at each step. We call the most impactful moments in the journey the moments that matter. Our design research combined with quantitative analysis and stakeholder interviews helped us to identify themes and insights into what moments matter and what to measure across the key customer journey. This is seen on the journey map created as an artifact to visually show the veteran’s customer experience (Figure 1. Outpatient Journey Map).

In order to create a successful transformation, impact was measured through three primary dimensions:

• Effectiveness – “I got the service I needed.”

• Ease – “It was easy to get what I needed.”

• Emotion – “I felt like a valued customer.”

After identifying drivers of the customer experience at specific moments in the journey, we determined how we can trigger the survey through backend data. Once information about customer experiences was gathered over the course of the service, we created a survey framework that included sampling and content guidelines. This outlined key customer satisfaction drivers for the journey being assessed and the journey’s key impact metrics. These measures vary in importance across the journey. Effectiveness, ease, and emotion are present at every moment in the veterans’ journey, including the Outpatient Experience. The graphic below shows how our survey questions and surveys mapped back to the moments that matter to veterans.

Figure 1. Outpatient Journey Map

The Service Level Measurement Surveys are transactional surveys that identify which measures are the most effective at certain points along the journey and are critical for us to meet veterans’ expectations. They are designed to have minimal burden on customers and ask about an interaction the customers had with the line of business. The invitation to the surveys is sent out by email directing the veteran to a web survey that allows for near real-time feedback. The structure of the surveys covers: the five customer satisfaction driver questions in chronological order, an impact metric question, a problem resolution/follow up question, a final comment question. Appendix D shows a copy of the outpatient experience questionnaire where the questions cover:

a. Relevance to the customer’s experience

b. Actionable

c. Based on business objectives

d. Part of VA’s brand

1. Survey Instrument

A.1. Questionnaire Design

The overall process for instrument design was based on four steps as follows:

1. Discovery: Interviews with veterans, employees, and business stakeholders to identify important aspects and feedback needed to refine the questionnaire design

2. Journey Mapping: Synthesis of veterans’ interviews and observations by identifying themes and insights into a measurable journey map

3. Survey Framework: VEO defines which factor is most important at each moment and what good looks like. We decide whether to focus on ease, effectiveness, or emotion. We define the audience, sampling rules, and business rules.

4. Writing Surveys: We write easy to understand questions that measure the key moments that matter based on the research with veterans and industry best practices.

For the Outpatient Experience, VEO and VHA started with a discovery ‘sprint’ to explore patient experiences at the VA -- interviewing veterans across multiple geographic locations and of various genders, races, life stages, military branches, and eras of service. A total of 84 veterans were interviewed across nine research locations for a total of 150 hours of interviews. Figure 2 below shows the distribution of the locations across states covered.

Figure 2. Design Phase Interviews

The research resulted in a journey map which represented the end-to-end patient experience as veterans seek and receive outpatient care. The map is organized into 5 phases: before the visit, arriving at the facility, during the appointment, departing the facility, and after the visit. Each phase included key moments that matter, bright spots, paint points, entry/exit points, and interaction channels.

A.2. Pre-testing of the survey instrument

After the questionnaire design stage was completed, the survey validation process was initiated. This validation step consisted of question comprehension testing with veterans, usability testing of the survey flow, and validation with business stakeholders.

Pre-testing of the Outpatient Journey Surveys was conducted by the VEO, working with the Midwest District and Field Office to recruit veterans from the Detroit and Ann Arbor area. Two facilities were chosen within a close proximity that had different populations and different facility ratings. The VEO completed two rounds of in-person small-scale iterative usability testing with veterans who represented a diverse set of demographics. Each interview totaled 45 minutes. A total of 14 veterans were included in the testing phase at Ann Arbor VA Healthcare System and John D. Dingall Medical Center in Detroit, Michigan. In addition to testing with veterans, the survey was validated with 34 staff members across the two facilities. Table 3 below shows the veteran small-scale usability testing demographic distribution for outpatient experience surveys.

Table 3. Demographic distributions of Veterans Covered in the Pre-test

Demographic	Number
Gender
Male	11
Female	3
Branch of Service/Duty
Army	5
Army, National Guard, Reserves	1
Navy	2
Airforce	4
Marines	1
Unrecorded	1
Age Group
<30	0
30-39	4
40-49	0
50-59	1
>=60	8
N/A
Race
Black or African American	5
White, Native Hawaiian or other Pacific Islander	1
White	8
Healthcare Mix
VA	4
VA, Medicare	3
VA, Private Through Employer Plan, Medicare	2
VA, Private Through Employer Plan	3
VA, Tricare though National Guard	1
VA, Tricare	1

Usability tests on four surveys and one example email invitation were conducted between March 6-10, 2017:

• March 6^th: Veterans were interviewed

• March 7^th: Staff from Primary Care, Specialty Care, Labs, Patient Advocates, and the Medical Center Director were interviewed

• March 8^th: Veterans were interviewed

The feedback from these sessions was used to make questions more specific and clear and led to the decision to create a separate survey for labs. Additionally, we mapped our questions back to CAPHS and SHEP Patient Satisfaction Survey questions. On March 8^th and 9^th we spoke with more veterans and staff and used that feedback to create our final list of surveys and questions. Ultimately, we ended up with four different surveys.

Finally, to test the user experience, a paper prototype was created and simulated the flow of the survey. A card sort exercise was conducted to allow participants to prioritize questions on the Healthcare Survey, and paper prototypes were used to test comprehension.

Part III: Sample Design

Our sampling approach is based on two key considerations:

• Limiting Bias: This is achieved by avoiding over-representing veterans who have a larger number of VA encounters.

• Limiting the impact of quarantine rules used for reducing burden on respondents

To achieve these goals, careful design of the sampling methodology is needed to achieve a full representation of the population of veterans and transactions while reducing the respondent burden to the minimal. Efforts took place in parallel to cover the following:

1. Sampling: This effort covers both the frame and the sampling design.

   1. Frame construction and maintenance:

      1. This effort is focused on constructing a comprehensive dataset of all transactions under considerations taking place during the targeted timeline. It also accounts for multiple transactions per veteran.

      2. The frame maintenance consists of updating transactions on a rolling basis and identifying/updating quarantine rules to reduce potential respondent burden across VA-Wide and MTM rounds.

   2. Sampling Design: The goal of this effort is to design a scheme for selecting veterans and transactions from the frame towards a representative sample of customer receiving VA services across all services.

2. Analysis plan and Dashboard: This effort ensures that the output of the survey caters to all types of decision makers from the executive level to the operational level, also known as role-based dashboarding. The aim is to achieve monthly analytical reports to reflect issues or concerns on a regular basis. Furthermore, it covers measures taken to preserve veterans’ confidentiality by reducing disclosure risk.

3. Operations: This effort is focused on survey implementation logistics and requirements to allow smooth operations of the sampling, data collections, and analysis processes within the Medallia environment over time.

4. Risk Assessment and Quality Control: This effort covers expected challenges and discusses potential mitigation plans. It also covers quality protocols used to reduce risk.

A. Target Population and Frame

The target population for the surveys consists of veterans and their corresponding encounters with VHA outpatient services. The current focus is on veterans who are seeking outpatient health services from VA facilities across the different encounters highlighted in Table 1. The primary sampling units are veterans and the secondary sampling units are the encounters with veterans. Thus, the data can be analyzed to represent either veterans or encounters as the unit of analysis.

It is important to note that each veteran is represented by a cluster of transactions and encounters in the design of this survey. Each transaction and encounter will be identified from the existing VHA databases to represent veterans’ experience. Some of the encounters within the transactions are similar and have been aggregated into groupings of encounters (see Appendix B covering stop codes). In other words, a given data entry will contain a unique record of each encounter a veteran had with the VA during that week as well as a unique identifier for each veteran. Therefore, the population file will contain multiple records for each veteran.

The population file is created on a weekly basis every Saturday night from queries to the VHA databases for all outpatient encounters in the previous week. All outpatient encounters for the week prior to sampling, with the exception of mail-order pharmacy encounters, will be queried from the VHA databases. Mail-order pharmacy encounters in the period of eleven days and four days prior to sampling are queried for the current sample. VEO will extract data on veterans who received outpatient services directly from VHA’s corporate data warehouse during the targeted week. Once the data are pulled by the VEO team, the following quality control checks will be executed:

• Identification of missing data in variables required for later merging and sampling

• Identification of duplicate records per encounter

• Deceased veterans for removal from the VHA files prior to concatenation

Constructing and cleaning of the frame, i.e. the population file, will require several steps as shown in Figure 2.

Figure 2. Flowchart Frame Construction

To reduce respondent burden, the frame will undergo a screening for eligibility process on a weekly basis. Veterans also have the option to opt-out of completing the current and all future surveys by requesting to be put on a “Do Not Contact” list stored in Medallia.

Thus, veterans will be quarantined from sampling based on the following rules:

• Rule 1: Veterans will be excluded from sampling for any transactions if they were sampled for another MTM survey or the APG survey in the past 30 days.

• Rule 2: Veterans will be excluded in a weekly sample if they were also sampled in the previous week.

• Rule 3: Veterans will be excluded from sampling within a transaction if they are sampled for another transaction in the current round of sampling.

• Rule 4: Veterans will be excluded from sampling for a transaction if they requested to be put on a “Do Not Contact” list in the MTM Survey for that transaction.

The quarantine rules will be applied during the creation of the sampling population frame updates on a weekly basis. The quarantined records will be examined during the weekly frame creation to determine if the rules are correlated with demographic strata and the excluded records could bias the final estimate. The distribution of the quarantine records will be compared to the population distribution to identify if there are potential concerns of biasing factors that might impact the analysis and results. Veterans are reintroduced to the sampling pool after their corresponding quarantine period is over unless they had opted out from all future VA surveys.

It is important to note as well the variables and information needed in support of the sampling process. These variables include:

• A unique veteran identifier. This identifier will be maintained over all transactions a veteran conduct.

• A unique encounter identifier must be created.

• A variable will be created that identifies each record based on its line of business.

• A variable will be created that identifies each record based on its type of transaction (i.e., appointment, outpatient, labs, pharmacy, etc.).

• Another variable will be created that identifies the type of encounter within a transaction (i.e., same-day appointment, cardiology visit, laboratory follow-up visit, etc.).

• A variable that identifies the sampling population in which the sampling veteran belongs.

• A variable that identifies the encounter within the sampling population that the veteran was selected for.

• A date of transaction variable needs to be created for all records.

• A certainty stratum identifier.

• A flag that identifies records not eligible for sampling by reason for ineligibility (previously sampled, future transaction)

• The age group variable

• The gender variable

2. Quality Control

With the access to the list of necessary variables for supporting the sampling process, the weekly sampling frames will be examined to understand if the frame distribution consistently resembles the population distribution. The frame distribution will also be compared against the distribution of the quarantine records to identify if any demographic and geographic strata that may introduce biases into the weekly sampling frames. These quality control procedures will be carried out at both veteran and transaction levels: veteran-level quality control, transaction level quality control, and system-wide quality control

C. Sampling Design

The sampling design is a complex systematic sample design with stratification and clusters across different stages (Figure 3). This design will allow the sample to achieve cumulative representation of the veterans across all transactions and encounters under investigation (currently outpatient services out of VBA). The sample will be selected across multiple steps as follows:

Figure 3. Stratified Cluster Design

1. Stage 1: Sampling Transactions

1. To reduce respondent burden, the sample is de-duplicated to keep only one encounter per veteran. The deduplication process will be conducted in a fashion that allows balance across encounters and locations.

2. Transactions are then grouped by line of business¹, encounters, and location (3 digit stations). These groupings are referred to as strata.

3. Encounters will be sampled within location with a minimum of 10 encounters per location for analytical purposes, i.e. to get enough encounters per location for reporting and informing decision makers.

2. Stage 2: Sampling Veterans

   1. Veterans are sorted by state, age group and gender². This ordering allows the sample taken from each stratum to represent each of the three groups according to the distribution of the email population.

   2. A sample of veterans is selected where veterans are the primary sampling units. The sample is proportionally allocated to get a suitable sample size within each group to match the veteran email population distribution.

To avoid a more complex design, any other variables of interest that need to be accounted for will be treated as analytical filters and used as implicit stratification variables during the analysis in the future.

Table 2 below shows an example using a hypothetical sample of 40 encounters selected from a email population of 243 encounters using a matrix design. Record counts and proportions (shown in parentheses) are shown for both the email population and the sample. The distribution of each cell within the population matrix is used to determine the sample distribution. Note that the sample cell proportions follow the population distribution.

Table 2. A hypothetical matrix sample design of encounters.

Counts and overall proportions (in parentheses) are shown.

Encounter	Population					Sample
	Small Facility	Med. Facility	Large Facility	N/A		Small Facility	Med. Facility	Large Facility	N/A
Transaction 1	5 (0.021)	20 (0.082)	60 (0.247)	0 (0.000)		1 (0.025)	3 (0.075)	10 (0.250)	0 (0.000)
Transaction 2	0 (0.000)	10 (0.041)	30 (0.123)	0 (0.000)		0 (0.000)	2 (0.050)	5 (0.125)	0 (0.000)
Transaction 3	0 (0.000)	0 (0.000)	0 (0.000)	100 (0.412)		0 (0.000)	0 (0.000)	0 (0.000)	16 (0.400)
Transaction 4	5 (0.021)	6 (0.025)	7 (0.029)	0 (0.000)		1 (0.025)	1 (0.025)	1 (0.025)	0 (0.000)

In the case of the MTM surveys, encounters within the sampled veterans are selected based on the joint distribution of the population between the type of encounter and the size of the facility. Facilities are classified into small, medium and large facilities based on the number of encounters that facility handles over the previous year. The facility size will not change between weekly samples. Encounters that are not tied to a physical location are classified as ‘N/A’. A new matrix distribution based on the sample frame will be calculated for each weekly sample.

Sampled veterans will then be stratified into two groups, veterans that have exactly one encounter in a week and veterans that have more than one encounter each week. Veterans that have exactly one encounter are assigned to the survey to which that encounter belongs. Veterans that have more than one encounter will be randomly assigned to an encounter based on the population distribution of encounters to assure sample representation and balance.

D. Sample Size

The full population was analyzed to determine a workable number of records to be sampled each week. Due to the fact some strata may not contain enough records to be sampled, the number of records for obtaining 10 responses per 6-digit station by encounter stratum was calculated. Additionally, the targeted sample sizes for the 3-digit station by encounter strata are included for reference. Table 3 below shows the desired number of records for each sample design.

The final transaction-level strata are 3-digit station and encounter. The 6-digit station stratum cannot sustain successive weeks of sampling. However, each 6-digit station is mapped to the parent 3-digit station by the first three digits in the station code. Furthermore, both station code variables are included on the Medallia invitation file, thereby allowing the potential of creating estimates at the 6-digit station level.

Table 3. Population and Sample Sizes

	3-digit station		6-digit station
Total Number of facility*encounter strata across all transactions	5,364	20,028
Required total weekly sample size (number of veterans)
30 responses per month per stratum	452,022	1,687,753
10 responses per month per stratum	150,674	562,584
Percent of facility*encounter strata whose weekly population counts are less than the target weekly stratum sample size
30 responses per month	35.6%	64.6%
10 responses per month	16.1%	46.9%
Percent of facility*encounter strata whose monthly population counts are less than the target weekly stratum sample size
30 responses per month per stratum	12.9%	42.8%
10 responses per month per stratum	6.7%	30.8%

Table 4 shows the stratification level of the 3-digit station field. Table 5 shows the stratification level of the encounter field.

Table 4. 3-digit station strata for each survey.

Table 5. Encounter strata for each survey.

Survey	Encounter
Appts	Regular appointments
Appts	Same-day requested and received
Appts	Same-day requested and did not receive
Appts	Walk-in
Labs/Imaging	Labs/Imaging
Pharmacy	CMOP
Pharmacy	Mailed
Pharmacy	Window
Visit	Allergy & Immunology
Visit	Alternative
Visit	Anesthesia
Visit	Cardiology
Visit	Dermatology
Visit	Dialysis
Visit	EEG/Neurology
Visit	Endocrine/Metabolic and Diabetes
Visit	Gastroenterology
Visit	General Medicine
Visit	General Surgery
Visit	Geriatric Medicine
Visit	Hematology/Oncology
Visit	Infectious Disease
Visit	Medical - All Other
Visit	Mental Health - Mental Health Clinic
Visit	Mental Health - MHSDP Homeless Program
Visit	Mental Health - MHSDP Methadone Treatment Program
Visit	Mental Health - Substance Abuse Clinic
Visit	MHSDP MHICM Program
Visit	Nephrology
Visit	Nuclear Medicine
Visit	Primary Care
Visit	Primary Care - Nurse/PA
Visit	Prosthetics
Visit	Psychosocial Rehabilitation and Recovery Centers
Visit	Pulmonary/ Respiratory Disease
Visit	Radiation Therapy
Visit	Radiology
Visit	Recreational Therapy
Visit	Rehab Medicine
Visit	Rheumatology
Visit	Surgery - Cardiovascular and Thoracic Surgery
Visit	Surgery - Colon Rectal Surgery
Visit	Surgery - ENT
Visit	Surgery - Eye Clinic
Visit	Surgery - General and All Other Surgery
Visit	Surgery - Neurological Surgery
Visit	Surgery - Obstetrics & Gynecology
Visit	Surgery - Orthopedics
Visit	Surgery - Plastic Surgery
Visit	Surgery - Podiatry
Visit	Surgery - Urology
Visit	Urgent Care

The sample will be analyzed to assess coverage and precision by examining the distributions and design effects within each stratum as compared to the true and sampling population. Such reports can be built into the sampling strategy.

E. Survey Invitation File

The sampled records will be compiled by the VA into a file called a survey invitation file that can be uploaded by Medallia. Medallia software will use this survey invitation file to send the veteran survey invitations, maintain the interface for the web survey, capture the data, and calculate estimates to be presented under the embedded dashboard tool. All variables needed by Medallia will be created and appended onto the survey invitation file after the sample is created.

Part IV: Survey Administration

On a semi-weekly basis, VEO provides the Medallia operations team with a stratified sample identifying the veterans who will receive a survey. Using this information, Medallia will distribute surveys to the identified sample of veterans. Service-Level Measurement (SLM) surveys are released semi-weekly, and will allow the veteran to provide feedback anytime over the course of two weeks. Therefore, VEO will contact Veterans 3-4 days after their interaction with VHA. Each weekly survey will have two waves of data collections, beginning on Tuesday with the second launch on Friday. Each wave will be independently sampled, according to the half the sample size targets described in this document. Veterans have two weeks to provide feedback; however, as soon as they complete their survey, their response data is immediately available for review within Medallia.

Part V: Analysis and Dashboarding

The sampling design allows for the unit of analysis to be either the veteran or the encounter. The analysis will be mostly model based rather than design based (no weights calculated and only unweighted analysis will be reported).

The dashboards are intended to give VA employees insights into operation improvements to be made at the local level and patterns occurring at the national and regional level to allow employees to make strategic decisions. Dashboards display data from the survey scores and allow users to view the scores by the different strata: age, gender, location. Depending on the VA employee’s role, the user will have access to different information.

Users should be able to filter information down to the level of the service name if there are 30 or more comments for that service. Users can see score data for each individual survey and a combined trust score from all three surveys. Dashboards will also give VE admins the opportunity to monitor the health of their survey campaign and email campaign by displaying analytics data on both surveys and emails.

Part VI: Limitations and Mitigations

This section highlights various challenges and how these challenges are mitigated by the current sampling design.

Sequencing and Conditionality

Transactions are defined within each VA line of business. Therefore, transactions may be conditional on other transactions. Given the rules for removing veterans from the sampling pool, this conditionality will affect how records are sampled. For example, a veteran may make an appointment for a set of laboratory tests the following day as an outpatient visit at a VA hospital. At that visit, he or she is also written a prescription that he or she then fills at a VA pharmacy. In this scenario, the veteran’s visit transaction only happens after the appointment transaction, and the pharmacy transaction happens after both transactions.

Given that the appointment happens before both the outpatient visit and the pharmacy visit, the sampling probabilities would otherwise be conditional for the transactions that take place after the appointment and sequentially. This excludes a census approach because, under a census approach, all records in the situation mentioned above are selected for the appointment and none of them will be eligible for the outpatient visit or pharmacy visit surveys. This also introduces a complication in the stratified sample design in that receiving or completing the survey becomes a stratification variable.

Thus, to mitigate the issues of conditionality, the sampling methodology includes the veteran into the sampling pools for all transactions. Then, the current matrix-based sampling plan allows a balanced sample across all transactions and encounters without being restricted by the chronological order of the events. For instance, a veteran in the situation above would be included in the sampling pools for all three transactions and would then be randomly assigned to only one of the surveys based on the types of transaction they had.

Compound Transaction

An individual transaction may be made up of a set of encounters, each of which may have its own survey. For example, a veteran may be directed to have a radiology test and a bloodwork test in one outpatient visit. And with each test being handled by different departments in a VA hospital, these tests are considered as individual transactions separated from one and another. The above sample plan will effectively address the compound issue by assigning a veteran to only one transaction among the three (i.e. healthcare visit, radiology test, and bloodwork).

Appendix A. Services by VA Administration

Appendix B. Eligible Stop Code List.

Service Name	Sub Services	Stop Codes	Inclusion Status
Allergy & Immunology
		302
Alternative
		159	EXCLUDE
Anesthesia
		427
Cardiology
		107
		231
		303
		311
		317
		333
		334
		369
Dental
		180
Dermatology
		304
Dialysis
		602
		603
		604	EXCLUDE
		606
		607
		608
Emergency Department
		130	EXCLUDE
Endocrinology
	Endocrine/Metabolic and Diabetes
		305
		306
Eye Clinic - Ophthalmology/Optometry
		438
Gastroenterology
	Digestive/GI/Endoscopy
		307
		321
General Medicine
		339
		340
		349
Geriatric Medicine
		190	EXCLUDE
		318
		319
		320
		351	EXCLUDE
		352
		353
Hematology/Oncology
		308
		316
		330
		431	EXCLUDE
Infectious Disease
		310
		717
Laboratory & Pathology
		108
		111	EXCLUDE
Medical - All Other
		329
		394
		420
		331	EXCLUDE
		328	EXCLUDE
		332	EXCLUDE
		336	EXCLUDE
Mental Health
		504	EXCLUDE
		507	EXCLUDE
		508	EXCLUDE
		511	EXCLUDE
		534
		555	EXCLUDE
		556	EXCLUDE
		586	EXCLUDE
		587	EXCLUDE
		591
		592
		593	EXCLUDE
		595	EXCLUDE
		596	EXCLUDE
		506	EXCLUDE
		599	EXCLUDE
	Mental Health - Day Treatment Center
		554	EXCLUDE
	Mental Health - Mental Health Clinic	502
		509
		512
		519
		524
		525
		531
		533
		538
		540
		562
		565
		580
		516	EXCLUDE
		550	EXCLUDE
		566	EXCLUDE
		572	EXCLUDE
		557	EXCLUDE
		561	EXCLUDE
		564	EXCLUDE
		571	EXCLUDE
	Mental Health - MHSDP Community MH Residential Care
		121	EXCLUDE
		503
		505
		584	EXCLUDE
		583	EXCLUDE
		553	EXCLUDE
		582	EXCLUDE
	Mental Health - MHSDP Homeless Program
		529
		522	EXCLUDE
		590	EXCLUDE
	Mental Health - MHSDP Methadone Treatment Program
		523
	Mental Health - MHSDP Work Therapy Program
		222	EXCLUDE
		535	EXCLUDE
		568	EXCLUDE
		573	EXCLUDE
		574	EXCLUDE
		575	EXCLUDE
	Mental Health - Psychology
		510
		558	EXCLUDE
	Mental Health - Substance Abuse Clinic
		513
		514
		548
		547	EXCLUDE
		560	EXCLUDE
	MHSDP MHICM Program
		552
		567	EXCLUDE
Nephrology
		313
Neurology
		215	EXCLUDE
		225	EXCLUDE
		346
		347	EXCLUDE
	EEG/Neurology
		106
		126
		128
		155
		315
		335
		345
Nuclear Medicine
		109
		146
		158
		145	EXCLUDE
Nursing		No Codes Identified
Pharmacy
		160
Primary Care
		120	EXCLUDE
		123
		124	EXCLUDE
		142
		301
		309
		322
		323
		341
		342
		348	EXCLUDE
		350
		371	EXCLUDE
		372
		373
		450	EXCLUDE
		690	EXCLUDE
		701
		704
		706
		707
		710	EXCLUDE
		999	EXCLUDE
	Primary Care - Nurse/PA
		117
Prosthetics
		588
		440	EXCLUDE
		449
		598	EXCLUDE
Psychosocial Rehabilitation and Recovery Centers
		104
		116
		312
Pulmonary/ Respiratory Disease
		481
Radiation Therapy
		144
		149
Radiology
		105
		110
		115
		127
		150
		151
		153
		154
		703
		202
		197
Recreational Therapy
		201
		205
Rehab Medicine
		195
		196	EXCLUDE
		206
		209
		210
		211
		212
		214
		217
		218
		220
		230
		240
		250
		417
		418
		423
		436
		437
		439
		213	EXCLUDE
		207	EXCLUDE
		208	EXCLUDE
		198	EXCLUDE
Rheumatology
		314
Surgery
		434
	Surgery - Audiology
		203
		204
	Surgery - Cardiovascular and Thoracic Surgery
		402
		413
		415
		421
	Surgery - Colon Rectal Surgery
		412
	Surgery - ENT
		402
	Surgery - Eye Clinic
		407
		408
		718
	Surgery - General and All Other Surgery
		327
		401
		416
		419
		429
		432
		433
		435
		457
	Surgery - Neurological Surgery
		406
	Surgery - Obstetrics & Gynecology
		404
		426
	Surgery - Orthopedics
		405
		409
		422
	Surgery - Plastic Surgery
		410
	Surgery - Podiatry
		411
	Surgery - Urology
		414
		430
Urgent Care
		102	EXCLUDE
Not Mapped
		651	EXCLUDE
		658	EXCLUDE
		669	EXCLUDE
		131	EXCLUDE
		103	EXCLUDE
		118	EXCLUDE
		119	Home-Based
		135	Post Deployment Integrated Care
		136	EXCLUDE
		137	EXCLUDE
		143	Sleep Study
		147	EXCLUDE
		148	EXCLUDE
		156	EXCLUDE
		157	EXCLUDE
		162	EXCLUDE
		165	EXCLUDE
		166	EXCLUDE
		167	EXCLUDE
		168	EXCLUDE
		169	EXCLUDE
		170	EXCLUDE
		171	EXCLUDE
		172	EXCLUDE
		173	EXCLUDE
		174	EXCLUDE
		175	EXCLUDE
		176	EXCLUDE
		177	EXCLUDE
		178	EXCLUDE
		179	EXCLUDE
		181	EXCLUDE
		182	EXCLUDE
		183	EXCLUDE
		184	Care/Case Manager
		185	EXCLUDE
		186	EXCLUDE
		187	EXCLUDE
		188	EXCLUDE
		189	EXCLUDE
		191	EXCLUDE
		199	EXCLUDE
		216	EXCLUDE
		221	EXCLUDE
		224	EXCLUDE
		229	EXCLUDE
		295	EXCLUDE
		296	EXCLUDE
		297	EXCLUDE
		324	EXCLUDE
		325	EXCLUDE
		326	EXCLUDE
		338	EXCLUDE
		354	EXCLUDE
		370	Long Term Care
		424	EXCLUDE
		425	EXCLUDE
		428	EXCLUDE
		441	EXCLUDE
		443	EXCLUDE
		444	EXCLUDE
		445	EXCLUDE
		446	EXCLUDE
		447	EXCLUDE
		448	EXCLUDE
		474	EXCLUDE
		490	EXCLUDE
		491	EXCLUDE
		527	EXCLUDE
		528	EXCLUDE
		530	EXCLUDE
		536	EXCLUDE
		542	EXCLUDE
		545	EXCLUDE
		546	EXCLUDE
		579	EXCLUDE
		597	EXCLUDE
		611	EXCLUDE
		644	EXCLUDE
		645	EXCLUDE
		646	EXCLUDE
		647	EXCLUDE
		648	EXCLUDE
		649	EXCLUDE
		652	EXCLUDE
		653	EXCLUDE
		656	EXCLUDE
		673	EXCLUDE
		674	EXCLUDE
		680	EXCLUDE
		681	EXCLUDE
		682	EXCLUDE
		683	EXCLUDE
		684	EXCLUDE
		685	EXCLUDE
		686	EXCLUDE
		692	EXCLUDE
		693	EXCLUDE
		694	EXCLUDE
		695	EXCLUDE
		696	EXCLUDE
		697	EXCLUDE
		698	EXCLUDE
		699	EXCLUDE
		708	Telehealth
		713	EXCLUDE
		714	EXCLUDE
		719	EXCLUDE
		720	EXCLUDE
		801	EXCLUDE
		802	EXCLUDE
		803	EXCLUDE
		901	EXCLUDE

1 At this point, we are only focusing on the outpatient service as the main line of business. However, in the future additional lines of businesses will be included. This is the main reasoning behind including line of business as a stratification/grouping variable.

2 State, gender and age group are treated as implicit strata within the sampling design.

14