Form CMS-10534 Cognitive Interview Guide, Screener and Consent Forms

Attachment A: Recruitment screeners and
related communications for each audience
Consumers
Recruitment guidelines


Recruit to have 6 individuals for 5 participants to show



Each interview will last 1.5 hours (90 minutes)



Participants will receive $100 for their time



General screening criteria for all interviews:
1. Health insurance status:
a. Enrolled in a health insurance plan through a state-based or federal marketplace
b. Uninsured: Do not currently have health insurance from any source (not parents,
not work, not Medicaid, not spouse, etc.)
2. 21 to 64 years of age (Mix of ages)
3. Mix of genders
4. Mix of education levels (e.g., less than high school, high school only, some college,
and college degree, preferably skewed towards lower educational levels)
5. Mix of races/ethnicities (as possible)
6. Mix of coverage needs (individual vs. family) (as possible)



Additional screening criteria, recruit two of each type of participant:
7. Participant Type 1: Chronic conditions
• Must currently be diagnosed with a chronic condition such as asthma, diabetes,
hypertension, etc.
8. Participant Type 2: Low income
o Must have an income below 250% of the federal poverty guideline (but
not enrolled in Medicaid), which is specified as follows:
People in household Income threshold
1
$28,725.00
2
$38,775.00
3
$48,825.00
4
$58,875.00
5
$68,925.00
6
$78,975.00
7
$89,025.00
8
$99,075.00

1

9. Participant Type 3: Young and healthy
o Must be between the ages of 21 and 34
o Must not have been diagnosed (now or ever) with any chronic health
condition
10. Participant Type 4: Healthy and over 35
o Must be between the ages of 35 and 64
o Must not have been diagnosed (now or ever) with any chronic health
condition

Screener
INTRO WHEN PERSON ANSWERS THE PHONE:
Hello, may I please speak with [FIRST AND LAST NAME]? My name is [RECRUITER FIRST
AND LAST NAME], and I'm calling on behalf of the American Institutes for Research (also
known as A-I-R), a nonprofit research organization.
IF RESPONDENT NOT AVAILABLE EITHER:
 DETERMINE A GOOD TIME TO CALL BACK OR
 LEAVE MESSAGE ON ANSWERING MACHINE
ANSWERING MACHINE SCRIPT:
Hello, my name is [RECRUITER FIRST AND LAST NAME]. I'm calling on behalf of
the American Institutes for Research, a nonprofit research institute. We would like for
you to participate in an interview about comparing and choosing health insurance
plans. Please call {INSERT PHONE NUMBER} at your convenience.
IF RESPONDENT IS AVAILABLE, IDENTIFY REASON FOR CALLING.
I’m calling today about a research project that we are doing about comparing and choosing
health insurance plans.
If you are interested in helping with this project, and you meet the requirements for participation,
we will invite you to come to our facility in [TBD] for an interview discussion. It would take two
and a half hours of your time, and we would give you $100 to thank you for participating.
May I ask you a few questions to see if you meet our criteria for interview participants?
IF NEEDED, EXPLAIN FURTHER: We need to include people with specific
characteristics and experiences so that we end up with a mix of people with different
backgrounds. So I have to ask you some questions to see if you fit the profile of
participants that AIR is looking for in the interviews.
IF YES, CONTINUE TO QUESTION 1. IF NO, THANK & END (END SCRIPT A).

2

Before we begin, I’d like to let you know that all information you provide will be held in
confidence. Your participation is voluntary, and you may ask me to skip any questions that you
do not wish to answer. You can stop at any time. None of the information that you provide to us
will be used for any purpose outside of this study.
1. Do you currently have health insurance? This includes health insurance from any source
such as Medicaid, Medicare, private insurance such as Aetna, Blue Cross Blue Shield,
Kaiser, etc., and insurance coverage through your employer or through your spouse.
YES

NO

IF NO, go to Q4.
2. Did you buy your insurance through the [Healthcare.gov federal marketplace, name of
state-based Marketplace]? [You could have enrolled online through the [Healthcare.gov
federal marketplace, name of state-based Marketplace], over the phone, or with the help
of [an insurance broker/ insurance navigator or assistant]
YES

NO

IF NO, THANK AND END (END SCRIPT B).
3. How many people, including yourself, are covered by your health insurance plan?
People
IF 2 OR MORE, who is covered by your health insurance plan?
Individual

Spouse/Partner

Dependents/Children
4. Have you ever worked for any of the following:
A health insurance company, THANK AND END (END SCRIPT B).
Doctor’s office or clinic, THANK AND END (END SCRIPT B).
Hospital, THANK AND END (END SCRIPT B).
Centers for Medicare and Medicaid Services , THANK AND END (END SCRIPT
B).

5. What is your age?
years
old
3

IF YOUNGER THAN 21OR OLDER THAN 64, END SCRIPT B
6. How many people, including yourself, currently live in your household?
People
7. Which of the following categories best describes the total income of your household in
2013, before taxes? Please include before-tax income from all sources such as salaries
and wages, Social Security, retirement income, investments, and other sources. Would
you say…(RECRUIT MIX)
Less than $20,000
$20,000 to $29,999
$30,000 to $39,999
$40,000 to $49,999
$50,000 to $59,999
$60,000 to $69,999
$70,000 to $79,999
$80,000 to $89,999
$90,000 to $99,999
More than $100,000
Refused to answer the question
USE QUESTIONS 5 AND 6 TO DETERMINE IF RESPONDENT MEETS THE INCOME
REQUIREMENTS for low income group. FOR EXAMPLE, ONE PERSON IN
HOUSEHOLD AND A YES TO $20,000 - $29,999 CATEGORY WOULD QUALIFY THE
RESPONDENT.
People in household Income threshold
1
$28,725.00
2
$38,775.00
3
$48,825.00
4
$58,875.00
5
$68,925.00
6
$78,975.00
7
$89,025.00
8
$99,075.00
RESPONDENT QUALIFIES AS A LOW INCOME PARTICIPANT?
 YES  NO

4

8. Have you ever been diagnosed with: {ask each}
A.
B.
C.
D.
E.
F.
G.
H.
I.
J.

Diabetes
Asthma
Heart Disease
Depression
Congestive Heart
Failure
Cancer
Arthritis
COPD (lung disease)
Kidney Disease
Other chronic
condition
Specify Other:

YES
YES
YES
YES

NO
NO
NO
NO

YES

NO

YES
YES
YES
YES

NO
NO
NO
NO

YES

NO

DOES RESPONDENT QUALIFY AS A YOUNG AND HEALTHY PARTICIPANT?
(BETWEEN THE AGES OF 21 – 34 (Q4) AND ANSWERS NO TO ALL IN Q7)
 YES  NO
DOES RESPONDENT QUALIFY AS A 35 AND OVER PARTICIPANT? (BETWEEN
AGES OF 35-64 (Q4) AND ANSWERS NO TO ALL IN Q7)
DOES RESPONDENT QUALIFY AS A CHRONIC CONDITIONS PARTICIPANT?
(ANSWERS YES TO ANY OF THE ITEMS IN Q7)
 YES  NO
IF YES TO ‘J’ ONLY, AND UNSURE IF THE OTHER CONDITION QUALIFIES AS
A CHRONIC CONDITION, PROCEED THROUGH SCREENER. IF THEY ARE NOT
DISQUALIFIED BY ANY OF THE OTHER QUESTIONS, READ END SCRIPT C
PROCEED IF RESPONDENT QUALIFIES FOR ANY OF THE FOUR PARTICIPANT
TYPES
9. What is your gender?
Male
Female
{Recruit a mix of men and women}

5

10. What is the highest grade or level of school you have completed?
Less than high school graduate
High school diploma
GED
Technical or vocational school or certificate program
Associate’s Degree (2 year college graduate)
Some college (1 – 3 years of college, no degree)
Bachelor’s Degree
Graduate degree (Master’s or Doctorate)
{Recruit a mix of educational levels, preferably skewed to the low end of the educational
spectrum}
11. Are you of Spanish, Hispanic, or Latino background?
YES

NO

{Recruit a mix of races/ethnicity using Questions 10 and 11}
12. How would you describe your race? (check all that apply)
American Indian or Alaskan Native
Asian or Pacific Islander
Black or African American
White
Another race (Specify other race: _________________)
{Recruit a mix of races/ethnicity using Questions 10 and 11}
INVITATION IF INDIVIDUAL MEETS RECRUITING CRITERIA:
Thank you for answering all of my questions. It looks like you’re eligible to participate in the
interview.
We are having interviews on [Dates, Location]. You will be given $100 as a thank you for
participating in the 90 minute interview. Would any of the following dates or times work for
you?





Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time

IF NO:
Okay, is it okay if I call you if other days and times become available? {GO TO END
SCRIPT A}

6

CONFIRM APPOINTMENT
So, let me confirm that you are agreeing to participate in an interview about comparing and
choosing health insurance plans on {Date and time}.
CONFIRM RESPONDENT’S INFORMATION
Now, let me just verify the spelling of your name and your address so we can send you a
confirmation letter with directions. We will not use this information for any purpose outside of
this study.
Name:______________________________________Telephone:_______________________
Address:_____________________________________________________________________
City, State:____________________________________________Zip:____________________
Also, please let me confirm that I have all your correct phone numbers so that we may call and
confirm with you.
Home phone: ___________________________ Other Phone: __________________________
If you have any questions or find that you can’t attend, please call us right away at {insert phone
number} so that we can find a replacement. We also ask that you do not bring any friends or
children with you as they will not be allowed to participate in the interview. You will be
reviewing printed materials, so please bring reading glasses if you need them. Thank you for
your time and for agreeing to help.
END SCRIPT A (GENERIC THANK AND END):
I appreciate your taking the time to speak with me and I hope you have a good day.
END SCRIPT B (FOR PEOPLE WHO DO NOT MEET SELECTION CRITERIA):
Thank you very much for answering my questions. As I said earlier, we’re trying to recruit
people with specific characteristics and a mix of backgrounds. Unfortunately {you do not meet
our selection criteria OR we already have enough people in our study with backgrounds that are
similar to yours}. I appreciate your taking the time to speak with me and I hope you have a good
day.
END SCRIPT C
Thank you very much for answering my questions. I need to check with my supervisor to verify
whether you are eligible for this research study. If you are, I will call you back to schedule.

7

Small business employers
Recruitment guidelines


Recruit to have 6 individuals for 5 participants to show



Each interview will last 60 minutes and be conducted by telephone



Participants will receive $100 for their time; Executives will receive $200



General screening criteria for all interviews:
1. Benefit managers or executives who are responsible for identifying or selecting health
plans for their organization
2. Organization with fewer than 200 covered lives or 100 employees
3. Mix of for-profit and non-profit organizations
4. Mix of self-insured, fully insured, and no insurance offered
5. Geographic diversity (e.g., northeast, southwest)
6. Industry diversity (e.g., manufacturing, information technology)

Screener
INTRO WHEN PERSON ANSWERS THE PHONE:
Hello, may I please speak with [FIRST AND LAST NAME]? My name is [RECRUITER FIRST
AND LAST NAME], and I'm calling on behalf of the American Institutes for Research (also
known as A-I-R), a nonprofit research organization.
IF RESPONDENT NOT AVAILABLE EITHER:
 DETERMINE A GOOD TIME TO CALL BACK OR
 LEAVE MESSAGE ON ANSWERING MACHINE
ANSWERING MACHINE SCRIPT:
Hello, my name is [RECRUITER FIRST AND LAST NAME]. I'm calling on behalf of
the American Institutes for Research, a nonprofit research institute. We would like for
you to participate in an interview about reporting the results of a survey of patient
experience in the [Health Insurance Marketplace/Qualified Health Plans in the
Marketplace]. Please call {INSERT PHONE NUMBER} at your convenience.
IF RESPONDENT IS AVAILABLE, IDENTIFY REASON FOR CALLING.
I’m calling today about a research project that we are doing about a survey of patient experience
in the [Health Insurance Marketplace/Qualified Health Plans in the Marketplace].
If you are interested in helping with this project, and you meet the requirements for participation,
we will invite you to participate in a telephone interview. It would take one hour of your time,
and we would give you [$100; $200 for executives] to thank you for participating.
May I ask you a few questions to see if you meet our criteria for interview participants?

8

IF NEEDED, EXPLAIN FURTHER: We need to include people with specific
characteristics and experiences so that we end up with a mix of people with different
backgrounds. So I have to ask you some questions to see if you fit the profile of
participants that AIR is looking for in the interviews.
IF YES, CONTINUE TO QUESTION 1. IF NO, THANK & END (END SCRIPT A).
Before we begin, I’d like to let you know that all information you provide will be held in
confidence. Your participation is voluntary, and you may ask me to skip any questions that you
do not wish to answer. You can stop at any time. None of the information that you provide to us
will be used for any purpose outside of this study.
1. How many covered lives does your organization cover under all health insurance plans?
____________ (If over 200, THANK AND END, END SCRIPT B)
2. How many employees work at your organization?
____________ (If over 100, THANK AND END, END SCRIPT B)
3. Is your organization: (Recruit mix)
For profit?
Not-for-profit?
Other/not sure?
4. What type of health insurance plan(s) does your organization offer? (check all that apply)
None
HMO (Health Maintenance Organization)
PPO (preferred provider organization)
Discounted fee-for-service
Self-insured plan
Full-insured plan
5. In what city and state is your organization’s primary offices?
_________________________________
6. What business sector does your organization operate? [If needed: Consulting?
Manufacturing? Information technology?]?
_________________________________
9

INVITATION IF INDIVIDUAL MEETS RECRUITING CRITERIA:
Thank you for answering all of my questions. It looks like you’re eligible to participate in the
interview.
We are having interviews on [Dates] by telephone. You will be given [$100; $200 if an
executive] as a thank you for participating in the 60 minute interview. Would any of the
following dates or times work for you?





Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time

IF NO:
Okay, is it okay if I call you if other days and times become available? {GO TO END
SCRIPT A}
CONFIRM APPOINTMENT
So, let me confirm that you are agreeing to participate in an interview about comparing and
choosing health insurance plans on {Date and time}.
CONFIRM RESPONDENT’S INFORMATION
Now, let me just verify the spelling of your name and your address so we can send you a
confirmation letter with directions. We will not use this information for any purpose outside of
this study.
Name:______________________________________Telephone:_______________________
Address:_____________________________________________________________________
City, State:____________________________________________Zip:____________________
Also, please let me confirm that I have all your correct phone numbers so that we may call and
confirm with you.
Home phone: ___________________________ Other Phone: __________________________
If you have any questions or find that you can’t attend, please call us right away at {insert phone
number} so that we can find a replacement. We also ask that you do not bring any friends or
children with you as they will not be allowed to participate in the interview. You will be
reviewing printed materials, so please bring reading glasses if you need them. Thank you for
your time and for agreeing to help.
END SCRIPT A (GENERIC THANK AND END):
I appreciate your taking the time to speak with me and I hope you have a good day.
10

END SCRIPT B (FOR PEOPLE WHO DO NOT MEET SELECTION CRITERIA):
Thank you very much for answering my questions. As I said earlier, we’re trying to recruit
people with specific characteristics and a mix of backgrounds. Unfortunately {you do not meet
our selection criteria OR we already have enough people in our study with backgrounds that are
similar to yours}. I appreciate your taking the time to speak with me and I hope you have a good
day.
END SCRIPT C
Thank you very much for answering my questions. I need to check with my supervisor to verify
whether you are eligible for this research study. If you are, I will call you back to schedule.

11

Health plan quality managers
Recruitment guidelines


Recruit to have 6 individuals for 5 participants to show



Each interview will last 60 minutes and be conducted by telephone



No incentives will be offered



General screening criteria for all interviews:
1. Work for a qualified health plan that exists in the Marketplace
2. Work in quality improvement or management for at least one year
3. Have interest in providing input on health quality measures and how best to present
the information
4. As possible, responsible for designing consumer materials for the Marketplace

Screener
1. Does your insurance company currently offer any products in the Health Insurance
Marketplace(s)?
 YES  NO
2. What is your title?
__________________________________________
3. Does your company offer a health plan under the federal or one of the state Health
Insurance Marketplaces?
 YES  NO
4. Does your primary role at the health plan include any of the following?
Quality improvement (collection, analysis, or report)
Patient satisfaction assessment
Public reporting
Development of materials for potential consumers
{If no to all, THANK AND END, END SCRIPT B}
5. How long have you worked in quality improvement or management, patient satisfaction,
public reporting, or developing materials for consumers in the Marketplace?
______________________

{If less than 1 year, THANK AND END, END SCRIPT B}

12

INVITATION IF INDIVIDUAL MEETS RECRUITING CRITERIA:
Thank you for answering all of my questions. It looks like you’re eligible to participate in the
interview.
We are conducting the 60-minute interviews on [Dates] by telephone. Would any of the
following dates or times work for you?





Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time

IF NO:
Okay, is it okay if I call you if other days and times become available? {GO TO END
SCRIPT A}
CONFIRM APPOINTMENT
So, let me confirm that you are agreeing to participate in an interview about comparing and
choosing health insurance plans on {Date and time}.
CONFIRM RESPONDENT’S INFORMATION
Now, let me just verify the spelling of your name and your address so we can send you a
confirmation letter with directions. We will not use this information for any purpose outside of
this study.
Name:______________________________________Telephone:_______________________
Address:_____________________________________________________________________
City, State:____________________________________________Zip:____________________
Also, please let me confirm that I have all your correct phone numbers so that we may call and
confirm with you.
Home phone: ___________________________ Other Phone: __________________________
If you have any questions or find that you can’t attend, please call us right away at {insert phone
number} so that we can find a replacement. We also ask that you do not bring any friends or
children with you as they will not be allowed to participate in the interview. You will be
reviewing printed materials, so please bring reading glasses if you need them. Thank you for
your time and for agreeing to help.
END SCRIPT A (GENERIC THANK AND END):
I appreciate your taking the time to speak with me and I hope you have a good day.

13

END SCRIPT B (FOR PEOPLE WHO DO NOT MEET SELECTION CRITERIA):
Thank you very much for answering my questions. As I said earlier, we’re trying to recruit
people with specific characteristics and a mix of backgrounds. Unfortunately {you do not meet
our selection criteria OR we already have enough people in our study with backgrounds that are
similar to yours}. I appreciate your taking the time to speak with me and I hope you have a good
day.
END SCRIPT C
Thank you very much for answering my questions. I need to check with my supervisor to verify
whether you are eligible for this research study. If you are, I will call you back to schedule.

14

Marketplace regulators
Recruitment guidelines


Recruit to have 6 individuals for 5 participants to show



Each interview will last 60 minutes and be conducted by telephone



No incentives will be offered



General screening criteria for all interviews:
1.
2.
3.
4.

Work for federal or state Marketplace or contractor
Work on or with the Marketplace(s) in a regulatory capacity for at least 6 months
Mix of respondents from federal and state-based Marketplaces
Mix of geographic locations

Screener
1. What is your title?
__________________________________________
2. Does your role at the Health Insurance Marketplace include regulation, quality
improvement, or quality management?
 YES  NO
3. How long have you worked in in a regulatory capacity for the Health Insurance
Marketplace?
______________________

{If less than 1 year, THANK AND END, END SCRIPT B}

4. What state(s) does your Marketplace serve?
___________________________________
INVITATION IF INDIVIDUAL MEETS RECRUITING CRITERIA:
Thank you for answering all of my questions. It looks like you’re eligible to participate in the
interview.

15

We are conducting the 60-minute interviews on [Dates] by telephone. Would any of the
following dates or times work for you?





Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time






Option: Date, time
Option: Date, time
Option: Date, time
Option: Date, time

IF NO:
Okay, is it okay if I call you if other days and times become available? {GO TO END
SCRIPT A}
CONFIRM APPOINTMENT
So, let me confirm that you are agreeing to participate in an interview about comparing and
choosing health insurance plans on {Date and time}.
CONFIRM RESPONDENT’S INFORMATION
Now, let me just verify the spelling of your name and your address so we can send you a
confirmation letter with directions. We will not use this information for any purpose outside of
this study.
Name:______________________________________Telephone:_______________________
Address:_____________________________________________________________________
City, State:____________________________________________Zip:____________________
Also, please let me confirm that I have all your correct phone numbers so that we may call and
confirm with you.
Home phone: ___________________________ Other Phone: __________________________
If you have any questions or find that you can’t attend, please call us right away at {insert phone
number} so that we can find a replacement. We also ask that you do not bring any friends or
children with you as they will not be allowed to participate in the interview. You will be
reviewing printed materials, so please bring reading glasses if you need them. Thank you for
your time and for agreeing to help.
END SCRIPT A (GENERIC THANK AND END):
I appreciate your taking the time to speak with me and I hope you have a good day.
END SCRIPT B (FOR PEOPLE WHO DO NOT MEET SELECTION CRITERIA):
Thank you very much for answering my questions. As I said earlier, we’re trying to recruit
people with specific characteristics and a mix of backgrounds. Unfortunately {you do not meet
our selection criteria OR we already have enough people in our study with backgrounds that are
similar to yours}. I appreciate your taking the time to speak with me and I hope you have a good

16

day.
END SCRIPT C
Thank you very much for answering my questions. I need to check with my supervisor to verify
whether you are eligible for this research study. If you are, I will call you back to schedule.

17

Attachment B: Consent forms for each audience
Consumers
What is this project about and what will you ask me to do?
We are interested in finding out your reaction to descriptions of results of a survey completed by enrollees
in Qualified Health Plans in the Health Insurance Marketplaces. This interview will take about 90
minutes. As a thank you for completing the interview, we will give you a cash incentive of $100.
Who is doing this project?
This project is being conducted by the American Institutes for Research (AIR), a not-for-profit social
science research organization headquartered in Washington, DC. The project is funded by the Centers for
Medicare & Medicaid Services. Steven Garfinkel, Ph.D. at American Institutes for Research is the Project
Director.
Do I have to participate in this project?
No. It is your choice whether to participate or not. Also, you have the right to stop participating at any
time, and you do not have to answer any questions that you prefer not to answer. If you choose not to
participate or stop participating, there are no penalties and you will receive the full incentive.
What are the risks and benefits?
There are no anticipated or known risks in participating in this project. There are no direct benefits to you
for participating in an interview. By participating in this project, you will receive the opportunity to
provide input on how to best describe results of these types of surveys.
How will you protect my privacy?
With your permission, we will be audio-recording the interview for reference, to ensure accuracy in
capturing what you share with us. The recordings will be destroyed no later than the end of the project
(approximately one year). We will keep your identity and the information you supply confidential and
will not share this information with anyone outside of the project staff.
What if I want more information?
• If you want more information about this project, please contact the director of the research project
at AIR, Steven Garfinkel, sgarfinkel@air.org, (919) 918-2306.
• If you have questions about your rights as a participant, contact the chair of AIR’s
Institutional Review Board, at IRB@air.org or toll-free at 1-800-634-0797 or c/o
AIR, Attn: AIR IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007.
By signing this form you are indicating that you have read and understood the information
provided to you and agree to participate in the interview.
Signature________________________________

Today’s date________________

Small Business Employers
What is this project about and what will you ask me to do?
We are interested in finding out your reaction to descriptions of results of a survey completed by enrollees
in Qualified Health Plans in the Health Insurance Marketplaces. This interview will take about 60
minutes. As a thank you for participating, we will give you a cash incentive upon completing the
interview.
Who is doing this project?
This project is being conducted by the American Institutes for Research (AIR), a not-for-profit social
science research organization headquartered in Washington, DC. The project is funded by the Centers for
Medicare & Medicaid Services. Steven Garfinkel, Ph.D. at American Institutes for Research is the Project
Director.
Do I have to participate in this project?
No. It is your choice whether to participate or not. Also, you have the right to stop participating at any
time, and you do not have to answer any questions that you prefer not to answer. If you choose not to
participate or stop participating, there are no penalties and you will receive the full incentive.
What are the risks and benefits?
There are no anticipated or known risks in participating in this project. There are no direct benefits to you
for participating in an interview. By participating in this project, you will receive the opportunity to
provide input on how to best describe results of these types of surveys.
How will you protect my privacy?
With your permission, we will be audio-recording the interview for reference, to ensure accuracy in
capturing what you share with us. The recordings will be destroyed no later than the end of the project
(approximately one year). We will keep your identity and the information you supply confidential and
will not share this information with anyone outside of the project staff.
What if I want more information?
• If you want more information about this project, please contact the director of the research project
at AIR, Steven Garfinkel, sgarfinkel@air.org, (919) 918-2306.
• If you have questions about your rights as a participant, contact the chair of AIR’s
Institutional Review Board, at IRB@air.org or toll-free at 1-800-634-0797 or c/o
AIR, Attn: AIR IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007.
Verbal Consent: Please answer yes or no to each of the following questions.
Do you understand the described project and interview and agree to be a participant as part of
this project?
Do you agree to have the interview recorded?






Do you understand that your name will not be associated with reports or documents related to
this project?



Do you understand that you can withdraw your consent at any time and stop participating in the
interview without any prejudice to you?



Name________________________________

Today’s date________________





Yes
No
Yes
No
Yes
No
Yes
No

Health plan quality managers, Marketplace Regulator
What is this project about and what will you ask me to do?
We are interested in finding out your reaction to descriptions of results of a survey completed by enrollees
in Qualified Health Plans in the Health Insurance Marketplaces or others that completed a survey about
their experience obtaining insurance in the Marketplaces. This interview will take about 60 minutes.
Who is doing this project?
This project is being conducted by the American Institutes for Research (AIR), a not-for-profit social
science research organization headquartered in Washington, DC. The project is funded by the Centers for
Medicare & Medicaid Services. Steven Garfinkel, Ph.D. at American Institutes for Research is the Project
Director.
Do I have to participate in this project?
No. It is your choice whether to participate or not. Also, you have the right to stop participating at any
time, and you do not have to answer any questions that you prefer not to answer. If you choose not to
participate or stop participating, there are no penalties and you will receive the full incentive.
What are the risks and benefits?
There are no anticipated or known risks in participating in this project. There are no direct benefits to you
for participating in an interview. By participating in this project, you will receive the opportunity to
provide input on how to best describe results of these types of surveys.
How will you protect my privacy?
With your permission, we will be audio-recording the interview for reference, to ensure accuracy in
capturing what you share with us. The recordings will be destroyed no later than the end of the project
(approximately one year). We will keep your identity and the information you supply confidential and
will not share this information with anyone outside of the project staff.
What if I want more information?
• If you want more information about this project, please contact the director of the research project
at AIR, Steven Garfinkel, sgarfinkel@air.org, (919) 918-2306.
• If you have questions about your rights as a participant, contact the chair of AIR’s
Institutional Review Board, at IRB@air.org or toll-free at 1-800-634-0797 or c/o
AIR, Attn: AIR IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007.
Verbal Consent: Please answer yes or no to each of the following questions.
Do you understand the described project and interview and agree to be a participant as part of
this project?
Do you agree to have the interview recorded?






Do you understand that your name will not be associated with reports or documents related to
this project?



Do you understand that you can withdraw your consent at any time and stop participating in the
interview without any prejudice to you?



Name________________________________

Today’s date________________





Yes
No
Yes
No
Yes
No
Yes
No

CMS-Sponsored Technical Assistance for Improving the Consumer’s Experience
The Health Insurance Marketplace Improvement Guide and Consumer Usability Testing of Marketplace Websites
Issue Brief #2

Purpose
This issue brief provides an overview of two upcoming
technical assistance (TA) activities available to the
Marketplaces, which include:
§ Consumer Usability Testing of the State-Based Marketplace
(SBM) websites.
§ The Health Insurance Marketplace (HIM) Improvement
Guide.
The TA activities are designed to support Marketplaces in
improving their consumer experience. The consumer
experience surveys were authorized by the Affordable Care Act
(ACA). These surveys were developed by the Centers for
Medicare & Medicaid Services (CMS) with support from
American Institutes for Research (AIR). The Qualified Health
Plan (QHP) Enrollee Experience Survey (or Enrollee Satisfaction
Survey) will assess enrollees’ experiences with their qualified
health plans offered on the Marketplace. The Health Insurance
Marketplace Survey (Marketplace Survey) will assess consumers’
perspectives on the services provided by the Marketplaces.
More specifically, the Marketplace Survey will evaluate
consumers’ experiences with the websites, telephone call centers,
and in-person support. The survey results will provide
actionable information that Marketplaces can use to improve
performance. For further information about the surveys, see
http://www.cms.gov/Medicare/Quality-Initiatives-PatientAssessment-Instruments/QualityInitiativesGenInfo/HealthInsurance-Marketplace-Quality-Initiatives.html.
The immediate need from State-Based Marketplaces (SBMs)
is assistance with the TA activities related to the Marketplace
Survey. CMS funded expert usability testing of some of the
SBM websites in 2014 and a summary of the findings will be
provided in a separate brief. In this brief, we describe the TA
activities, how they will benefit the Marketplaces, and how
the SBMs can participate.
We are asking each SBM to indicate whether interested in:
§ Receiving a consumer usability assessment of its
Marketplace website to identify ways to improve the
consumer eligibility and enrollment experience. This
assessment is available at no cost to the SBMs.
§ Participating in an informal advisory group to inform the
development of the HIM Improvement Guide.
If interested in either or both of these activities, please send
an email to Marketplace_Quality@cms.hhs.gov.

June 2014

Consumer Usability Testing of the StateBased Marketplace Websites
What is it?
Marketplace websites are one of the primary ways that
consumers learn of their eligibility for enrolling in QHPs and
public subsidies, access information to compare and choose a
QHP, and enroll in a plan. CMS is funding a usability
assessment of State-Based Marketplace (SBM) websites
during the 2015 open enrollment period.
Usability testing uses the method of cognitive interviewing to
assess how a respondent understands or interprets information.
The usability testing will be used to gather information from
consumers about:
§ Whether the SBM websites provide information about what
consumers want to know;
§ Whether consumers are able to readily find the information
they want (navigation); and
§ Whether consumers interpret the language used and
information as intended (comprehension). 1, 2
Frequently consumers interpret information very differently
than experts expect and in ways that cannot be anticipated. This
is the fundamental premise of all cognitive testing, including
usability testing.

Why is it needed?
The website usability testing will inform the interpretation of
Marketplace Survey results and help the Marketplaces
improve the consumer’s experience.

Who is it for?
Consumer usability testing is available for SBMs that are
operating their own websites, if the SBM indicates interest in
participating. There are no costs to the SBMs for participating

1

2

Nielsen, J., & Loranger, H. (2006). Prioritizing Web usability.
Berkeley, CA: Nielsen Norman Group.
Robert Wood Johnson Foundation and American Institutes for
Research. How to get consumer feedback and input into websites.
Retrieved from http://forces4quality.org/node/2550

Issue Brief #2 | CMS-Sponsored Technical Assistance for Improving Marketplace Survey Scores
in this activity. The results will be used by CMS and the SBMs
for their own internal performance improvement efforts.

§ Strategies for using the Marketplace Survey results to
identify the best opportunities for improvement

When and how will it occur?

§ How to use quality improvement processes to improve
specific consumer experience performance goals

Usability testing will be done during the 2015 openenrollment period. The results are intended to support
ongoing future improvement.
During this testing, we will monitor consumers’ use of the
website with their permission. We will ask each participant to
perform a series of activities on the SBM website and then
ask questions as he/she is performing the activities. For
example, we will present the participant with the SBM Web
page to compare and choose plans. Interviewers would then
ask questions such as: What do you notice on the page? What
would you want to do next? What do you think of when you
see this?

What is needed from the Marketplaces?
§ Indicate interest. SBMs interested in receiving a
consumer usability assessment of their Marketplace
website should send an email indicating their interest to
Marketplace_Quality@cms.hhs.gov.
§ Discuss creating test accounts, when needed. For
SBMs that require a user name and password to access
their websites—especially pages pertaining to eligibility,
enrollment, and comparing and choosing a health plan—we
would like to talk with you about the possibility of obtaining
a test account that could be used by consumers for the
testing.

Health Insurance Marketplace
Improvement Guide
What is it?
The HIM Improvement Guide will provide Marketplaces with
a roadmap on how to use the Marketplace Survey results to
improve the consumer’s experience.
The guide will be modeled after the Consumer Assessment of
Healthcare Providers and Systems (CAHPS) Improvement
Guide.3 The content of the guide will present approaches
to improving processes related to determining eligibility,
comparing and choosing plans, and enrollment. The guide will
describe:
§ Why it is important to improve the consumer experience
§ How to assess a Marketplace’s readiness to improve the
quality of its services
§ How to analyze the Marketplace Survey results
3

For more information visit https://cahps.ahrq.gov/qualityimprovement/improvement-guide/improvement-guide.html.

2

§ Examples of interventions for improving specific aspects of
the consumer’s Marketplace experience
§ Case studies to highlight implementation of some of these
strategies
In order to develop the HIM Improvement Guide, we need
feedback from the Marketplace staff that will be using
the survey data to implement performance improvement
strategies. We would like to form an informal advisory group
to solicit feedback from the people who will be using the guide
in the future.

Why is it needed?
The HIM Improvement Guide will help Marketplace staff
understand their Marketplace Survey results and translate the
results into actionable strategies that they can implement to
improve the consumer experience. Input from Marketplace
staff is critical in order to ensure the HIM Improvement Guide
will meet the needs of its users.

Who is it for?
All Marketplaces—SBMs, State Partnership Marketplaces, and
the Federally Facilitated Marketplace states.

When and how will it occur?
We will hold a series of informal conference calls and
webinars to solicit feedback from Marketplace staff. We
anticipate holding up to three conference calls/webinars from
June 2014 to February 2015. In addition, we will follow up
with individual Marketplace staff via email or conference call,
as needed.

What is needed from the Marketplaces?
§ Indicate interest. Marketplaces interested in informing
the development of the HIM Improvement Guide should
send an email indicating their interest to Marketplace_
Quality@cms.hhs.gov.
– We ask that there be one primary representative and
one alternate representative from each interested
Marketplace.
§ Participate. If you are interested in participating, we ask
that you join the conference calls and webinars. More details
about these activities will be forthcoming.