Direct pay Usability Testing

Direct Pay Usability Test

Winter 2015 Test Materials

Contents
Direct Pay Usability Test...........................................................................1
Direct Pay Usability Test Script ...................................................................2
Welcome and Purpose ...........................................................................2
Test Facilitator’s Role ...........................................................................2
Test Participant’s Role ..........................................................................2
Moderator Checklist ...............................................................................5
Paperwork Reduction Act Statement ............................................................6
Consent & Recording Release Form ..............................................................7
Pretest Demographic and Background Information Questionnaire ..........................9
Single Ease Question..............................................................................10
Recruitment Email ................................................................................11
Confirmation Email ...............................................................................12
Reminder Email....................................................................................13
Receipt Form: Usability Test Compensation ..................................................14
Recruiting Screener ...............................................................................15

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Direct Pay Usability Test Script
Welcome and Purpose
Thank you so much for coming in today. I want to give you a little information about what
you will be looking at and give you time to ask any questions you might have before we get
started.
Today we are asking you to evaluate a website and complete a short task. Our goal is to find
out how easy it is or how difficult it is for you to use the site
Test Facilitator’s Role
I am here to record your reactions to and your comments about the website. In a conference
room nearby, I have a colleague helping me take notes and observe your interactions with the
site as well.
During this session, I would like you to think aloud as you work to complete the tasks. I will
not be able to offer any suggestions or hints, but from time to time, I may ask you to clarify
what you have said or ask you for information on what you were looking for or what you
expect to have happen.
Demonstrate how to think out loud.

Test Participant’s Role
Today I am going to ask you to find some information and then tell me how easy or difficult it
was. There are no right or wrong answers. This is an evaluation of the website, not you! If
you have any questions, comments or feel confused about anything while you are working,
please let me know.
If you ever feel that you lost or that you cannot complete the task, please let me know. I will
ask you what you might do in a real-world setting and then either put you on the right track
or move you on to the next step in the process.
I will provide you with a made up name, address and other credentials to use on the website
today. As you work through the tasks, please try and act as if you are at home or work, but
use the fake credentials in place of your own.
Consent: Request participant read and sign consent/recording release form.
Each session is recorded. We will be recording your face, your voice and what you see on the
screen. Your actual name will not be used or reported with data or findings associated with
this evaluation.
I’ll ask you questions as we go along and we will have wrap up questions at the end.
Do you have any questions before we begin?
IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Pre-Task Questions
Ok, let’s start with a few questions.
First, please fill out this questionnaire. Provide pretest demographic and background

information questionnaire.
Have you ever visited IRS.gov before? If so, can you recall the purpose of your visit?
Which of the following methods have you used to make a payment to the IRS
•
•

Online
By mail

What was your experience like for each?
Thank you. Let’s get started.
Usability Tasks
Today you’ll be using the name, address and credentials of [NAME], a fake persona. Please use
these credentials and pretend you are [NAME] today while you use the website.
Task 1 - Make a payment on an installment agreement.
Your tax preparer completed your tax return for 2013. It showed you owe a $2000 balance to
the IRS. An installment agreement was set up on your behalf. Schedule a payment of $200 for
your installment agreement.
Single Ease Question (on completion of task: Overall, how difficult or easy was it to complete
the sign in process?
If you were to make another payment from this Confirmation screen, how would you go about
doing that?
If you wanted to look up a payment from this screen how would you go about doing that?
Now let’s go ahead and sign out. How would you do that?

Task 2 - Lookup a payment
[Take the user to the Payment Lookup landing page.]
A couple weeks have gone by and you would like to check the status of your $200 payment.
Find out the status of the payment for confirmation #{insert number here}.
Single Ease Question (on task completion, hand the participant SEQ form): Overall, how
difficult or easy was it to complete the sign in process?

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Task 3 - Edit a payment
You discover your estimated tax payment of $500 for 2015 was incorrect. It should be $350.
Change the payment amount for confirmation #{insert number here} to $350.
Single Ease Question (on task completion, hand the participant SEQ form): Overall, how
difficult or easy was it to complete the sign in process?
Task 4 - Cancel a payment
You mistakenly made a duplicate payment. Find the payment for confirmation #{insert number
here} and cancel this payment.
Single Ease Question (on task completion, hand the participant SEQ form): Overall, how
difficult or easy was it to complete the sign in process?
Post-Task Questions
Thanks. We’ve completed the tasks for our session. I have a few more questions for you.
How do you feel about the overall experience you just had?
If you have made a payment online to the IRS in the past, how do you feel that this
experience compared?
Would you be likely to use this application to make payments?
What suggestions do you have for improving the experience?
Ask any follow up questions based on the participants experience with Direct Pay.
That’s it for our session! Do you have any questions or additional comments?

Post Test Check List:
□ Incentive
□ Sign Incentive Receipt
□ Close and save Morae recording
□ ID number on notes, consent forms, receipt

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Moderator Checklist
1. Look at and remember person’s name prior to going to meet him/her.
2. Introduce yourself and other team members who might interact with participant.
3. Ask if person would like something to drink and/or to use the restroom.
4. Ask person to read and sign the consent form.
5. Reference the OMB Control Number and PRA Statement (provide printed copy):
The Paperwork Reduction Act requires that the IRS display an OMB control number on
all public information requests. The OMB Control Number for this study is 1545-2256.
Also, if you have any comments regarding the time estimates associated with this
study or suggestions on making this process simpler, please write to:
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224
6. Review the purpose of the session.
7. Ask participants if they have any questions.
8. Ensure that participant completes post-study questionnaire and discuss his/her
responses with him/her. Ask if (s)he has any other comments about the site.
9. Have participant sign receipt form and record honorarium check number on signed
form.

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Paperwork Reduction Act Statement

The Paperwork Reduction Act requires that the IRS display an OMB control number on
all public information requests. The OMB Control Number for this study is 1545-2256.
Also, if you have any comments regarding the time estimates associated with this
study or suggestions on making this process simpler, please write to:
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Consent & Recording Release Form
I agree to participate in the study conducted and recorded by the Internal Revenue
Service (IRS). I understand that the study will take approximately 40 minutes.
I understand and consent to the use and release of the audio, video, and/or digital
recording by the IRS. I understand that the information and recording is for research
purposes only and that my name and image will not be used for any other purpose. I
relinquish any rights to the recording and understand the recording may be copied
and used by IRS without further permission.
I understand that participation in this usability study is voluntary and I agree to
immediately raise any concerns or areas of discomfort during the session with the
study administrator.
The Paperwork Reduction Act requires that the IRS display an OMB control number on
all public information requests. The OMB Control Number for this study is 1545-2256.
Also, if you have any comments regarding the time estimates associated with this
study or suggestions on making this process simpler, please write to:
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224
Please sign below to indicate that you have read and you understand the information
on this form and that any questions you might have about the session have been
answered.

Date
Please print your
name
Please sign your
name

Thank you! We appreciate your participation.
IRS Direct Pay Usability Study | OMB Control Number 1545-2256

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Pretest Demographic and Background Information
Questionnaire
1. What is the primary language spoken at home? (Please select one)
□
□
□
□

English
Spanish
Chinese
Vietnamese

□ Korean
□ Russian
□ Other (please specify):________

2. Do you have a computer at home?
□ Yes
□ No
3 . What payment option do you use the MOST to pay your bills?
□ Mail in payments
□ In person
□ Over the phone
□ Through the Internet via scheduled electronic bill pay
□ Through the Internet via same day bill pay
4. Have you made electronic payments to the IRS in the past?

□ Yes
□ No
5. Do you own and use a mobile phone?
□ Yes, regular mobile phone
□ Yes, Smartphone (mobile phone with applications and internet access)
□ No


IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Single Ease Question
Task One
Overall, how difficult or easy was it to complete the login process?
Very
Difficult

Very
Easy

1

2

3

4

5

6

7

□

□

□

□

□

□

□

Task Two
Overall, how difficult or easy was it to complete the login process?
Very
Difficult

Very
Easy

1

2

3

4

5

6

7

□

□

□

□

□

□

□

Task Three
Overall, how difficult or easy was it to complete the login process?
Very
Difficult

Very
Easy

1

2

3

4

5

6

7

□

□

□

□

□

□

□

Task Four
Overall, how difficult or easy was it to complete the login process?
Very
Difficult

Very
Easy

1

2

3

4

5

6

7

□

□

□

□

□

□

□

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Recruitment Email
When possible, we will recruit using listserves or known pools of participants.
SUBJECT LINE: The IRS invites you to participate in a website study and earn $40!
Hello,
My name is [NAME], and I’m helping to run a usability study for the Internal Revenue
Service (IRS). In an effort to improve services, we’re looking for people who are
interested in exploring new ways to access their tax records. Afterward we will
simply ask for your feedback. If you participate we will pay you $40.
What will you be doing in a usability study?
You will be asked to perform several short tasks using a website. You will also be
asked questions about your experience and what you think of the website.
How long is a study session?
40 minutes.
When and where?
The study will be held [DAYS, DATES, LOCATION]. You will need to participate in
person.
Interested in participating?
Please reply to this email with your contact information or call me at [ADD PHONE]. I
will give you a call to ask you some questions so we can decide if you qualify for the
study.
If you have any questions, please contact me at [EMAIL].
Thank you for your interest,
[NAME AND TITLE]

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Confirmation Email
NOTE: This will be sent as an email or in an Outlook meeting notice.
SUBJECT LINE: Confirmation: Your participation in our usability study
Dear [PARTICIPANT NAME]:
Thank you for agreeing to participate to test the Internal Revenue Service website. As
I mentioned, you will be asked to perform a few tasks on the website and give us
thoughts about your experience. There is nothing to prepare before the session.
You are scheduled to participate as follows:
DATE: [DAY, DATE]
TIME: [TIME]
PLACE: [ADDRESS, LINK TO MAP]
A few key reminders:
•
•
•

You will be given $40 in exchange for your participation. (NOTE: Government
employees and/or grantees are not eligible for the honorarium unless you are
participating on your own time.)
During the study, we will ask you to complete some tasks using the website.
You’ll talk out loud as you work so the facilitator can follow along.
With your permission, the session will be recorded. We will only use the
recording in our research to decide how to improve the website. Your name will
not be used for any purpose beyond this session.

Also, we have only one person scheduled at a time for these sessions so if you find
that you cannot participate on your scheduled day, please contact me as soon as
possible so we can reschedule your session.
Thank you again!
[NAME OF FACILITATOR AND CONTACT INFORMATION]

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Reminder Email
SUBJECT LINE: Reminder: Website study tomorrow
Dear [PARTICIPANT NAME]:
Thanks again for agreeing to help us out with testing the Internal Revenue Service
website. We are looking forward to talking with you.
You are scheduled to participate as follows:
DATE: [DAY, DATE]
TIME: [TIME]
PLACE: [ADDRESS, LINK TO MAP]
A few key reminders:
•
•
•

You will be given $40 in exchange for your participation. (NOTE: Government
employees and/or grantees are not eligible for the honorarium unless you are
participating on your own time.)
During the study, we will ask you to complete some tasks using the website.
You’ll talk out loud as you work so the facilitator can follow along.
With your permission, the session will be recorded. We will only use the
recording to decide how to improve the website. Your name will not be used
for any purpose beyond this session.

Also, we have only one person scheduled at a time for these sessions so if you find
that you cannot participate on your scheduled day, please contact me as soon as
possible so I can reschedule your session.
Thanks again!
[NAME OF FACILITATOR AND CONTACT INFORMATION]

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Receipt Form: Usability Test Compensation
Please sign below to indicate that you have received the promised compensation for
your participation in testing today.

Date
Amount received
Please print your
name
Please sign your
name

Thank you!
We appreciate your participation.

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Recruiting Screener
Hello, I’m [NAME] from [COMPANY] calling on behalf of the Internal Revenue Service.
The IRS wants to hear from you about how we are doing. Have you heard of our
website IRS DOT GOV? What we would like to do is ask you a few questions to see if
you qualify to participate in a 40 minute study we are conducting of our web site. If
you do qualify, we’d like to schedule you as a participant during the week of [DATE].
We are very interested in getting taxpayers opinions on the design of our website.
These evaluations will be used for research purposes only and any personal
information you provide will not be used for anything other than this particular study.
We will pay you to come to a testing facility and give us feedback on how we are
doing. Are you be willing to answer a few questions to see if you qualify?
IF NO, Can I schedule a time to call you back at a more convenient time? IF NO,
terminate. IF YES, schedule a time to call back.
IF YES, CONTINUE.
[Reference the OMB Control Number and PRA Statement as noted below before
continuing with the interview]
The Paperwork Reduction Act requires that the IRS display an OMB control number on
all public information requests. The OMB Control Number for this study is 1545-2256.
Also, if you have any comments regarding the time estimates associated with this
study or suggestions on making this process simpler, please write to us at:
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224
Because we are seeking a wide variety of people, let me start by asking you some
questions about yourself.
1. Have you ever taken part in a market research interview or usability study, either
one on one or in a focus group type setting? IF NO SKIP TO Q4
2. Do you or any member of your immediate family currently work for or did they
ever work for any of the following?
Marketing, Market Research Firm

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

Advertising
Web Site Design Company
IF YES TO ANY, TERMINATE
For a city, state or federal government
agency? If so, which one?
IF IRS, TERMINATE

3. Do you have access to the internet where you can use it for your own personal use? 

YES

NO TERMINATE
4. On a scale of 1 to 5 where 1 means ‘not at all comfortable’ and 5 means ‘very
comfortable’, how comfortable are you with…
TERMINATE IF 1 OR 2
using a computer mouse

1

2

3

4

5

using a computer
keyboard

1

2

3

4

5

using the Internet

1

2

3

4

5

using a scroll bar

1

2

3

4

5

5. Do you use the Internet to do any of the following?
e-commerce (shop online)

YES

NO

Surf the web

YES

NO

pay bills online

YES

NO

work related reasons

YES

NO

research

YES

NO

online banking

YES

NO

pay taxes online/software

YES

NO

TERMINATE IF REPLY NO TO ALL
6. Are you employed? IF NOT EMPLOYED SKIP TO 8A

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

If employed:
What is your occupation?
What industry are you
in?
GET A GOOD MIX OF DIFFERENT OCCUPATIONS AND INDUSTRIES
7a. If Not Employed:
Retired
Student
Homemaker
In between jobs
GET AT LEAST 1 NOT EMPLOYED. 


7. What is the highest level of education you have completed?
DO NOT READ ANSWER CHOICES - GET A MIX
Some high school or less
High school graduate
Some college, vocational or trade school
College graduate
Graduate school
8. Which age group from the following list do you belong?
GET A MIX - NO MORE THAN THREE PARTICIPANTS PER AGE GROUP
Under 18

TERMINATE

18-29

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

30-39
40-49
50-59
60-65
66+

9. Have you ever visited the website IRS DOT GOV?
YES
NO

SKIP TO Q17

10.What was the nature of your visit to the website?
to find forms
to review tax regulations
to find phone numbers
file downloads
online tax tools
other
11.Do you remember if the visit to the site was:
less than 15 minutes
15 minutes to 30 minutes
30 minutes to 1 hour
longer than 1 hour
Don’t remember

12. Approximately how many times have you visited the website?
13.What ways have you ever contacted the IRS?

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

visited a local branch
phone call
spoken with a specific representative
other
Thank you, now, the following questions are for classification purposes only.
14.Please indicate your most recent tax year filing status? RECRUIT A MIX
Single
Married filed joint return
Married filing separate return
Head of household
Qualifying widower with dependent child

RECRUIT TO OBTAIN 30 INDIVIDUALS
RECRUIT A MIX OF MEN AND WOMEN
RECRUIT 50% OF INDIVIDUALS IN ‘MARRIED FILED JOINT RETURN’ FILING STATUS
Thank you for answering my questions. Based on your responses, you appear to qualify
for participation in the study. Are you interested in helping by participating in a paid
research project?
No
Yes

POLITELY DISCONTINUE
CONTINUE

The IRS will pay you $40, to test a portion of our web site and share your opinion with
us in a 40 minute evaluation.
This is how the process works: As part of our research, we are conducting study
sessions with a number of people from your area. If you choose to participate we will
ask you to attempt to do specific tasks on a web site and then discuss your
experience. We are evaluating our web site so we can better serve taxpayers. We will
simply observe your experience and ask your opinions. The IRS will not use any
personal information for anything other than this particular study. These evaluation
sessions will be held on [DATE, TIME AT LOCATION]. They will last approximately 40
minutes. Would you be available for that particular date and time?

IRS Direct Pay Usability Study | OMB Control Number 1545-2256

No
Yes

POLITELY DISCONTINUE
CONTINUE

Do you have any questions up to this point or special needs for coming to our facility?
[Answer inquiries appropriately and thoroughly.]
Again, let me thank you for your decision to participate. This research is very
important to us, and your help is highly appreciated.
Now, for the rest of the steps in the process:
After I have gathered your information, we will send you a confirmation email with a
map to the testing facility. You will also receive a reminder email prior to the
evaluation session. In a moment, I will need your name and email address. I’d also
like to get a phone number where we can reach you.
We will send you a reminder email prior to your scheduled appointment on [DATE,
TIME]. We schedule one person at a time for these sessions so if you find that you
cannot participate on your scheduled day, please give notice to [NAME, EMAIL
ADDRESS, PHONE NUMBER] as soon as possible so we can reschedule your session.
Thank you. If you have any questions, please email [NAME] at [EMAIL].

IRS Direct Pay Usability Study | OMB Control Number 1545-2256