ETAP User Feedback Interviews and Protocol

Usability testing plans and
protocols

97

Plans for Usability Testing of the ETAP System
This document describes the draft plans for conducting usability testing of the Education and Training
Application (ETAP) system.

A. Background and Usability Testing Goals
Mathematica has developed an electronic data system for the National Science Foundation (NSF) to
support applications to NSF-funded training opportunities. Although usability testing has already been
conducted on the processor system, the ETAP is expected to undergo significant enhancements and
revisions during its pilot, making it important to conduct additional user testing. The primary goal of the
usability testing the ETAP system is to ensure that the system is intuitive and easy for users to use. This
includes a) ensuring that instructions and items are clear, b) determining if testers can complete assigned
tasks, c) identifying changes needed to improve user performance and satisfaction, and d) ensuring there
are no bugs or defects. The secondary goal of the testing is to determine the amount of time testers take
to complete tasks (and verify predicted burden estimates).
Mathematica will conduct usability testing at multiple times throughout the pilot (coinciding with
development sprints involvement significant changes) to inform further development and revisions.

B. Description of Test Participants
ETAP system supports a common registration and a common application. Testers are largely
college/university applicants and current or former principal investigators (PI). Specifically:
• Applicants testing the common registration will be asked to create an account in ETAP and apply
to an opportunity that only requires the submission of basic demographic and contact information.
• Applicants testing the common registration and common application will first navigate through
the common registration as described in the prior bullet, and then proceed to the common
application through which they will submit additional information required by training opportunities
using the ETAP common application, such as a personal statement and contact information for
reference writers.
• PIs testing the common registration or the common application will create an account and
choose whether they would like to run competitive (common application) or non-competitive
(common registration) applications for their award, and will be ask to review applications, and
record application decisions (whether student was admitted and accepted an offer to participate)
and the participation status of admitted applicants (whether they actually participated at their Site
or not).

C.

Roles

Each testing session will involve a facilitator, a test participant, and when possible, a test observer to assist
with note taking. They will perform the following functions during usability testing:
Facilitator
98

• Explains the purpose of usability testing to participants
• Prompts testers for their input and encourages them to think out loud.
• Responds to participant's requests for assistance before, during, and after testing
• Assists in debriefing
Test Participant
Attempts to complete a set of tasks presented to them in as efficient and timely a manner as
possible (these tasks have been bundled strategically to ensure that testers do not end up
testing the same elements of the system and systematically skipping other elements)
• Provides feedback regarding the usability of features and functions. The participant will be
encouraged to provide honest opinions regarding the usability of the application, during and
after the test.
Test Observer
•

•
•
•

D.

Silent observer
Serves as note taker
Participates in internal team debriefs to identify critical action items

Procedure

Each usability test will be scheduled for 2 hours (although it may take less than 2 hours) and will include
one test participant per session. The testing will be conducted remotely using the WebEx platform and
test sessions will be recorded. Test participants will be sent an email invitation with WebEx information.
They will be asked to install a WebEx plug-in in advance of the testing session.
The participant will be seated at their workstation in their home environment. Testers will share their
screens with the facilitator and observers so that testers’ actions can be observed. Verbal communication
will be supported via WebEx.
The facilitator will brief the participant on the purpose of the website, and the goals of the usability test.
The facilitator will emphasize the fact that the participant should evaluate the website, and that the
facilitator will not evaluate or judge the participant’s assessments or opinions. Section J includes the draft
scripts to be used for applicant testing and PI testing.
Testing will begin when the tester opens the ETAP website at the home page. The facilitator will instruct
the participant to perform the first task. The facilitator will encourage the participants to ‘think out loud’.
The facilitator and test observers will observe and enter user behavior and comments.
Testers will be directed to experience different elements of the system—say apply to a named
opportunity, or search for possible opportunity that might of interest to someone with named interests or
characteristics—based on scenarios created to ensure that all system functionalities are tested by
intended users of the system. These scenarios are described in Section E below and reflected in scripts
that the facilitator will use to guide testers (available in Section J).
Facilitators will assist testers in navigating around bugs should they appear during testing.

99

E. Test Scenarios
Applicants
Common testing elements
All testers will:
1. Create an account
2. Apply to at least one training opportunity
3. Be given transcript and resume in pdf format
Variable testing elements: key log-in functionalities
Half of the testers will be sorted into one of two test conditions:
1. Forgot Password (PW)
2. Complete partial record, log out, and log back in
Variable testing elements: system experiences
Testers will test different combinations of the following:
1. Apply to a non-competitive opportunity only (to ensure users is not confused with common
application elements that are not applicable to them)
2. Apply to two opportunities: one competitive and one non-competitive (to ensure users
understand that different opportunities have different requirements)
3. Apply to an opportunity through a direct opportunity link provided as opposed to searching for it
(to ensure alternative functionality is intuitive for users)
Principal Investigators
Common testing elements
All tester will:
1. Be given two accounts:
a. One account that requires them to register and set up their award.
b. One account that will already be set up. It will permit them to accept students into their
programs and test the reporting functionality.
2. Enter data:
a. Typing data through the interface
b. Uploading bulk data
3. Filter applicants and download data
4. Authorizing or de-authorizing designees who assist in managing applications and other
administrative functions
Variable testing elements: key log-in functionalities
Not applicable if system is using Single Sign On with research.gov.
Variable testing elements: system experiences
1. Choose to run a competitive application
2. Choose to run a non-competitive application

100

F.

Test Data
Testers will be instructed not to use their or other individuals’ names, addresses, and phone numbers
to protect their PII. However, they will be asked to provide honest responses to all other questions to
help flush out real-life situations not considered in the design of the system.
1.

2.

3.

4.
5.

G.

Student testers will be given the following prior to testing:
a. Names, addresses, and phone numbers
b. Resume
c. Transcript
d. References
e. Personal statement
f. Answers to questions
PI Testers will be given the following prior to testing:
a. Names, addresses, and phone numbers
b. Additional materials pdf file
c. Additional questions
Site data will be added before the student testing begins so that they have opportunities from
which to choose. All website information will be valid sounding (that is to say, not “bad” test data)
to avoid confusing the tester.
a. Name of opportunity
b. Location
c. Discipline and research topics
d. Program Dates
e. Application Time Period
f. Additional Questions
Student data will be added before the PI testing begins so that they have students for which to
view applications and track statuses.
Authorized designees will be added before the PI testing begins so that they have that
information to view and de-authorize if the assigned testing scenario requires.

Analysis of Usability Testing

Information gathered through testing will be analyzed to inform system revisions along the following
dimensions:
o

o
o
o

Efficiency – Did testers find it easy to complete the task? This is measured by time on task (time the
person begins the scenario to the time he/she signals completion), verbal remarks during testing,
and comments provided during debriefing sessions with testers.)
Success Rate – Could the testers complete the task? This is measured as the percentage of test
participants who successfully complete the task without critical errors.
Accuracy – Did testers complete the task as intended? This is assessed by whether tester correctly
interpreted an item or instruction.
Satisfaction – Were testers satisfied with their website experience? This is captured by users
reported impressions, and observation of user struggles/success.
101

H.

Incentives
Applicant testers will be offered an Amazon gift card of $40 for their participation. PIs will not receive
a gift for testing the system as they are beneficiaries of NSF funding.

I.

Protocols
ETAP System Usability Testing
Applicant Protocol
(1-2 Hours)

INTRODUCTORY REMARKS
Hi, my name is [NAME]. [My colleague, [NAME], is also here today observing]. [I/We] work for
Mathematica. Mathematica is helping design the data system we’d like you to test today for the National
Science Foundation (NSF). [I am/We are] going to be walking you through this session today.
Before we begin, I have some information for you, and I’m going to read it to make sure that I cover
everything.

Background Information
I want to give you some background information so you have an idea of why you’re here and what you’ll
be doing.
The NSF supports the education and training of individuals at different stages of their careers, as
undergraduate students, graduate students, post-docs, teachers, early career faculty, and so on, through a
wide range of program offerings. These training opportunities are funded by NSF but provided by faculty
or researchers in colleges, universities, and in other organizations such as research centers. NSF has asked
Mathematica to build a website about these training opportunities.

Purpose
The purpose of this session is to pretend that you are a student/teacher who is interested in applying to a
training opportunity funded by NSF. The session should take about one to two hours. The first thing I
want to make clear right away is that we’re testing the website, not you. You can’t do anything wrong
here. In fact, this is probably the one place today where you don’t have to worry about making mistakes.
As you go through the website, keep in mind our focus is not on your answers to the questions – we are
most interested in what you think about how easy the questions are to understand, and your overall
experience in the system.
As you use the website, I’m going to ask you as much as possible to try to think out loud: to say what
you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to us.
Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to improve the website,
so we need to hear your honest reactions.
If you have any questions as we go along, just ask them. I may not be able to answer them right away,
since we’re interested in how people do when they don’t have someone sitting next to them to help. But if
102

you still have any questions when we’re done, I’ll try to answer them then. And if you need to take a break
at any point, just let me know.
After the test, I’m going to ask for any additional feedback and thoughts on the website. We’ll go over
questions and get your feedback once you’re done.

Permission to record WebEx
If it’s okay with you, I’d like to record this call in case I need to refer back to it later. Do I have your
permission to record this meeting?
[If yes]: With your permission, we’re going to record what happens on the screen and our conversation.
The recording will only be used to help us figure out how to improve the site, and it won’t be seen by
anyone except the people working on this project. And it helps me because I don’t have to take as many
notes. The meeting recording and any information you save in the website will not be shared with anyone
outside the research team. Before we start the recording, please close all other windows on your
computer. We will need you to access your email, however, during the testing. You may want to keep it
open for the testing.

Hit the RECORD button on WebEx.
Do you have any questions so far?

INSTRUCTIONS
Ok, great. We can start looking at things.
Earlier today, I sent you an email with the website link. Can you please go to your email and open the
website?
Once you have the website loaded, can you now share your screen through the WebEx?

 If tester has trouble sharing screen, refer to page 37 of
the webex manual .
Great, I can now see your screen. [Jump to task-specific scenarios]

TASK-SPECIFIC SCENARIOS
[See task-specific script file.]

DEBRIEFING AFTER THE TEST
Questions for Facilitator to ask at the end if applicable:
-

Are there specific sections or pages that were confusing or hard to navigate?
103

-

Were there specific sections or pages where you had to guess how to do something?
Were there any questions in the section that you found confusing or were unable to answer?
Were there specific sections or pages that you were not sure how to proceed?
What suggestions do you have about how we might make this website better for other students to
complete? What would make it easier?

End remarks

Do you have any questions for me, now that we’re done?
Thank you for helping out with the testing! We really appreciate it! As a token of our appreciation, we
will send you a $40 amazon gift card over email to compensate for your time.

 Stop the screen recorder and save the file.
 Thank them and ask them to contact you if they have
any follow up questions.
 Take time before the next session to jot down a few
notes about things you observed.
THANK YOU! Please contact me if you have any follow up questions.

104

ETAP System Usability Testing
PI Protocol
(1 Hour)

INTRODUCTORY REMARKS
Hi, my name is [NAME]. [My colleague, [NAME], is also here today observing]. [I/We] work for
Mathematica. Mathematica is helping design the data system we’d like you to test today for the National
Science Foundation (NSF). [I am/We are] going to be walking you through this session today.
Before we begin, I have some information for you, and I’m going to read it to make sure that I cover
everything.

Background Information
I want to give you some background information so you have an idea of why you’re here and what you’ll
be doing.
As you may know, the NSF supports the education and training of individuals at different stages of their
careers, as undergraduate students, graduate students, post-docs, teachers, early career faculty, and so
on, through a wide range of program offerings. These training opportunities are funded by NSF but
provided by Principal Investigators like yourself working in colleges, universities, and in other
organizations such as research centers.
To be considered for these training opportunities, students must apply. Currently, students apply directly
to the research site (for example, a particular lab in a college or university). But NSF has asked
Mathematica to build a website that would provide a centralized location of information on training
opportunities funded by NSF in part to make it easier for applicants to find and apply to opportunities of
interest to them.
The system will enable principal investigators (PIs) to choose whether applications will be competitive or
non-competitive and to access information provided by students. PIs can also report to NSF who
participated in the award and obtain aggregate statistics on applicants (if applicable) and participants.
The goal of the system is to take advantage of a natural moment when applicants/participants provide
data that NSF needs for reporting and monitoring purposes. NSF seeks to collect those data and offer
access to PIs. This will hopefully reduce burden on PIs (to collect data for admissions purposes and to
submit data to NSF) and on NSF (to obtain information from PIs and participants).

Purpose
We are interested in understanding how the ETAP website works for PIs. The purpose of this session is to
pretend that you heard about the website and want to use it for an NSF award you have. The session
should take about one hour. The first thing I want to make clear right away is that we’re testing the
website, not you. You can’t do anything wrong here. As you go through the website, please keep in mind
our focus is not on the data you are entering – we are most interested in what you think about how the
website works and your overall experience using the system.
As you use the website, I’m going to ask you to try to think out loud as much as possible: to say what
you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to us.

105

Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to improve the site, so we
need to hear your honest reactions.
If you have any questions as we go along, just ask them. I may not be able to answer them right away,
since we’re interested in how people do when they don’t have someone sitting next to them to help. But if
you still have any questions when we’re done, I’ll try to answer them then. And if you need to take a break
at any point, just let me know.
After the test, I’m going to ask for any additional feedback and thoughts on the website. We’ll go over
questions and get your feedback once you’re done.

Permission to record WebEx
If it’s okay with you, I’d like to record this call in case I need to refer back to it later. Do I have your
permission to record this meeting?
[If yes:]

 Hit the RECORD button on WebEx.
Thank you. With your permission, we’re starting to record what happens on the screen and our
conversation. The recording will only be used to help us figure out how to improve the website, and it
won’t be seen by anyone except the people working on this project. And it helps me because I don’t have
to take as many notes. The meeting recording and any information you save in the website will not be
shared with anyone outside the research team. Before we start the recording, please close all other
windows on your computer. We will need you to access your email, however, during the testing. You may
want to keep it open for the duration of the testing.
[If no:] We understand and appreciate your participation. We will be sure to take good notes.
Do you have any questions so far?

INSTRUCTIONS
OK, great. Let’s start looking at things.
Earlier today, I sent you an email with the website link. Can you please go to your email and open the
website?
Once you have the website loaded, can you now share your screen through the WebEx?

 If tester has trouble sharing screen, refer to page 37 of
the webex manual .
Great, I can now see your screen. [Jump to task-specific scenarios]

106

TASK-SPECIFIC SCENARIOS
[See task-specific script file.]

DEBRIEFING AFTER THE TEST
Questions for Facilitator to ask at the end if applicable:
-

Are there specific sections or pages that were confusing or hard to navigate?
Were there specific sections or pages where you had to guess how to do something?
Were there any questions that you found confusing or were unable to answer?
Were there specific sections or pages in which you were not sure how to proceed?
What suggestions do you have about how we might make this website better for other PIs to use?
What would make it easier?

End remarks
Do you have any questions for me, now that we’re done?
Thank you for helping out with the testing! We really appreciate it!

 Stop the screen recorder and save the file.
 Thank them and ask them to contact you if they have
any follow up questions.
 Take time before the next session to jot down a few
notes about things you observed.
THANK YOU! Please contact me if you have any follow up questions.

107

ETAP System Usability Testing
Task-Specific Scripts for Applicants
1.

HOME PAGE
[Give student URL to the home page.]
First, I’m going to ask you to look at this ETAP home page and tell me what you make of it:
o What are your thoughts about what you’re seeing?
o What can you do here?
o What is this website for?
Just look around and do a little narrative. You can scroll if you want to, but don’t click on anything
yet.
o

Where do you feel like clicking next and why?

Allow this to continue for three or four minutes, at most.
Now I’m going to ask you to try doing some specific tasks. As you navigate through each page,
I'm going to ask you about your initial thoughts about each page. And again, as much as possible,
it will help us if you can try to think out loud as you go along.

 Read the participant the first scenario
 Allow the user to proceed until you don’t feel like it’s
producing any value or the user becomes frustrated.
 Repeat for each task or until time runs out.
2. APPLICANTS PAGE
Go to the applicant tab
o What are your thoughts about what you’re seeing?
o What is the text telling you?

 Since this is more of an informational page, have the
tester read the text and decide what to do next.

79

o
o

How would you go about searching for opportunities?
What about the Show Filters button? What do you think it does?

Search for opportunities
Can you pretend that you are [undergraduate student/graduate student/teacher] interested in
applying to an award funded by the [name program]? Use the filter to narrow the list of
opportunities interesting to you.
o Are the results what you expected?
o What do you think of the filters?
o Is there anything you would change about it?
3. ACCOUNT CREATION PAGE
Test alternative ways of navigating to the account creation page:
o
o
o

Now I am going to email you a link to an opportunity and ask you to apply to it. (Send email)
Can you apply to the opportunity [name] you are seeing in the search list?
Now let’s create an account. Think out loud as you navigate.

Now, fill out the information needed. Please think out loud as you complete the information.

 When user gets to enter personal info, prompt:
Remember to use a fake name but use your real email
address.
 Remind the tester to think out loud.
Probing questions to ask:
o Why did you click on the links that you did?
4. DASHBOARD
o
o

What are your thoughts about what you’re seeing?
What would you do next?

 Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
109

5. REGISTRATION PAGES
Now your goal is to apply to [opportunity name].
o What are your thoughts about what you had to complete on this page?

 Since these are several pages to fill out, have the tester
read the items, complete the form and decide what to
do next.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend be changed?
 Is it clear what format to enter the phone number?
 How difficult was it to get to this [section name] page?
 How difficult was it to fill out the Personal
Information?
 Why did you click Save/Save and Continue?

6. ON CURRENT ENROLLMENT INFORMATION PAGE
[This is for both registration and common application students]
o What are your thoughts about what you’re seeing?
o What do you think you need to do?

 Remind the tester to think out loud.
 Confirm that the tester will test one of the following
options:






a. Name tab
b. Geographical tab
c. Tester’s choice
d. “I cannot find my institution on the list”
e. “I am not currently enrolled at a college or university”

a. Choose University – Name (For students who will be testing University = Name)
110

Have student complete application and when they need to choose their university, have
them use the Name tab search function to find it.
Now we are going to find your college or university. On this screen, use the Name tab search
function to find it.

 Remind the tester to think out loud
Probing questions to ask:
 What are your thoughts about entering the Name of
the college or university?
 How difficult was it to find your college or university?
 Did you get the results you expected?
b. Choose University – Geographical (For students who will be testing University =
Geographical)
Have student complete application and when they need to choose their university, have
them use the Geographical function to find it.
Now we are going to find your college or university. On this screen, use the Geographical tab
search function to find it.

 Remind the tester to think out loud.
Probing questions to ask:
 What are your thoughts about using the
Geographical tab to find your college or
university?
 How difficult was it to find your college or
university?
 Did you get the results you expected?
c. Choose University – Tester’s choice (For students who will be testing University =
Tester’s Choice)
Have student complete application and when they need to choose their university, have
them use any function of their choice to find the college or university.
Now we are going to find your college or university.
111

 Remind the tester to think out loud.
Probing questions to ask:
 How difficult was it to find your college or
university?
 Did you get the results you expected?
d. Choose University – I cannot find ... (For students who will be testing University = “I
cannot find my institution on the list”)
Have student complete application and when they need to choose their university, have
them check “I cannot find my institution on the list.”
Now pretend that you are unable to find your institution. How would you proceed?
e. Choose University – I am not currently enrolled ... (For students who will be testing
University = “I am not currently enrolled at a college or university”)
Have student complete application and when they need to choose their university, have
them check “I am not currently enrolled at a college or university”
Now pretend that you are not currently enrolled at a college or university. How would you
proceed?
Once students complete the search, have students continue completing the remaining
information on the Current Enrollment Page.
Continue to complete the remaining information on this page.

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 How difficult was it to fill out the Current Enrollment
page?
 Is there anything you would recommend to change?
7. MY APPLICATIONS PAGES
o What do you think you need to do?

112

Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
8. ADDITIONAL INFORMATION [This is for applying to a competitive application]
Next, you are going to fill out additional information required to apply to this opportunity.
 What are your thoughts about what you’re seeing?
 What do you think you need to do?
Participation in this program [This will differ by tester]
a. Have you participated in this program before = YES
OK, now we are going to pretend that you have previously applied to an REU program. Go
ahead and complete the information requested.
b. Have you participated in this program before = NO
OK, now we are going to pretend that you have not previously applied to an REU program.
Go ahead and complete the information requested.
Prior Research Experience [This will differ by tester]
a. HAVE YOU HAD ANY OTHER PRIOR RESEARCH EXPERIENCES OUTSIDE OF THE NSF
REU PROGRAM?= YES
OK, now we are going to pretend that you have prior research experience outside of the
REU program. Go ahead and complete the information requested.
b. HAVE YOU HAD ANY OTHER PRIOR RESEARCH EXPERIENCES OUTSIDE OF THE NSF
REU PROGRAM?= NO
OK, now we are going to pretend that you do not have prior research experience outside of
the REU program. Go ahead and complete the information requested.
9. APPLICATION MATERIALS
[This will be for common application testers only]
o What are your thoughts about what you’re seeing?
o What do you think you need to do?
Personal Statement

 Remind the tester to think out loud.
Probing questions to ask:
 How difficult was it to enter your personal statement?

113

Supporting Documents
Now let’s upload your application materials. In the email that I sent you earlier today, I attached a
transcript, personal statement and CV/resume. Can you download those application materials to
your desktop? Then once you have those documents downloaded, can you go back to the website
and upload the required documents?

 Remind the tester to think out loud.
Probing questions to ask:
 What are your thoughts about this section?
 How difficult was it to upload your supporting
documents?
 What do you think about how this section works?
References
[This will be for common application testers only]
Great! Now let’s add your references.

 Remind the tester to think out loud.
 Prompt tester to enter in the fake reference that we
have created for them (using dummy data
spreadsheet)
Probing questions to ask:
o What are your thoughts about this section?
o What do you think the Send Email Request button
does?
o [After they click the button] Did you get the result you
expected?
Additional Materials: upload materials [This will be for common application testers only]
The REU site that you applied to requires you to upload additional materials. Go ahead and
complete the information requested.
114

 Remind the tester to think out loud.
Probing questions to ask:
 What are your thoughts about this section?
 How difficult was it to upload additional materials to
the application?
Additional Questions: enter response into space provided [This will be for common
application testers only who applied to]
The opportunity that you want to apply to requires you to answer additional questions. Go ahead
and complete the information requested.

 Remind the tester to think out loud.
Probing questions to ask:
 What are your thoughts about this section?
 How difficult was it to add the answers to the
questions?
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
10. CERTIFY AND SUBMIT
[For both registration and common application]
Answer to overall thoughts will depend on whether errors exist in the
registration/application.
o
o

What are your thoughts about what you’re seeing?
What do you think you need to do?

If there were errors and the tester went to fix them but came back to this page and saw the
message that there are still errors, ask:
o
o

Is what you are seeing what you expected?
What do you think needs to be done?

115

If there were errors and the tester went to fix them and came back to this page and saw the
certify statement, ask:
o
o

Is what you are seeing what you expected?
What do you think needs to be done?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?


Is there anything you would recommend to change?

You are almost done!
If student clicks on “Review your information”, ask:
o

Is what you are seeing what you expected?

Review the certification statement and click the submit button to complete your application.
Application Status
Congratulations! You just completed an application to an NSF-funded opportunity!
Now that you have completed your application, what are your thoughts on this page?
This now concludes our testing. [Go to Debriefing after the Test section].
11. SELECT OPPORTUNITY
[This will differ by tester]
Now let’s pretend you want to apply to another opportunity. Go ahead and select another
opportunity.
o

What else do you think you need to do to apply to this new opportunity?

 Remind the tester to think out loud.
Probing questions to ask:
 How difficult was it to find an opportunity?
 How difficult was it to select your opportunity?
116

 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?

12. FORGOT Password (For only some testers)
OK, so at this point, we’re going to pretend that you need to step away from your computer. So go
ahead and log out. [Be sure tester logs out okay.] You have returned to finish the application,
but you’ve forgotten your password. How do you continue and finish your application?

 Remind the tester to think out loud.
 Probing questions to ask:
 How difficult was it to re-enter the website?
 Is there anything you would recommend to change?
13. COMPLETE PARTIAL RECORD, LOG OUT, LOG BACK IN, CONTINUE (For only some testers)
Have student complete a couple screens first.
OK, so at this point, we’re going to pretend that you need to step away from your computer. Make a
note of your email and password. Then go ahead and log out. [Be sure tester logs out okay.] You
have returned to finish the application. Go ahead and continue your application.
Once they log back in, ask:
o
o

What are your thoughts about what you’re seeing?
What do you think you need to do?

 Remind the tester to think out loud.
Probing questions to ask:
 How difficult was it to log out, log back in, and
continue the registration?

117

ETAP System Usability Testing
Task-Specific Scripts for PIs
SCRIPT FOR NEW REGISTRANTS. WILL NOT HAVE STUDENTS TO CONSIDER FOR THEIR
PROGRAMS.
1.

HOME PAGE
[Give PI URL to the home page.]
First, I’m going to ask you to look at this ETAP home page and tell me what you make of it:
What are your thoughts about what you’re seeing?
What can you do here?
What is this website for?

o
o
o

Just look around and do a little narrative. You can scroll if you want to, but don’t click on anything
yet.

 Remind the tester to think out loud.
 Allow this to continue for three or four minutes, at most.
Now I’m going to ask you to try doing some specific tasks. As you navigate through each page,
I'm going to ask you about your initial thoughts about each page. And again, as much as possible,
it will help us if you can try to think out loud as you go along.
2. GENERAL INFO PAGE
o What are your thoughts about what you’re seeing?
o What is the text telling you?
o What about the Show Filters button? What do you think it does?
Show Filters (For only some testers)
Use the filter to narrow the list.
o
o
o

Are the results what you expected?
What do you think of the filter?
Is there anything you would change about it?

3. ACCOUNT CREATION PAGE
Pretend that you are PI interested using ETAP for your award. Can you navigate to the registration page
to register? Think out loud as you navigate.
(If using Single Sign On with research.gov at time of testing, instruct to log in using their NSF
credentials.)
o What are your thoughts about what you’re seeing?

118

o Remind the tester to think out loud.
 Probing questions to ask:
o Why did you click on the links that you did?
4. SELECT AWARD
Have the PI enter their Award Number to start a competition round.
 What are your thoughts about what you’re seeing?

 Remind the tester to think out loud.
 Have the tester read the text and decide what to do
next.
5. AWARD ROUND INFORMATION PAGE
o What are your thoughts about what you’re seeing?
Have the PI complete information about the award round using fake data.

o
o

Go ahead and fill out the remaining fields on this page.
How difficult was it to enter information?
Is there anything on this page that is unclear or confusing?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
6. USER DESIGNATION PAGE
o What are your thoughts about what you’re seeing?
a. Authorizing a New User
119

Go ahead and fill out the page. You can enter the email address [insert email address] and click
“Authorize”.
b. De-authorize a User [ask if there is an existing authorized user]
Now let’s pretend you want to de-authorize a user who is no longer affiliated with the award.
How would you proceed?

 Remind the tester to think out loud.
Probing questions to ask:
 Once you clicked the Authorize button, did you get
the results you expected?
 Once you clicked the De-authorize button, did you get
the results you expected?
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
7. APPLICATION MATERIALS PAGE (for competitive awards)
o What are your thoughts about what you’re seeing?
o What do you think you need to do?

 Remind the tester to think out loud.
Probing questions to ask:
 Any thoughts about uploading the additional
materials pdf?
 Any thoughts about adding the additional questions?
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend be changed?

120

STOP HERE AND HAVE THE TESTER SIGN OUT OF THE SITE AND SIGN IN WITH A PRE-REGISTERED
ACCOUNT.
Thank you for registering and setting up your account! Now, we’ll move on to the second part of our testing.
Can you log out of your account? We are now going to test a second account that have applicants. In the
email that I sent you earlier today, I sent you the login information for a pre-registered account. Can you go
to your email and log in with the second account?
OR
Can you enter the following log in and password?
[Give the tester the log in and password of the pre-registered account]
SCRIPT FOR REGISTERED USERS. WILL HAVE STUDENTS TO CONSIDER FOR THEIR PROGRAMS.
1. YOUR AWARD PAGE
Have tester describe each of the sections and what they think they mean.
o

What are your thoughts about what you’re seeing?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?

2. APPLICANTS PAGE
OK, let’s navigate to the Applicants Page.
o What are your thoughts about what you’re seeing?
Sort the list of applicants by name. Now sort by number of reference letters received.
o

Are the results what you expected?

Click on one of the applicants.
o

What are your thoughts about what you’re seeing?

Using the interface, update all of the applicants who [provide list of type of student to
update].
o

What are your thoughts about using the interface to update the applicants’ status?

121

Add/Edit Multiple Applicants admissions decisions.
o

What are your thoughts about what you’re seeing?

Using this screen, update all of the applicants who [provide list of type of students to
update].
o
o
o
o

What are your thoughts about this function?
What are your thoughts about the download file?
What are your thoughts about the information to add to the file?
How difficult was it to use this method of updating the applicants’ status?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
Admission Decisions.
Now let’s go through and update the applicants’ admission status using the dropdown menu.
o
o
o
o

What are your thoughts about this function?
What are your thoughts about the dropdown menu?
Is it clear what the “Applicant Decision” field will be?
How difficult was it to use this method of updating the applicants’ status?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
3. PARTICIPANTS PAGE
o What are your thoughts about what you’re seeing?
122

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
4. DATA DOWNLOAD
o What are your thoughts about what you’re seeing?
Download the information for a subset of students using “Show filters”. Filter the list of
applicants by [provide list to filter].
o Are the results what you expected?
o What are your thoughts about this function?
o What are your thoughts about the download file?

 Remind the tester to think out loud.
Probing questions to ask:
 Is there anything on this page that is unclear or
confusing?
 Is there anything you would recommend to change?
5. DATA REPORTS
[If tester clicks on this tab]
o

Overall, what are your thoughts of the potential functionality?

SCRIPT FOR TESTERS REVIEWING THE REFERENCE WRITTER REQUEST
Provide a copy of the reference email to the tester.
o
o

What are your thoughts about what is in the email?
Is there anything you would recommend to change?

v

123