Various Pretesting Activities (see attached list)

Attachment D: Test protocol
Thank you for your time today. My name is , and I will be working with you today. We
will be evaluating a new design of the American Community Survey (ACS) Internet form by
having you answer some survey questions. Your experience with the site is an essential part of
our work. We are going to use your comments to give feedback to the developers of the Web
survey. Your comments and thoughts will help the developers make changes to improve the
site. I did not create the survey, so please do not feel like you have to hold back on your
thoughts to be polite. We want this session to be as realistic and natural as possible, so please
follow the instructions for taking the survey as if you had received this envelope and form in the
mail at home. Please share both your positive and negative reactions to the online survey. We
are not evaluating you or your skills, but rather you are helping us see how well the survey
works.
First, I would like to ask you to read and sign this consent form. It explains the purpose of the
session and informs you that we would like to videotape the session, with your permission. Only
those of us connected with the project will review the tape. We will use it mainly as a memory
aid. We also may use clips from the tape to illustrate key points about the design of the Web
survey.
[Hand consent form; give time to read and sign; sign own name and date.]
[Start the tape when the participant signs the form.]
So today, you will be helping us test the usability of the new ACS online survey form. Your
feedback is valuable, and we appreciate your help. We are going to do some eye tracking as
well as have you work on some survey questions online.
Before we get started, please take a moment to complete this computer usage and Internet
experience questionnaire. This questionnaire includes some questions on your background.
[Hand the participant the background and computer experience questionnaire.]
During the session, I will ask you to work on completing the survey. I would like you to tell me
your impressions and thoughts about the survey as you work through the questions. I would like
you to “think aloud” and talk to me about your answers. So if you expect something to happen,
tell me what you expect. If you expect to see some piece of information, tell me about what you
expect. This means that as you work on answering a question, talk to me about what you are
doing, what you are going to do, and why. Talk to me about why you clicked on a link or where
you expect the link to take you.
Finally, during the session, I will remind you to talk to me if you get quiet, not to interrupt your
thought process but simply to remind you to talk to me. Please focus on verbalizing what you
are thinking and expecting to happen. We are interested in the reasoning behind your actions,
not just what you are doing.
Before we get started, let’s practice thinking aloud. [This is an example of what this practice
might involve: Please go to www.wtop.com. Describe your thought process as you select a new
story that you find interesting to read.]
Now I am going to calibrate your eyes for the eye-tracking. I am going to have you position
yourself in front of the screen so that you can see your nose in the reflection at the bottom of the

35

monitor. To calibrate your eyes for the eyetracker, please follow the blue dot across the screen
with your eyes.
[Do Calibration]
IF EYES CALIBRATE: Now that we have your eyes calibrated, we are ready to begin.
IF EYES DO NOT CALIBRATE: We are going to proceed without the eyetracker.

36