Various Demographic Area Pretesting Activities

2013 American Community Survey Internet Instrument Usability Test: Focus on reducing break-offs for questions in detailed person section, & evaluating security questions to allow for rest PIN

Thank you for your time today. My name is XX and I work here with the Human Factors and Usability group and I will be working with you today. In this lab, we evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Today, we will be working on a few projects together. There are three parts to our session:

1: First, we’ll do a word recall task.

2: Second you will help us evaluate the design of the online American Community Survey by completing it online. I’ll ask you some satisfaction questions and debriefing questions about the survey as well.

3: Third you will work on a set of questions.

Then we’ll wrap up and I or my colleague will escort you back out.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to videotape the session to get an accurate record of your feedback. Only those of us connected with the project will review the tape and it will be used solely for research purposes. Your name will not be associated with the tape or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

If you run into any difficulties, it will be the result of the design of the Web survey, not your skills or abilities. This Web survey is intended for people like you. If you have a problem using parts of it, please don’t blame yourself. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

While you are working, I’d like you to think aloud. In other words, I’d like you to tell me what you are thinking, describe the steps you are taking, what you are expecting to see, why you are doing what you are doing, what you are going to do, and why. If you expect to see some piece of information or expect something to happen, tell me about it.

Since thinking aloud is not something that we normally do on a regular basis, we are going to do a practice of thinking aloud. [open www.craigslist.com and do practice question. Practice can be that they have a friend who moved into a new apartment and asked for help finding a couch. Can also do the how many windows where you live task.]

Ok, that was fine. Do you have any questions about the “think-aloud” process we’ve just practiced and that I have asked you to use? If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

[Eye tracking]

In addition to the survey that we are about to work on, and thinking aloud that we just practiced, we are also going to record where you look on the screen. In a moment, we will do a simple task that will allow the computer to find your eyes. I am going to have you position yourself in front of the screen so that you can see your nose in the reflection at the bottom of the monitor. To calibrate your eyes, please follow the dot across the screen with your eyes.

[Calibrate the participants’ eyes]

Ok great. I now have a demographic questionnaire that I’d like you to fill out.

[Hand the demographic questionnaire to participant and wait in the room while they fill it out and return it to you]

Now I would like us to work on the word recall task. This is a short exercise that gives us information about memory.

[Proctor the memory task—this is in testing docs folder on M drive: 2013Commitment Research Study]

Thank you. Here are the materials we will use for the evaluation of the Survey. When we are ready to begin, you will be working with them. If you were to receive the survey at your home, the mailing materials would have your real address. Since we cannot replicate that for the lab setting, all participants will use the same address. For the purposes of this study, please pretend that your address is:

198 Young Rd

Anytown, MD 03612

Please respond to the online survey as you would at home. You may answer the survey questions as they apply to you in your real life. Although the materials will give you an Internet address, or URL, to enter to access the survey, you will not need to enter that because our testing software will open the survey for you.

[Hand participant the mailing materials]

Now I am going to go around to the other room and do a sound check. While I am doing that, please take a moment to complete this Background Questionnaire. I am going to leave, but we will still be able to communicate through a series of microphones and speakers. Do you have any questions?

[Open background questionnaire on computer and Internet experience]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

[Leave room. Once in control room do a sound check and start the eye-tracking software: Tobii Studio. Encourage R to think aloud while completing the survey. Ask probe questions about what they are thinking if they appear to be having trouble with any part of the survey, e.g., keep talking, etc.

During survey:

Pay attention to following screens:

PIN screen:

Observe if participant writes or makes a note of the pin

Observe if participant has issues with answering the security question.

(e.g., selecting a question, providing an answer, reluctance to provide an answer)

Email Reminder

Observe whether participant provides a valid email address.

Observe whether participants checked the email opt-out box.

*Observe with AOI & Eye-tracking if participant reads the opt out box, if participant reads the information: “We may contact you if there is a question.”)

Unfolding Questions

Observe whether participants notice the highlighting after they click on the radio button (yellow color w/ thick dark outline of box—see example screen below) for the following:

--Mark if participant completed the drop down/write in box without being prompted by an error message.

--Mark which ones they skipped and got the error message

--Mark NA if they did not get that screen:

Screen	Completed w/out being prompted	Skipped and got error message
Place of birth
Year built
What grade level
Highest grade level
Citizenship
Residence one year ago
Health insurance
Computer use, and Internet subscription
Race and Hispanic (check boxes)

Combined Saved Person/Pick Next Person Screen

Observe participant’s reaction to this screen each time it appears:

Observe:

--Do they appear to read the instructions about continuing even when they do not know all of the answers for everyone?

--What do they do next (i.e., select a person and continue, hesitate, click Save & Logout)?

--If they continue with the survey, note whether they seem to have any issues answering questions (or leaving them blank) for related and unrelated household members.

PIN screen LOG-OUT LOG Back in During Session

At the detailed person question for the 2^nd person in the household -- somewhere in the detailed person questions—aprox around Q.18-21]

[Tell participant:]

Now I want you to imagine that you had to go somewhere and needed to save and continue answering the questions later. What would you do?

Okay now you need to log back in. Please do that now.

[Need participant to save and log out and then log in again, will have to help them if they cannot do it by themselves.]

--Observe if they have any trouble re-entering the survey?

--Observe if they used the PIN to re-enter, watch how they do it.

--Have them enter using the a security question (e.g., say they forgot)

Now I want you to pretend you forgot your PIN. What would you do to get back in to continue the survey?

--Have them click the link “Click here if you do not know your PIN.”

--Observe if they have any trouble answering the security question?

--Observe if they have any trouble re-entering the survey with the new PIN?

After final SUBMIT is clicked, have participant answer satisfaction questions. Open QUIS and have them answer before beginning the debriefing

Now I would like you to answer these satisfaction questions

After Survey: Debriefing Questions

Pull up each screen on the computer (use screen shots as a guide which ones to open) and allow the participant to look at each screen as you ask questions about it.

FOR Debriefing, you will need to log back in using the User ID and PIN and then enter this URL to get back to the beginning of the survey: https://acs-oneal312.centurion-qa.ssd.census.gov/mpt/.

PIN Screen

Open up the security questions screen and show them the questions again.

What were your thoughts about the security question and answer?

Do you have any concerns or reluctance about providing an answer to any of the security questions?

Unrelated Household members: Combined saved person / pick next person

What is your reaction to the text on this screen?

Were you feeling like you wanted to exit the survey at this point? Why or why not?

Did you feel like you provided as much information as you could for the people in your house who were unrelated to you?

Did you have any problems moving forward and answering the questions for unrelated household members?

How do you think you would you answer the survey questions if you were at home: would you answer as many questions as you could or would you log out if you did not know the answers?

Email Reminder

What were your thoughts when you were asked to provide an email address?

What do you think we will do with your email address? Will we contact you? If so, about what?

How many email addresses do you have and (if more than one) why did you provide the one that you did (i.e., did you give us a real working one, an infrequently used one? would you be inclined to give an email address that you might use for junk or spam mail?).

Do you have any privacy concerns regarding providing your email address?

If you were taking the survey at home would you provide this email address? A different one? None? Why or why not?

Email Message

Hand the participant the email message reminder and have them read it:

Take a minute to read this.

What are your initial thoughts or concerns about this email? Would you open the message if you saw it in your inbox? Why or why not?

Would you return to the survey to complete it? Why or why not?

[If they say yes they will return to complete the survey ask:]

When would you return to complete the survey? (e.g., immediately after opening the message, later on the same day, the next day, etc.).

Highlighting on the unfolding questions

Navigate to a few of the unfolding questions that the participant used (e.g., place of birth, citizenship, race and Hispanic origin, etc.) and ask them:

What do you think about the write-in boxes?

Did you notice the drop down-write in boxes?

Did you use/see the drop down write in boxes while you were answering the questions?

Finishing Survey

What were your reactions to the text on this screen? Was your first inclination to exit the survey at this point? Why or why not?

Ask if any observers have any final questions about the survey.

Email message reminder

Example Email:

From: U.S. Census Bureau

Subject line: The Census Bureau’s American Community Survey

Recently, the Census Bureau sent you a request to answer questions online for the American Community Survey. If you have already completed the survey, thank you for your responses. If you have started, but not yet completed the survey, click here to return to the survey where you left off. Enter your USER ID XXXXX-XXXXX followed by your assigned PIN. If you do not remember your PIN or your answer to the security question, please call toll free 1-888-369-3603.

If you have not yet started the survey, please click here to begin.

Completing the survey online saves time and money by reducing the need for further contact by Census Bureau interviewers. Thank you for your time and cooperation.

_____________________________________________________________________________________

Questions? Contact Us

 

STAY CONNECTED:

SUBSCRIBER SERVICES:
Manage Preferences  |  Unsubscribe  |  Help

[After debriefing is complete-- Now proctor the research component]

I just need to step out and get something.

[Exit the room to retrieve and open the randomly selected envelope.

If envelope says “C” go back into participant’s testing room and say:]

Now it’s time for our final set of questions. Please answer these questions, I’ll be in the other room.

[Open the link FQ Questionnaire—located at (M:\USABLE\2013 Commitment Research Study\Questionnaire Links) Then leave the room.]

[After completion of 10 questions, go back into testing room and Give payment.]

If the envelope says “T” go back into the participants testing room and run treatment condition,

Now it’s time for our final set of questions. When you answer the questions that you’ll see on the following screens, it’s very important that you take your time and think carefully so that you answer each question as accurately as possible.

Are you willing to commit to answer each question as accurately as you can?

[After the respondent says yes, the test administrator will lay down the commitment paper]:

Please read this over, check this box,

[point to the box and the paper that you will be having them sign]

and sign your name if you are willing to commit to answer each question as accurately as you can.

[Once receiving the commitment and signature, Open the link FQ Questionnaire—located at (M:\USABLE\2013 Commitment Research Study\Questionnaire Links) Then leave the room.]

[After completion of 10 questions, go back into testing room and Give payment.]