Various Demographic Pretesting Activities

Protocol for Testing the Census Quality Survey Web Survey

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen

I would like you to tell me your impressions and thoughts about the screens as you look at them. In other words, I would like you to ``think aloud'' and talk to me about your impressions. If you expect to see some piece of information or expect something to happen, tell me whether or not it was met.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work. Go to a common Web site like www.espn.com or yahoo news and talk me through your thought process as you try to find something interesting to read.

Ok, that’s exactly what I would like for you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Do you have any questions about the think aloud technique that we just practiced?

After think--aloud practice is complete

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey online as you would at home. You may answer the survey questions as they apply to you in your real life.

I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand P questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and Start the eye-tracking software: Tobii Studio. The mouse tracing software will start when Studio opens Internet Explorer.

Encourage R to think aloud while completing the survey. Ask probe questions about what they are thinking if they are having trouble with any part of the survey.

Make note of the following items:

Do Rs understand the three “do you live here”, “does anybody live here”, “are there temporary people here” screens.
Roster – do Rs understand how to add people, are they adding all relevant people.
Do they understand topic-based format?
Do they understand the CCM screens at the back (were you anywhere else, group quarters, where was this, provide an address, etc.)
Do they choose to review? If so, do they understand how to edit?
Do they access help/FAQs/instructions? If so, is the content understood? Is it helpful?

CQS Internet Test Debriefing Questions

After the participant is finished with completing the survey, begin the debriefing questions.

Review their responses to QUIS and probe about any drastic scores.

What was your overall impression of the survey?

What are some things that you liked about the design?

What are some things that you disliked about the design?

Is there anything about the design that you didn’t understand?

How does this survey compare to the Web surveys that you have taken in the past in terms of visual appeal?

Do you have any additional comments?

Pull up each screen and allow the participant to look at each screen as you ask questions about it.

http://idc4.ssd.census.gov:3006/decennial

Login Screen

Did you have any difficulty finding the access code?

Did you have any difficulty entering the access code?

Did you notice the warning message at the bottom of the screen? If not, please read it now. In your own words, what is this message telling you?

Address Verification <confirm>

Did you notice that a new feature appeared on the right side of this screen?

If YES: What did you think was the purpose of this feature? If they say “It tells me where I am” (mimicking the label), ask, “Could you say a little bit more about what that means to you?”

If NO: What do you think would be the purpose for this kind of feature?

Right Respondent 1 <address>

They should say yes to the living or staying there.

Right Respondent 2 <right_resp>

PIN Creation <PIN>

Why did the survey generate this PIN for you?

Did you realize that you needed to make a note of the PIN given to you here in order to access the survey at a later time in case you were not able to finish it during this session?

Do you usually write down PINs that are provided to you?

IF YES: Where do you keep the PIN?

IF NO: Why not?

Do you think there is a way to retrieve the PIN if you forget it?

Do you think there is a way to re-enter the survey without your PIN?

People

In your own words, what does "living and sleeping" mean to you in this question?

What time frame did you use to answer this question? [specific day vs. general time]

Miss

Did you read the list of examples above the response options?

What did you think this question was asking about?

Note: The "Temp" screen is no longer valid and has been deleted.

Did you fill out the form? <completed>

Did you notice the “Help” link that followed the question?

If you clicked on it, what information would you expect to find there?

What do you think the "No" answer means?

How would you answer if you did not know who filled out your household's 2010 Census form?

Tenure <home>

Ownership <owner>

Relationship <relation>

Ask if they now noticed that the progress indicator moved to "Person Information".

Sex

DOB/Age <age>

Did you notice that the question was asking for Age as of November 1?

Did you notice that the Age box was filled in automatically based on the birth date you entered?

If YES: Did you expect that auto-filling to happen?

Did you like that the application filled in the age based on the birthdate? (assuming they entered a full birthdate!)

What would you do if the age displayed was incorrect? What you would change -- dob or age – and how would you do it? Would you expect a recalculation if you changed the date of birth?

Did you notice that the tab automatically moved to the next box after you completed the first box? Did you like this feature? Why or why not?

Did you notice the labels above the dob boxes? Were the labels above the dob boxes helpful or not very helpful?

Hispanic Origin <origin>

Did you think you were limited to just one response or could you mark as many as you wanted to?

If Yes or No: Why?

If you wanted to answer "Bolivian", how would you do it?

Race

What did you think of the arrangement of the response options for this question?

If you were American Indian and Asian, do you think the survey would allow you to select both of these responses?

Did you notice that you had to scroll down to find the Next button? Is this design easy to follow and understand or did you have to think about it before you scrolled?

Live or Stay Elsewhere? <elsewhere>

Will have one screen for Path Smith and one for Chris Smith.

How many responses do you think the survey would allow you to select for this question?

Review/Submit Screen <review2>

If you were taking this online survey at home, would you review your answers before submitting them?

Review/Edit <review3>

If you wanted to change the age of Pat Smith, how would you do it?

What does the [No Answer] tag next to Chris Smith's date of birth represent?

Thank You

Would you normally print this screen for you records?

Do you think this is all you need to do for the survey or is there some other step involved in submitting your answers? If yes, what?

Applies to whole survey:

Did you notice the Instructions, FAQs, and Save & Logout at the top of the page?

Did you realize that these were clickable?

If NO: Would you have clicked on any of these if you had known they were clickable?

If you were to click this, what would you expect to see? (Ask for each individual tab.)

Did you notice the Accessibility/Privacy links at the bottom of each page? If you were to click these links, what would you expect to see?

What did you think about the placement of the Previous and Next buttons? On a scale of one to ten with one being very easy and ten being very hard, how easy or difficult was it to navigate back and forth between the screens?

Other than what we have already talked about, did you have any other comments or suggestions about the Web survey?