Various Demographic Pretesting Activities

Attachment B

General Introduction: Community Survey Internet Instrument (Roster Test –Third Round)

Thank you for your time today. My name is XX and I work here with the Human Factors and Usability group and I will be working with you today. We will be evaluating the design of the online American Community Survey by having you complete it. Your experience with the survey is an essential part of our work. I did not create the survey, so please share both your positive and negative reactions to it. We are not evaluating you or your skills, but rather you are helping us see how well the survey works. The entire session should last about an hour. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to videotape the session, with your permission. Only those of us connected with the project will review the tape and any other data collected during the session; and it will be used solely for research purposes. We may also use clips from the tape to illustrate key points about the survey to the Web design team. In addition, there may also be observers from the project team observing this session in another room.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen and we will record your mouse movements to see how you are interacting with the survey.

I would like you to tell me your impressions and thoughts about the screens as you look at them. In other words, I would like you to ``think aloud'' and talk to me about your impressions. If you expect to see some piece of information or expect something to happen, tell me whether or not it was met.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work or at home and talk me through your thought process as you try to find something interesting to read.

Ok, that’s exactly what I would like for you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Do you have any questions about the think aloud technique that we just practiced?

If you were to receive the survey at your home, the mailing materials would have your real address. Since we cannot replicate that for the lab setting, all participants will use the same address. For the purposes of this study, please pretend that your address is 198 Young Rd in Anytown MD.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey online as you would at home. You may answer the survey questions as they apply to you in your real life. Although the materials will give you an internet address, or URL, to enter to access the survey, you will not need to enter that because our testing software will open the survey for you.

I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand Participant questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and Start the eye-tracking software: Tobii Studio. The mouse tracing software will start when Studio opens Internet Explorer.

Encourage R to think aloud while completing the survey. Ask probe questions about what they are thinking if they are having trouble with any part of the survey.

IMPORTANT: IF the Participant enters more people for the Add_1 screen, make a note of it for the debriefing session.

Debriefing

You listed XX people who live at or have been staying at your house for more than two months. Could you tell me about each of these people and how long they have been living in your household? Or how often they stay there?

Is there anyone else staying in your household that you did not list while taking the survey?

PROBE IF SOMEONE MENTIONED: Why did you decide not to list that person/those people?

Is there anyone else who stays with you sometimes, but isn't here now?

Please tell me how much time that person has spent at each of his/her addresses in the past year.

Does anyone in your household have another place to live or stay for work?

If ``yes'': Please tell me how much time that person has spent at each of his/her addresses in the past year.

Are there any children in your household who stay somewhere else, like for boarding school or for a joint custody situation?

If ``yes'': Please tell me how much time that person has spent at each of his/her addresses in the past year.

Screens

Debriefing Questions

Pull up each screen and allow the participant to look at each screen as you ask questions about it.

Login Screen

Did you have any difficulty finding the User ID?

Did you have any difficulty entering the User ID?

Generate PIN Screen

Why did the survey generate this PIN for you?

Do you think there is a way to retrieve the PIN if you forget it?

Did you notice how long it would take to complete the survey?

Resp Name Screen

Why do you think we are asking for your name and telephone number?

If you received this survey and took it at home, would you be concerned about proving your telephone number? Why or why not?

If you received this survey and took it at home, would you be concerned about providing your name? Why or why not?

Roster_a

Did you notice that your name was already entered on the first line?

Did you notice the link “Click here to add more people”?

What do you think would be displayed when you clicked it?

Roster_b

Does anybody on the list above fit any of the categories listed in this question – roommate, foster child, boarder, live-in employee?

Add_1

Did you notice that a follow-up question appeared at the bottom of the question that you just answered? They only get this if they said “yes” to roster_b so this may not be relevant to R.

Was this confusing?

Did you read the question again? If any names were entered, ask: Why didn’t you include this person/these people on the original list?

Roster_c

If any names were entered on the previous screen (Add_1), ask: Did you notice that the names you entered on the previous screen were added to the list?

Add_2

Did you notice that a follow-up question appeared at the bottom of the question that you just answered? They only get this if they said “yes” to roster_b so this may not be relevant to R.

Was this confusing?

Did you read the question again?

If any names were entered, ask: Why didn’t you include this person/these people on the original list?

Remove_one

If any names were selected, ask: Why did you select this person/these people?)Was anything in this question confusing?

Another_home

If any names were selected on the previous screen, ask: Did you notice that the names you selected on the previous screen are not on this list?

What do you think is meant by “another place where they usually stay”?

Roster_check

In your own words, what is the text telling you?

Reference Person (if applicable)

Was this question confusing?

If YES, why?

Did you realize you could mark more than one person, if necessary?

After the participant has completed the survey for their own household, have them complete the vignette.

Imagine that I am your neighbor who does not have Internet access and I ask you to help me fill out my ACS Internet form online. I do not want to fill out the paper form by hand and believe having you help me do it online will be the fastest and easiest way to complete it. It is a large household. The experimenter will represent Maggie and verbally provide this household information to the participant and also provide a printout of the household for reference. The same experimenter will represent Maggie for every session. A similar procedure was used previously for the Census 2010 Internet usability testing.

Maggie G. Robinson  - Respondent/Neighbor - Asked for help filling out Internet form.
Sam F. Robinson  - Respondent's Spouse - Job is in Philadelphia, so he stays with his sister in Philadelphia Monday through Thursday nights. He returns home to D.C. for the weekends
Lucy G. Robinson - Respondent's Sister - Attends college and currently  lives in a dorm there and returns home for holidays and for the summer.
Jim N. Robinson  - Respondent's Brother - Used to live elsewhere in an apartment but has been living at home for the last  two weeks while he looks for a new apartment.
Alice L. Robinson  – Respondent’s daughter – Used to live in the house until four months ago, when she went away to begin boarding school.
Nicholas S. Robinson  – Respondent’s son - Is in a shared custody arrangement and stays with his father most of the time, but is staying at this address with his mother on the day this survey is filled out, which would be today.
Carolyn T. Robinson  - Respondent's Mother--in--law - Used to live in the house, but has been placed in a nursing home for a three month trial.
Mary T. Carpenter  - Live-in housekeeper - Stays with the respondent's family during the week, but spends weekends with her own husband and children at their apartment.

The participant will answer the roster questions for this household.  The experimenter will read each scenario to the participant.

The experimenter will then compare the final roster to the 'correct' roster, listed below:

Correct final roster
Should include: Maggie G. Robinson , Sam F. Robinson , Jim N. Robinson, Nicholas S. Robinson  
Should Exclude: Alice L. Robinson , Carolyn T. Robinson , Mary T. Carpenter, Lucy G. Robinson

After the roster A, B, and C screens, it should include all eight family members.  If it does not, ask:
Why they did not include the missing person(s)?

If the roster is fully correct, ask the following:
On a scale of 1 to 10 with 1 being very easy and 10 being very difficult, how easy or difficult were these questions to answer for Maggie's household?


If the roster is not correct, ask the following:
I'd like to ask you some questions about what you were thinking about at some of the screens.

If Maggie, Sam, Jim, or Nicholas are not on the roster, ask:
When you answered roster A/B/C, why did you/did you not include Maggie/Sam/Jim/Nicholas ?  

When you answered AWAY NOW,  why did you/did you not consider Maggie/Sam/Jim/Nicholas?

When you answered ANOTHER HOME,  why did you/did you not consider Maggie/Sam/Jim/Nicholas?

When you answered MORE THAN TWO,  why did you/did you not consider Maggie/Sam/Jim/Nicholas?

If Alice, Carolyn, Mary, or Lucy are on the roster, ask:
When you answered roster A/B/C, why did you/did you not include Alice/Carolyn/Mary/Lucy ?  

When you answered AWAY NOW,  why did you/did you not consider Alice/Carolyn/Mary/Lucy?

When you answered ANOTHER HOME,  why did you/did you not consider Alice/Carolyn/Mary/Lucy?

When you answered MORE THAN TWO,  why did you/did you not consider Alice/Carolyn/Mary/Lucy?

For each person not correct included (or correctly excluded), ask:
Why did you/did you not think that (person's name) should be listed as part of Maggie's household?