Various Demographic Area Pretesting Activities

Attachment C

Protocol for the National Census Test Web Questionnaire

Thank you for your time today. My name is XX. I work here with the Human Factors and Usability Research group and I will be working with you today. We will be evaluating the design of the online National Census Test Questionnaire by having you complete it. Your experience with the survey is an essential part of our work. I did not create the survey, so please share both your positive and negative reactions to it. We are not evaluating you or your skills, but rather you are helping us see how well the survey works. The entire session should last about an hour. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to videotape the session, with your permission. Only those of us connected with the project will review the tape and any other data collected during the session; and it will be used solely for research purposes. We may also use clips from the tape to illustrate key points to the Web design team. In addition, there may be members from the project team observing this session in another room.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen.

I would like you to tell me your impressions and thoughts about the screens as you look at them. In other words, I would like you to “think aloud” and talk to me about your impressions. If you expect to see some piece of information, tell me whether you saw it or not.

Pull up www.wtop.com in Firefox.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while working online. Pretend that you have a minute or two to kill at your desk at work. Talk me through your thought process as you try to find something interesting to read on this news site <or any other if this one is not one that you would normally look at>.

Ok, that’s exactly what I would like you to do throughout the session. If at any time during the session you get quiet, I may remind you to talk to me. This is not to interrupt your thought process, but simply to remind you to keep talking to me. Please focus on verbalizing what you are thinking as you complete the survey.

Do you have any questions about the think aloud technique that we just practiced?

This is the card you would receive and use to access the survey online if you were taking this survey at home. If you were to receive the survey at your home, the mailing materials would have your real address. For the purposes of this study, please pretend that your address is 123 Any Street in Anytown, US.

Now I am going to calibrate your eyes for the eye tracking.

Do Calibration

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey online as you would at home. You may answer the survey questions as they apply to you in your real life. Although the materials will give you an Internet address, or URL, to enter to access the survey, you will not need to enter that because our testing software will open the survey for you.

I am going to go around to the other room to do a sound check. While I am doing that, please take a moment to complete this questionnaire. [Hand P questionnaire on Computer experience and demographics]

I’m going to leave but we will still be able to communicate through a series of microphones and speakers. Do you have any questions before we begin?

Leave room. Once in control room do a sound check and Start the eye-tracking software: Tobii Studio. The mouse tracing software will start when Studio opens Internet Explorer.

Encourage P to think aloud while completing the survey. Ask probe questions about what they are thinking if they are having trouble with any part of the survey.

Make a note and probe if the P has any difficulty with or comments on a prefill or predictive text on the race question..

Make note of any signs of respondent confusion in any coverage question, and probe retrospectively.

Note any signs of respondent fatigue in any coverage question, and probe retrospectively.

Note any questions respondent interprets as redundant and note which questions those are, and why they sound redundant. If respondent does not spontaneously remark about perceived redundancy, ask them explicitly in the debriefing if any questions seemed redundant, which ones, and why that was.

Pay close attention to (and note) how the respondent experiences the instrument and the fills for any questions in which an alternative address is displayed as part of the question or response set. Probe retrospectively if any difficulties, hesitation are evident.

When P has completed the survey, tell him/her:

Thank you for completing the National Census Test Questionnaire. Please complete this satisfaction questionnaire. I will be back around to ask you a few more questions about your experience.

NCT Internet Test Debriefing Questions

After the participant is finished with completing the survey, begin the debriefing questions.

Review their responses to satisfaction survey and probe about any drastic scores.

W hat was your overall impression of the survey?

W hat are some things that you liked about the survey?

What are some things that you disliked about the survey?

Is there anything about the survey that you did not understand?

H ow does this survey compare to the Web surveys that you have taken in the past in terms of visual appeal?

Login Screen/Security Questions

W hat was your overall reaction to the security questions?

If you forgot your PIN, how easy or difficult would it be for you to remember your answer to the security question?

Very Easy Somewhat Easy Somewhat Difficult Very Difficult

1 2 3 4 5 6 7 8 9 10

D id you think the login process was more secure than other logins you have completed, like bank sites and email accounts, about the same level of security, or less secure?

On a scale of 1 to 10 with 1 being very easy and 10 being very difficult, how easy or difficult was it to log into the survey?

Very Easy Somewhat Easy Somewhat Difficult Very Difficult

1 2 3 4 5 6 7 8 9 10

Age/Date of Birth

D id you notice that the age was automatically calculated for you? If so, did you like, not have an opinion, or dislike this feature?

On a scale of 1 to 10 with 1 being very easy and 10 being very difficult, how easy or difficult was it to indicate your date of birth using the drop-down menus?

Very Easy Somewhat Easy Somewhat Difficult Very Difficult

1 2 3 4 5 6 7 8 9 10

When entering your date of birth, do you prefer a drop-down menu option like the one in this survey, or a text-entry option where you just type the date in, or do you have no preference? The text entry option usually looks like this:

W hat would you do if the age displayed was incorrect? What you would change -- DOB or age – and how would you do it? Would you expect a recalculation of age if you changed the date of birth?

Coverage Questions/People

In your own words, what does "living and sleeping" mean to you in this question?

What time frame did you use to answer this question? [specific day vs. general time]

Miss

Was there anyone whose name you thought of including, but didn't? If yes, probe about that person's situation (i.e., living situation).

Miss

Did you read the list of examples above the response options?

Yes No

What did you think this question was asking about: We do not want to miss any people who might have been staying at 123 Main Street, Apt. G on June 5, 2012. Were there any additional people that you did not already list?

Please consider the following stories and answer the questions about them.

b abies - Brenda was eight months pregnant and delivered her baby prematurely. Brenda is allowed to leave the hospital, but the newborn has to stay in the hospital and gain a little weight before coming home. Brenda is at home, filling out this survey. How should Brenda answer the undercount question [We do not want to miss any people who might have been staying at 123 Main Street, Apt. G on June 5, 2012. Were there any additional people that you did not already list?]? How should Brenda answer the following questions? Where was the newborn on Census Day? Where does the newborn live and sleep most of the time? Note (probe if necessary) participant 's rationale.

inlaws - Barry's in-laws have been staying with him and his wife for the past three months, since their home burned down. The in-laws have awesome homeowners insurance and their home will be rebuilt within the next eight months. Until then, and much to Barry's dismay, his in-laws are going to be sleeping in the guest bedroom. As Barry is filling out this survey, how should he answer the following questions? Where were your in-laws on Census Day? Where do your in-laws live and sleep most of the time? Note (probe if necessary) participant 's rationale.

custody - Suzy is eight years old and lives with her mother. Suzy spends every summer with her father, who lives in another state. The day Suzy's mom fills out this survey, Suzy has been with her father for the last few weeks and will not return home for another few months. How should Suzy's mom answer the following questions? Where was Suzy on Census Day? Where does Suzy live and sleep most of the time? Note (probe if necessary) participant 's rationale.

college - Maria is a college student who lives in a dormitory on campus during the week. On the weekends, Maria stays at her parents house so her mom can do her laundry and Maria can catch up with her high school friends. Maria's mother is filling out the survey on the weekend, so she lists Maria as someone who lives in the household. How should Maria's mom answer the following questions? Where was Maria on Census Day? Where does Maria live and sleep most of the time? Note (probe if necessary) participant 's rationale.

military - Jessica is married to Tom, and they are both in the Army. Jessica has just been redeployed to Afghanistan for the third time. She's only been gone a few days and Tom is filling out this survey. Tom includes Jessica's name on the survey. How should Tom answer the following questions? Where was Jessica on Census Day? Where does Jessica live and sleep most of the time? Note (probe if necessary) participant's rationale.

t enuously attached - Matt and his wife, Sharon, are free spirits. They have an open-door policy at their house, which means that anyone can crash there at any time. A guy named Rudy sometimes shows up and sleeps on their couch. Matt and his wife have no idea where Rudy sleeps when he's not on the couch at their house. When Matt is filling out this survey over his morning coffee, Rudy wakes up and asks if he can stay over one more night. In addition, he wants to know what's for breakfast. What response should he give to the following question? Were there any additional people staying here on Census Day? Where was Rudy on Census Day? Where does Rudy live and sleep most of the time? Note (probe if necessary) participant’s rationale.

s easonal/second residence - Kathy is retired and maintains two homes. One in Maryland and one in North Carolina. She hates the cold winters in Maryland, so she tries to stay in North Carolina during the winter. However, she loves the warm summer months, and so she stays in her Maryland home a little more often during the summer. Kathy happens to be staying at her home in Maryland when she is filling out the survey that lists the address of her Maryland home. How should Kathy answer the following questions? Where was Kathy on Census Day? Where does Kathy live and sleep most of the time? Note (probe if necessary) participant 's rationale.

Race

Did you have any trouble finding the response option(s) that you were looking for?

What would you do if you wanted to select more than one response option?

D id you notice that you had to scroll down to find the Next button? Is this design easy to follow and understand or did you have to think about it before you scrolled?

When you started to type in some of the response boxes, a list of possible answers appears and you can select one if it applies to you. Did you notice this happening during your session? If yes, did you like, have no opinion, or dislike this feature? Did you think you had to select one of the possible answers that came up?

At the end of the survey, it asked for your email address. Why do you think your email address was requested?

On a scale of 1 to 10 with 1 being very easy and 10 being very difficult, how comfortable were you with providing your email address to the Census Bureau?

Very Comfortable Somewhat Comfortable Somewhat Uncomfortable Very Uncomfortable

1 2 3 4 5 6 7 8 9 10

Other than what we have already talked about, did you have any other comments or suggestions about the Web survey?