2017 Census Test Usability Testing

2017 Census Test

Thank you for your time today. My name is XX and I work with the Human Factors and Usability group at the Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

Ok great. Today, we have about 5 or 6 different tasks we are going to ask you to do.

The first task is to complete two questionnaires. They collect information about you and how often you use different electronic devices.

[Have participant fill out the demographic questionnaire and the background mobile questionnaire. You can leave the room and watch from observation room when they are ready to begin again. If off site—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.]

Thank you.

Now, you will complete the main task.

Today you will be helping us evaluate the design of the 2017 Census Test by completing it using [your smartphone/ your tablet/ this laptop].

If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.

Think Aloud: I want you to answer the questions exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

Great that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

[Eye tracking for census staff cases]

We are also going to record where you look on the screen as we work on our study. Now we will do a simple task that will allow the computer to find your eyes. To do this, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

[interviewer: if with ID]

[Hand participant the mailing materials with their ID]

This is an example of mailing materials you would receive at your home if you were selected for the 2017 Census Test. Please take a moment to read it. Notice that the address is not your real address. If you were to receive the survey at your home, the mailing materials you would get would have your real address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter/card came to your address and the address in the survey is your address.

That is the only part of the study that is pretend. Please answer the survey questions as they apply to you in your real life.

Laptop/Desktop: We will bring up the online questionnaire for you.

[interviewer: if without an ID]

[Hand participant the paper with URL]

We would like you to pretend that you heard about the 2017 Census Test on the radio and wanted to log onto the URL you heard to complete your census survey. This is the URL [point to the URL on the paper]

Now go online and complete the online census questionnaire. You should answer the survey questions as they apply to you in your real life.

Laptop/Desktop with eye tracking: We will bring up the online questionnaire for you.

All Sessions:

[STOP Respondent after they enter demographics for themselves and ask them to:]

Okay, now I’d like you to pretend that you had to leave the survey to do something else.

[Once they are exited, then ask them to resume. Depending on the screen they arrive at, you might have to text them or type the correct URL for them if they had difficulty in the first round.]

Okay now you can go ahead and get back into the survey to finish it up. What would you do?

After final SUBMIT is clicked, have participant answer satisfaction questions.

Now I would like you to answer these questions about how satisfied you were with the questionnaire you just completed. (Use automated satisfaction questionnaire.)

Now that you have filled out the census form, what do you think is the most important use of the Census data?

How do you think the census data may benefit your community? And do you see any benefit of the census data at the personal level?

Now we are onto the next set of tasks:

[Only complete these tasks if you have enough time and if the situation didn’t come up during the interview.]

1^st vignette: Date of birth screen – PIN starts on the DOB screen for roommate Jamie

We would like to get more feedback on the date of birth question. Now log back in with this ID and this PIN. This should take you to the age and date of birth screen. We want to get some more feedback on that screen, so for this you will have to pretend: You recently had a new roommate move in named Jamie Doe. You don’t know Jamie’s age but Jamie recently went out to celebrate her birthday this month. She also recently graduated from college.

Please show me what you would do to answer this question for Jamie and tell me why you are answering this way.

2^nd vignette: Changing language of survey – continue on DOB screen

Now pretend that you speak Spanish and you wanted to see the question in Spanish. What would you do?

3^rd vignette: Second residence – start from home screen of decennial census – PIN opens user to the overcount screen

Now we have a third pretend situation. Please enter this ID and this PIN. Jane is completing this census form mid-April and she knows that her new roommate, Jamie, moved in April 1 to the place. How should Jane answer this question for Jamie?

4^th vignette: Relationship edit check – start from home screen of decennial census – PIN opens user to the sex question

Here is a 4^th situation still with Jane. Please log back in with this ID and this PIN. "Jane, who is a woman, is married to John, who is a man.  Jane has completed the detailed questions for herself.  She is completing/has completed the questions about her husband and gets to this screen.    You are Jane, what would you do?  Go ahead and do that and remember to keep talking."

[DO 5^th VIGNETTE WITH FOREIGN-BORN RESPONDENTS]

5^th vignette: Race/ethnicity of US-born children – start from home screen of decennial census – Pin/login opens user to screen from the race and detailed origin question.]

Here is a 5^th situation. Use this login and PIN. Maria and her husband Pedro are both Hispanic and from Mexico. They have a son who was born here in the US. How should they answer the next 2 questions about their son? [

The next task is on the laptop. We are going to go over each of the screens you saw.

Debriefing:

On each screen: Do you have any comments.

Any screen that caused user issues:

Tell me more about your experience with this screen.

Coverage:

At what residence should college students be counted in the census?

Should babies be counted in the census?

If someone received this letter on March 26, but was going to move out and change residences on April 1^st, what should they do with this survey?

Did anyone you listed stay at multiple residences in the past year?

Tenure:

What does the phrase “include home equity loans” mean to you in this question?

If you were completing the Census at your house, do you think you would need to mail any information to the Census Bureau after reading this question?

Relationship in Spanish:

Take a look at the question: What does that question mean in your own words?

What if the survey asked instead, “Como esta emparentado _____ con usted?" What does that mean? (If necessary) Would your answer have changed?

Given those response choices, is “Como esta emparentado _____ con usted?" more clear than the original question?

What does the word “roommate” mean to you in this question?

Race:

What were you thinking about when you answered this question?

(Point to question stem if necessary). What does that question mean in your own words?

(If not already observed). What should a person do at this question if they were multi-racial, let’s say white and black and identified with both races?

(Be sure that you understand what motivated their action if they select 2 choices.) “What on the screen let you know you could do that?”

What does the word [race/ethnicity/origin/category] mean to you in this question?

Ethnicity/ancestry:

What were you thinking about when you answered this question?

Would your answer have changed if you had to type your entry instead of selecting the boxes?

Dashboard

What were you thinking after you returned to this the first time?

How about the second time?

How about when you wanted to turn in / finish up / submit your answers.

Any comments about the text here?

[Point to and wording text that may/may not be made on the dashboard, esp if it changes]

How about the order of the people listed here?

Address screen for non ID

Tell me more about your experience with this screen.

Do you have any comments on the way it is laid out?

Mapping (only if this was triggered during testing)

Tell me more about your experience with the mapping interface.

Mailing materials

Do you have any comment on these materials?

If this came to your home, what would your initial reaction be to receiving this in the mail?

Are there any indications to you that these materials are actually from the Census or does anything indicate that it could be from something other than a legitimate source? What are those things/indicators? Would you verify that the letter was legitimate before completing the survey? (If yes) How would you do that?

Now I am going to ask you a series of questions.

Do you speak a language other than English at home?

If yes,

A. What language?

B. Do you speak any other languages at home?

If yes, how did you decide which one to answer first?

Do you speak a language any other languages that you didn’t mention?

IF YES: What language(s)?

IF YES: Where or with whom do you speak them?

[Participants who only speak English]

Do you know anyone whose first language is a language other than English?

If yes: Ok, think of one specific person you know. *How well does that person speak English? Very well, well, not well, or not at all?

[Participants who speak another language]

*How well do you speak English? Very well, well, not well, or not at all?

Can you tell me more about why you answered the way you did?

When you answered how well (you speak)/(the person speaks) English, did you take reading into account?

When you answered how well (you speak)/(the person speaks) English, did you take their accent into account?

[IF R ANSWERED WELL AT PROBE *]

What would you/the person have to be able to do in order to answer “very well”?

ALL RS: For you, what is the difference between speaking English “well” and speaking “very well”? [IF NEEDED: What can people who speak ‘very well’ do that people who only speak ‘well’ cannot do?]

[Speakers of languages other than English][IF THEY REPORT ANY ENGLISH PROFICIENCY]

At what age did you start speaking English?

When a form is available in both English and [FILL LANGUAGE THAT PERSON MENTIONED ABOVE], in which language do you normally complete it? 

1. In [FILL LANGUAGE MENTIONED ABOVE]

2. In English

FOR Spanish speakers only

Today I’ve asked you to complete survey questions in Spanish, but I imagine that sometimes you receive surveys or government forms in English. When that happens, what do you normally do? [IF NEEDED: Would you look for a Spanish help number, try to find a friend or relative to help you, or just ignore it?]

Notes page: P#___

TA:______________________

Date: _____________________

Location: Headquarters / Other :________________________

Device: _____________________

WiFi or Dataplan

Browser type used:_____________________________________

Other browsers on device:________________________________

(Circle) ID / NonID-

# people:

Any multi-racial: Yes No

Any other residences: Yes No

Any edits:

Accessing URL :

TA assist: Y / N

Oral comments (circle): None Other: (Describe)

Re-entry:

Problems Y/ N : Describe

Used (circle): PIN or Verification question

Problems Y/ N

Email or phone number

Email provided Y / N Comments:

Phone # provided: Y / N Comments

Spontaneous comments on instruction (circle): None Confidentiality Other: (Describe)

Map (Non-ID)

Map triggered? Y / N Comments:

Roster

# hhld members:

Any errors:

Demographics:

Tenure and reference person answer:

Description of household:

Any comments on relationship

Any comments on combined race and ethnicity choices

Any comments on countries of origin under race

Any comments on the map if they enter an address

Vignette 1: DOB

Notes:

Administered it / Did not administer it / Happened during “live” survey

Entered mm= dd= yyyy=

Edit appeared: Yes/ No

Read edit: Y/N/ Not sure

Entered age / Age calculated / No age

Accessed help / Did not access help

Vignette 2: Toggle for Spanish language

Administered it / Did not administer it / Happened during “live” survey

What happened:

Vignette 3: Multiple addresses

Administered it / Did not administer it / Happened during “live” survey

What happened:

Vignette 4: Relationship edit check

Administered it / Did not administer it / Happened during “live” survey

What happened:

Vignette 5: Foreign-born parents/ U.S.-born child

Administered it / Did not administer it / Happened during “live” survey

What happened:

17