Various Demographic Pretesting Activities

Date_______________________; Participant #________; Experimenter:_______________

General Introduction: Measuring Question Difficulty on the American Community Survey Internet Instrument

Thank you for your time today. My name is Rachel Horwitz and I am a student here in the Joint Program of Survey Methodology and am also employed at the US Census Bureau in the American Community Survey Data Collection area. I will be working with you today. If you have a cell phone, please turn it off or put it in vibrate. In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys. I did not create the survey, so please share both your positive and negative reactions to it. The entire session should last between 45 minutes and one hour. Your comments and feedback will be given to the developers of the survey and may be used to improve it.

First, I would like to ask you to read and sign this consent form. It explains the purpose of today’s session and informs you of your rights as a participant. It also tells you that we would like to record the session, with your permission. Only those of us connected with the project will review the recording and any other data collected during the session, and it will be used solely for research purposes. We may also use clips from the recording to illustrate key points about the survey to the Web design team.

Hand the participant the consent form; give time to read and sign; sign own name and date if you have not already done so.

Start the tape.

While you are completing the survey, we will record the movements of your eyes with our eye-tracking monitor to get a record of where you are looking on the screen and we will record your mouse movements to see how you are interacting with the survey.

Now I am going to calibrate your eyes for the eye-tracking.

Do Calibration

A random five to ten respondents will take the survey using a think aloud procedure:

I would like you to tell me your impressions and thoughts about the questions as you read and answer them. I would like you to "think aloud" and talk to me about your impressions. If you expect to see some piece of information, tell me about that expectation as well. Finally, during the session, I may remind you to talk to me if you get quiet. I’ll do this not to interrupt your thought process, but simply to remind you to talk to me. Please focus on verbalizing what you are thinking as you read.

Before we get started, let's practice thinking aloud, since it's not something that you would normally do while reading. Can you tell me how many windows are in your house or apartment? [PROBE as appropriate to the participant's responses to this question.]

After think aloud practice is complete:

Now that we have your eyes calibrated, we are ready to begin. Please respond to the survey questions online. Please answer the questions as they apply to you in your real life.

Do you have any questions?

Leave room. Once in control room do a sound check and Start the eye-tracking software: Tobii Studio. The mouse tracing software will start when Studio opens Internet Explorer.

Do not offer any assistance to respondents or answer questions other than with “Please use your best judgment” or “how would you answer if you were at home?”

Overall Probe: Make a note if a person left a page with a blank answer, asked a question to the researcher, or displayed signs of confusion (hovers, regressions, using the mouse as a marker).

What was your overall impression of the survey?

Were there any questions you found to be difficult or challenging to answer?

If yes, show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

Were there any responses you were unsure of?

If yes show the respondents the relevant questions again and ask them to explain what was confusing to them and what they were thinking about while answering the question.

If respondents reported being confused by a question or displayed signs of confusion, ask them

What was it about this question that was confusing (understanding the question, understanding the response options, applying their situation to one of the response options)?

How did you come up with your answer?

For any questions where the respondent displayed signs of confusion, ask if they had any trouble answering the question and what type of trouble they had. If they were not sure which answer category to select, ask them to describe their situation.

If they mentioned that someone in their household uses more than one mode of transportation to get to work (such as bus and subway) and they chose one, ask why they chose that one.

For attended school in the past 3 months

What does attended school mean to you?

For work questions

Do you currently have more than one job?

If yes, did you answer about one of those jobs or all of them?

For Relationship Question

Why did you answer the relationship question the way you did?

Were there any other choices that you considered?

Do you think your answer adequately describes your situation? Why or why not?

For Marital Status Question

Why did you answer the marital status question the way you did?

If R reports now married:

• What was the date of your marriage?

• In what city and state did the marriage take place?

• Was this a commitment ceremony or a legal marriage ceremony?

If R reports registered domestic partner or civil union:

• What was the date of your domestic partnership/civil union?

• In what city and state did the registration take place?

Do you think your answer adequately describes your situation? Why or why not?

What does “registered domestic partnership or civil union” mean to you in Q. 23?

If necessary, ask respondents in opposite-sex relationships if they know anyone who might select they are in a domestic partnership or civil union and have them explain what type of living arrangement/relationship that was.

Thank you again for your participation today. It is greatly appreciated.

Participant#:

RACE: White Black Hispanic Asian or other Pacific Islander
Native American or Alaska Native

AGE RANGE: < 30 31-45 46-60 61+

GENDER: M F

EDUCATION: HS/GED Some Coll/AA Bachelor’s Some grad

DATE OF INTERVIEW:

INTERVIEWER:

4 | Page