Various Demographic Area Pretesting Activities

Usability Testing of ACS Version ACS-1(X)QD280

Protocol Guide

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2012

INTERVIEWER NAME:

START TIME: ______: ______ AM / PM

SECTION I. INTERVIEW CONSENT
	PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

A. Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Section I. End Time: _____ : _____ AM/PM

SECTION II. FILLING OUT THE FORM

A. Introduction

[SHOW RESPONDENT ACS FORM] This is a form called the American Community Survey (ACS), and is conducted by the Census Bureau.

• The Census Bureau counts the U.S. population and also conducts various kinds of surveys.

• The Census Bureau wants to see how well the ACS works. We want to hear your thoughts and opinions on this form.

• There are no right or wrong answers. We are simply trying to make sure the content of this form is clear and easy for most people to understand and use it.

• Your participation in this review of this survey is very important because it will help the Census Bureau improve the ACS.

• In a few minutes, we will ask you to fill out the form as if you were selected to participate in the American Community Survey.

• Do you have any questions before we begin?

B. Observation

INTERVIEWER: PLACE THE ACS QUESTIONNAIRE AND A PEN IN FRONT OF THE PARTICIPANT.

Let’s start filling out the form on your own. Now, imagine that you are participating in the ACS. Please complete the whole form from beginning to end as if you were doing it at home. If you had received this form at home, your address would be located on the front of the questionnaire in the white area. I will be listening, but will not be able to provide any feedback on what you are saying or doing at the moment. Again, this is NOT any kind of test and there are no right or wrong answers.

INTERVIEWER: AS THE RESPONDENT FILLS OUT THE FORM, MAKE THE FOLLOWING OBSERVATIONS ASSOCIATED WITH EACH SECTION BELOW. WHEN THE RESPONDENT FINISHES COMPLETING THE FORM, ASK THE PROBES ASSOCIATED WITH EACH SECTION:

INTERVIEWER OBSERVATIONS DURING RESPONDENT RECEIVES FORM:

• What did R do upon receiving the form? For example: What part of the form did R look at first? What did R do with it? Did R skim through it? Read it carefully? Open quickly then flip back and forth? Read carefully? Did R comment on the length of the form?

• What is your assessment of R’s emotional reaction to the ACS questionnaire? For example: Positive, negative, neutral? Did R exhibit confusion, frustration, relief?

GENERAL REACTION:

“Start Here” page and “Persons 1-12” pages (pages 1-4)

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R notice the Internet option? Did R say he/she would prefer to use Internet?

• Did R have problems with entering names or races in the write-in boxes?

• Did R have problems or make comments about the write-in boxes for numerical responses, like the phone number on page 1 and the birthdates?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

Housing Questions (Pages 5-7)

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R appear to understand skip instructions “A” through “E” for the Housing questions section and find the right questions/answers? If not, what instructions did the R find problematic? What was the R’s reaction?

• Did R seem to understand the difference between questions 14 a and b on PAGE 6 which ask about monthly utility costs and 14 c and d which ask about yearly utility costs? If not, does it appear that R reported monthly costs in 14 c and d?

• Did R have problems or makes comments about the write-in boxes for numerical responses, like month, year, or dollar amounts?

• Did R make any comment about write-ins - that aren’t numerical write-ins (like Question 9 and 11) - not having the individual white boxes for characters?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

INTERVIEWER: DURING THE DEBRIEFING, DIRECT R TO RE-READ SKIP INSTRUCTION “C” IN THE LEFT COLUMN ON PAGE 7. “Now, I would like you to read these lines again and let me know when you are done reading.”

POINT TO SKIP INSTRUCTION “C.”

• “Can you tell me what this instruction asks you to do?”

Detailed Person Questions for Persons 1 – 5 (Pages 8 – 27)

OBSERVE HOW R FILLS OUT THE FORM AND NOTE DOWN THE ISSUES OUTLINED BELOW:

• Did R appear to understand skip instructions “F” through “L” for the Detailed Person questions section and find the right questions/answers? If not, what instructions did the R find problematic? What was the R’s reaction?

• Did R have problems or makes comments about the write-in boxes for numerical responses, like year, school grade, or dollar amounts?

• Did R make comment about write-ins that aren’t numerical write-ins (like question 7, 12, 13, etc) not having the individual white boxes for characters?

• How many answer boxes did R select in Question 11? Question 41?

INTERVIEWER: DURING THE DEBRIEFING, ASK THE FOLLOWING QUESTIONS AS NEEDED TO FOLLOW UP ON EACH OF THE OBSERVATIONS ABOVE:

• “I noticed you were ...., can you please tell me what led you to do …?” IF NECESSARY, POINT TO THE INSTRUCTION.

INTERVIEWER: DURING THE DEBRIEFING, DIRECT R TO RE-READ SKIP INSTRUCTION “K” AT THE TOP OF THE MIDDLE COLUMN ON PAGE 14. “Now, I would like you to read these lines again and let me know when you are done reading.”

POINT TO SKIP INSTRUCTION “K.”

• “Can you tell me what this instruction asks you to do?”

SECTION III. FORM COMPARISON

INTERVIEWER: NOW SHOW THE RESPONDENT THE OTHER FORM (ACS 2013)

And now take a look at this other form [give a few moments for R to read ACS CONTROL FORM ].

What are some of the most noticeable differences between this and the form you saw first? Anything else?

[Show R the black answer boxes and black numerical write-in boxes on Page 6 of form 2013]. What do you think about the black answer boxes used in this form and the green answer boxes used in this form you filled out. What color boxes do you prefer: green or black? Why?

Between these two forms, which one do you like best? Why?

Debriefing questions

• OK, now that you have completed the questionnaire, what drew your attention when you first looked at the form?

• I noticed that you spent a lot more time looking through a particular part of the form [POINT TO THIS PART OF THE FORM]. Was there any particular reason why?

• I notice you were [laughing/frowning/sighing…] when you saw _____. What were you thinking about at that point? I noticed you skipped this part/page, what made you do so?

	INTERVIEWER: NOW GO BACK AND ASK FOLLOW UP QUESTIONS AND PROBES BASED ON THE OBSERVATIONS YOU MADE, STARTING WITH OBSERVATIONS ON “START HERE” AND “PERSONS 1-12 PAGES.” REITERATE THAT THERE ARE NO RIGHT OR WRONG ANSWERS AND THAT THIS IS A TEST OF THE FORM AND NOT THE RESPONDENT.

• Can you tell me if it was easy or difficult it was for you to fill out the answers in the questionnaire? Why?

• [IF NOT PROBED ON OR DISCUSSED PREVISIOSLY] Was writing in numbers in the number write-in boxes easy or difficult?

• Which parts were most confusing or problematic?

• What do you think of the entire experience/process?

• Do you have any final comments or questions?

I want to thank you very much for your participation. I will now give you $40 and I will ask you to sign a receipt form verifying that you received the money.

INTERVIEWER: TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT.

End Time: ____________ AM / PM

INTERVIEWER ONLY: ANSWER THESE QUESTIONS AFTER THE INTERVIEW.

N_Debr3. HOW DIFFICULT WAS IT FOR YOU TO CONDUCT THE INTERVIEW WITH THIS RESPONDENT?

1 Very difficult 2 Somewhat difficult 3 Somewhat easy 4 Very easy

N_Debr4. WHAT FACTOR DO YOU THINK ATTRIBUTE TO THIS RESPONDENT’S COGNITIVE INTERVIEW MOST?

1 Age 2 Education 3 Others (Specify: _____________________________________)

CSM Project 10-26-2012