Cognitive Testing of Civic Engagement Supplement

15

Overview of Testing Plan

Cognitive Testing

Cognitive testing will be conducted by the Abt Associates Cognitive Testing Laboratory. Tasks conducted with respect to cognitive testing will be as follows.

1. Develop cognitive testing protocol and research informed consent form;

2. Obtain Abt Institutional Research Board approval for research conducted on human subjects;

3. Train cognitive testing interviewers on project;

4. Recruit respondents using outside vendor;

5. Conduct up to 30 video-recorded cognitive testing interviews of 60 minutes in length with subjects purposively recruited to provide diversity with respect to gender, age, education, household income, and household size;

6. Provide incentives in the form of $40 American Express gift cards to cognitive testing subjects;

7. Prepare interview summaries and analyze cognitive testing results;

8. Provide recommendation on suitability of changed/new Civic Engagement Supplement items for inclusion in November 2011 Current Population Survey;

9. Prepare a report of key findings and recommendations for revisions.

Analysis of Cognitive Testing

The data generated by cognitive testing will be analyzed in order provide feedback to the Census Bureau as to whether the instrument “works” and report on the cognitive testing.

Start Date

Cognitive Interviewing will begin on February 22, 2011. A preliminary report of findings will be submitted by March 8, 2011. This preliminary report will summarize the degree to which the instrument “works” based on cognitive testing. In addition, a final report summarizing the results of the respondent debriefing will be prepared.

Testing Protocol

Participant ID #: | | | | | |

Interview Date: | | | / | | | / | 2 | 0 | 1 | 1 | (mm/dd/yyyy)

Interviewer Initials: | | |

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Section 1: Interviewer: Read/ Paraphrase the following text:

Hello. My name is ________________. I work for Abt Associates, a research company that does work under contract mainly for federal agencies. Thank you for agreeing to participate in our study.

Human subjects requirement (prior to starting the recorder)

Confidentiality: Before we begin, I need to be sure you understand that our session today is completely confidential. Your participation in this study is completely voluntary and you can decline to answer any particular question.

Incentive: In appreciation for your participation, you will receive an American Express gift card for $40.

Recording: So I don’t have to rely on my memory later on, this session is being video recorded. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes.

Observers: Some members of our research team may be observing the interview.

** HAVE PARTICIPANT READ THE CONSENT FORM**

Before you sign this, do you have questions or see anything that is not clear?

** HAVE PARTICIPANT SIGN THE CONSENT FORM**

START RECORDING

Describe the interview objectives and procedures. (This should begin a conversational interaction with the respondent; it is not necessary to read these descriptions verbatim.):

Before I go into the details of what we’ll be doing, I’d like you to tell me what you were told about why you were asked to come in today.

(INTERVIEWER: Confirm respondent’s understanding if it is correct. If there are misconceptions, let the respondent know that you’ll clarify some things as you describe the plan for the interview session.)

In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys.

How: I want you to answer the questions exactly the way you would if an interviewer had come to your home for an interview, but with one major difference: I would like you to tell me your thoughts as you decide on your answers.

Think aloud: I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the thoughts that occur to you as you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

I don’t want you to try to plan what you say. Whatever you’re thinking as you decide on your answer is what I’d like to hear.

Usually, it helps to try this out once or twice before we get to the survey questions.

I’d like for you to think aloud as you decide how to answer the question: “How many windows are there in your house or apartment?”

(INTERVIEWER: Use this second practice question if necessary: “Thinking about yesterday, starting with the time you got up until you went to bed, how many phone calls did you make?”)

Probes: As we go through the survey, from time to time I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers.

I really want to hear your opinions and reactions, so don’t hesitate to speak up whenever something is unclear, is hard to answer, or doesn’t seem to apply to you.

Do you have any questions before we begin?

(INTERVIEWER: Wait for respondent to complete thinking aloud or commenting before asking the probes.)

Q1 In any election, some people are not able to vote because they are sick or busy or have some other reason, and others do not want to vote. Thinking about recent local elections, have you voted in all of them, most of them, some of them, or none?

(1) All of them

(2) Most of them

(3) Some of them

(4) None

Probe: What do you think of as being a “local election”?

Probe: What did you consider as being “recent” in this question?

Q2 I am going to read a list of things some people have done to express their views. Please tell me whether or not you have done any of the following in the last 12 months, that is between November 2010 and now:

(a) Contacted or visited a public official - at any level of government - to express your opinion?

(1) Yes

(2) No

(b) Bought or boycotted a certain product or service because of the social or political values of the company that provides it?

(1) Yes

(2) No

No probes. Preexisting questions.

(c) Gave money to a candidate, party, or political organization?

(1) Yes

(2) No

Probe: IF YES: Can you tell me about what gave to?

Probe: IF NO: What kinds of things do you think this question is asking about?

Q3 I’m going to read to you a list of different Internet technologies. Please say whether you have used each of these to express your opinions about political or social or community issues within the last 12 months:

(INTERVIEWER: Repeat as necessary: Have you used this within the last 12 months to express your opinions about political or social or community issues?)

(a) Email

(1) Yes

(2) No

(b) Text messaging

(1) Yes

(2) No

(c) Chat room

(1) Yes

(2) No

(d) A social networking site like Facebook, MySpace, or Twitter

(1) Yes

(2) No

(e) My own blog

(1) Yes

(2) No

(f) Commenting or writing on someone else’s blog

(1) Yes

(2) No

Probe: For selected items (or a subset): Can you tell me what you used [ ] for?

Generic follow-up probe (if necessary): What (other) kinds of [social / community] issues have you expressed opinions about?

Probe: For unselected items (or subset): Have you used [ ] for something other than giving your opinions about a political or social issue?

Q4 Have you watched an online video that supports or opposes a candidate, party, political organization, or public policy campaign in the past 12 months, that is between November 2010 and now?

(1) Yes

(2) No

Probe: Would you tell me, in your own words, what this question wants to find out?

Probe: What do you think an online video means in this question?

Q5 Next, I will give you a list of types of groups or organizations in which people sometimes participate. Have you participated in any of these groups during the last 12 months, that is between November 2010 and now:

(a) A school group, neighborhood, or community association such as PTA or neighborhood watch groups?

(1) Yes

(2) No

(b) A service or civic organization such as American Legion or Lions Club?

(1) Yes

(2) No

(c) A sports or recreation organization such as a soccer club or tennis club?

(1) Yes

(2) No

(d) A church, synagogue, mosque or other religious institution or organization, NOT COUNTING your attendance at religious services?

(1) Yes

(2) No

(e) Any other type of organization that I have not mentioned?

(1) Yes Continue

(2) No Skip to Q7

No probes. Preexisting questions.

Q6 What type of organization is that?

Record verbatim

No probes. Preexisting question.

Q7 In the last 12 months, between November 2010 and now, have you been an officer or served on a committee of any group or organization?

(1) Yes

(2) No

No probes. Preexisting question.

Q8 Have you worked on a community project in the past 12 months?

(1) Yes

(2) No

Probe: IF YES: Can you tell me a little about the project(s) you worked on?

Probe: IF NO: What sorts of things do you think the question means by “community project”?

Q9 Within the past twelve months, have you personally walked, ran, or bicycled or done anything else for a charitable cause? This is separate from sponsoring or giving money to this type of event.

(1) Yes

(2) No

Probe: IF YES: Can you tell me about what you did?

Follow-up probe (if not volunteered): What was the charitable cause?

Probe: IF NO: What kinds of events do you think this question is asking about?

Q10 In the last 12 months, have you volunteered for a political organization or candidates running for office?

(1) Yes

(2) No

Probe: IF YES: Can you describe what you did as a volunteer?

Probe: IF NO: What do you think the question means by “volunteered for a political organization”?

Q11 These next questions ask how often you did something during a TYPICAL MONTH in the past year. How often were politics discussed when communicating with family and friends, -- basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day Skip to

(2) A few times a week Q13

(3) A few times a month if one

(4) Once a month person

(5) Not at all household

No probes. Preexisting question.

Q12 How often did you eat dinner with any of the other members of your household -- basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

No probes. Preexisting question.

Q13 During a TYPICAL MONTH in the past year, how often, if at all, did you spend time visiting friends–basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

Probe: Would you tell me, in your own words, what this question wants to find out?

Q14 During a TYPICAL MONTH in the past year, how often, if at all, did you communicate with friends and family by email or on the internet -- basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

No probes. Preexisting question.

Q15 How often did you talk with any of your neighbors -- basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

No probes. Preexisting question.

Q16 How often did you and your neighbors do favors for each other? By favors we mean such things as watching each other’s children, helping with shopping, house sitting, lending garden or house tools, and other small acts of kindness -- basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

No probes. Preexisting question.

Q17 During a TYPICAL MONTH in the past year, how often, if at all, were you in the home of a neighbor–basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

Probe: How long have you lived at your current address?

Follow-up [if less than a year]: Did you think mostly about your current neighborhood, your last neighborhood, or both neighborhoods?

Probe: Would you tell me, in your own words, what this question wants to find out?

Q18 We’d like to know how much you trust people in your neighborhood. Generally speaking, would you say that you can trust them a lot, some, only a little, or not at all?

(1) A lot

(2) Some

(3) Only a little

(4) Not at all

Probe: (Unless the respondent answered “Not at all”) How did you decide on [choice] rather than [next closest option]?

Probe: What sorts of things come to mind when you hear “trust the people in your neighborhood”?

Probe: What do you think of as “your neighborhood”?

Probe: What comes to mind when you hear “people in your neighborhood”?

Q19 How many friends or family members do you have, if any, whom you could count on in an emergency? Don’t worry if your answer is not exact, just give me your best estimate.

Accept integers 0 through 9999

Probe: What sorts of things do you think the question means by “emergency”?

Probe: What does “count on” mean to you in this question?

Probe: How many of the [number >1 mentioned] are [friends/family]?

Probe: Do any live outside your neighborhood?

Q20 I am going to read some ways that people get news and information. Please tell me how often you did each of the following during a TYPICAL MONTH in the past year:

(a) Read a newspaper in print or on the Internet - basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

(b) Read news magazines such as Newsweek or Time, in print or on the Internet - basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

(c) Watch the news on television or get news from television internet sites - basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

(d) Listen to the news on radio or get news from radio internet sites - basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

(e) Obtain news from any other Internet sources that we have not previously asked about such as blogs, chat rooms, or independent news services - basically every day, a few times a week, a few times a month, once a month, or not at all?

(1) Basically every day

(2) A few times a week

(3) A few times a month

(4) Once a month

(5) Not at all

No probes. Preexisting questions.

Q21 I am going to name some institutions in this country. As far as the people running these institutions are concerned, would you say you have a great deal of confidence, only some confidence, or hardly any confidence at all in them?

[INTERVIEWER: Repeat the response options unless the respondent starts to answer before you read them; or if the respondent appears to forget the options.]

(a) Banks and financial institutions

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(b) Major companies

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(c) Small and local businesses

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(d) Organized labor

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(e) The print media, such as newspapers and news magazines

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(f) Electronic news media, such as television and cable news

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(g) Independent or citizen media, such as an online blog

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(h) The scientific community

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(i) Public schools

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

(j) Religious institutions

(1) Great deal of confidence

(2) Some confidence

(3) Hardly any confidence

Probe: Who would you think of as “the people running” the scientific community?

Probe: (Use especially if respondent usually chose the middle category) If you really didn’t have an opinion about the people running [insert from list that was answered “some”], how would you have answered?

Probe: What were you thinking when you said you had [response choice] in [insert from list]?

Probe: What do you think the question means by “citizen media”?

Probe: How would you answer about [insert from list] if you had a great deal of confidence in some of the people running [insert from list] but hardly any confidence in others?

Q22 How would you describe your overall state of health these days: Would you say it is excellent, very good, good, fair, or poor?

(1) Excellent

(2) Very good

(3) Good

(4) Fair

(5) Poor

No probes. Preexisting question.

Q23 On the following scale, how would you describe the quality of your life overall? Excellent, good, fair, poor, or very poor?

(1) Excellent

(2) Good

(3) Fair

(4) Poor

(5) Very poor

Probe: What did you think the question means by “the quality of your life overall”?

DEBRIEFING QUESTIONS

Overall, would say the survey questions were easy or difficult to answer?

Follow-up: Why?

Which questions did you have to think about the most to decide on an answer?

Follow-up: Why was that?

Which questions or topics were [most/least] interesting to you?

Do you think there are questions people would find difficult?

Follow-up: Which ones?

Follow-up: Why?

Do you think there are questions that people would find sensitive?

Follow-up: Which ones?

Follow-up: Why?