Various Demographic Pretesting Activities

Cognitive Interview Protocol

Interviewer: Through the probing, you are trying to uncover any source of response error with respect to the reporting of sources of income, and amounts of income – especially with respect to under-reporting, known to be a major problem in the ASEC.

During administration of the ASEC questions, whenever you observe any potential difficulty in understanding a question or generating a response, probe to determine as specifically as possible what the issue is. Some generally useful probes for this are:

• I noticed it took you a minute to answer that one – can you tell me about that?

• What all did you think about as you were coming up with your answer?

• So when you answered with _______ , can you tell me how you arrived at that answer?

• What is it about that question that makes it difficult to answer?

Probes Regarding Income Sources

These probes should be asked immediately after finishing the questions determining the sources of income for the household, and before any income amounts are collected.

• So let’s pause here and review some things. First let’s do something different. Try to forget about all of these questions I asked you for a moment. Just tell me in your own words the story of your household income situation last year – where the money came from, and who it came to.

  • Anything else? (Did you/NAME get money from anywhere else?)

  • Were there any times where you/NAME got money just for a very short time? (What was that and how much?)

  • In addition to these regular (weekly/monthly) sources of money, was their money coming in on an irregular basis? Perhaps only once or twice over the whole year, for example? (What was that?

• As applicable – probe to determine why any income sources were not reported when asked in the ASEC: One of the questions I asked you was _______ . Can you tell me why you answered “no” to that question? (etc.)

PROXY REPORTING (IF APPLICABLE)

• This survey, in addition to asking about yourself, asked you about the income of others in your household (NAMES). How well do you know their situations when it comes to where their money comes from, and how much?

• Were there any times during the interview when you said (NAMES) did not receive any income from a place last year, but it would be more accurate to say you don’t really know? (Tell me about that – what sources were those?)

IF RESPONDENT IS A LOW INCOME RECRUIT (IN PUBLIC HOUSING AT LEAST PART OF LAST YEAR):

PUBLIC ASSISTANCE/TANF

• One of the questions I asked you was “At any time during 2010, even for one month, did (you/ anyone in this household) receive any CASH assistance from a state or county welfare program such as (STATE PROGRAM NAME)?” As you understand it, what is that question asking about?

• What do you think is meant by “cash assistance?”

• Do you recognize [STATE PROGRAM NAME]?

• Is there another name you or others here have for that program?

• [IF TANF NOT REPORTED IN ASEC] Did you or others in your household get any kind of assistance like this in 2010 – even if for only a very short time? (IF YES: What was it? How is that different from what this question was asking about? When did you first get it? What months did you receive it during last year?) (IF NO: Did you apply for it? What exactly did you apply for? Where did you go to apply? What did they tell you?)

• [IF TANF WAS REPORTED IN ASEC] When did you first get it? What months did you receive it during last year? How is this assistance given to you?

FOOD STAMPS/SNAP

• One of the questions I asked you was “Did (you/ anyone in this household) get food stamps or a food stamp benefit card at any time during 2010? As you understand it, what is that question asking about?

• What do you think is meant by “food stamp benefit card?”

• Is there another name you or others here have for food stamps or this card?

• (IF APPLICABLE) Another question I asked was “At any time during 2010, even for one month, did (you/ anyone in this household) receive any food assistance from (State Program name)? As you understand it, what is that question asking about?

• What do you think is meant by “food assistance?”

• Do you recognize [STATE PROGRAM NAME]?

• Is there another name you or others here have for that program?

• [IF FOOD STAMPS NOT REPORTED IN ASEC] Did you or others in your household get any kind of assistance with buying food in 2010 – even if for only a very short time? (IF YES: What was it? How is that different from what this question was asking about? When did you first get it? What months did you receive it during last year?) (IF NO: Did you apply for it? What exactly did you apply for? Where did you go to apply? What did they tell you?)

• [IF FOOD STAMPS WERE REPORTED IN ASEC] When did you first get your food stamp benefit? What months did you receive it during last year? How is this assistance given to you?

Probes at End of ASEC Interview

OVERALL IMPRESSIONS

• So those are all of the questions in this survey we are trying to improve. What was your overall impression of the questions in it?

• What stands out, if anything, as especially difficult to answer?

• Did any part of it make you uncomfortable?

ALL SOURCES FIRST, THEN ALL AMOUNTS

• In this survey we first asked whether or not you (and others in your home) had received income from a variety of things – just getting “yes or no” answers to each. Then, for each where you had answered “yes” we asked for the amount. Do you have any reactions to that?

• Suppose we had done something different – ask for the amount of income received last year immediately after you “answered” yes for a given source, rather than going through all the sources first. Do you think that would be better or worse than the way it was done here? (Why is that?)

IF R HAD INVESTMENTS IN STOCKS/MUTUAL FUNDS:

• How did you feel about the survey asking you for the value of your stocks or mutual funds? (Is asking for that information any more personal or intrusive than the other questions in this survey?) Note: we are trying to assess the sensitivity of asking about the values of these assets.

• Which is easier for you to answer – questions about the value of your stocks and mutual funds at the end of last year, or the amount of dividends you received from them these investments last year? Which answer do you believe is more likely to be accurate? (Why is that?)

• We asked you about the value of these investments and the dividends they generate separately for retirement accounts and nonretirement accounts. What is your feeling about that? (How easy or difficult is it to report these amounts separately? Would it have been any more easy or difficult to report them together in a combined amount?)

SOCIAL SECURITY AND DISABILITY INCOME

• When did you/NAME receive the first social security/disability payment in 2010? What was the amount of that first payment? (Was it larger than later payments received? Did you include that larger amount in the amount you gave me, which was [$TOTAL]? (Can you tell me why you did not report that amount?)

• [IF PAYMENTS RECEIVED FOR ONLY PART OF 2010] So you/NAME got this income for only part of the year. So in responding to a survey like this, how easy or difficult is it come up with an amount received for only part of the year?