REPORT: CPS ASEC Cognitive Testing

The Annual Social and Economic Supplement to the Current Population Survey (the CPS-ASEC) is one of the most widely-used of all the surveys conducted by the U.S. government. The official poverty statistics – announced each fall for the prior calendar year – are derived from the CPS ASEC. The survey gives researchers a unique combination of features – detailed demographic data, detailed income data, data on a wide range of benefit programs, sufficient sample size at a state-level, cross-year consistency, and very timely data production.

As noted in the Census Bureau RFP, the ASEC data reflect errors of varying magnitudes across income types, and by particular subpopulations. For example, overall, enrollment in benefit programs is under-reported (as is the case in virtually all surveys). Even after Census Bureau imputations for missing data, enrollment in public benefit programs is under-reported by approximately 26 percent (SSI) to 43 percent (food stamps) (Wheaton, 2007). Additionally, the ASEC data also require a substantial level of imputation and allocation. Specifically, 35 percent of Social Security income, and 63 percent of asset income is allocated in the ASEC (Czajka and Denmead, 2008). The Census Bureau has begun a research program to investigate methods to address misreporting and item nonresponse by taking greater advantage of the functionality and flexibility afforded by computer assisted interviewing (CAI) within the ASEC interview.

This report covers two rounds of cognitive interviewing completed using a modified version of the ASEC instrument content. The first round reflects modifications addressing the data quality issues noted of concern directly by the Census Bureau, as well as specific findings from the literature review completed as part of this contract by Urban Institute and Westat (delivered to BOC December, 2010). The second round of cognitive interviews used another version of the ASEC instrument content that built upon the results of the first round of cognitive interviews, as well as incorporated the findings from the Data Analysis Report, a deliverable completed as part of this contract by the Urban Institute (delivered to BOC in April of 2011).

The overall goal of this initial testing was to develop a viable alternative approach for collecting the ASEC data that will be further tested in the future using an actual CAPI/CATI instrument, and in comparison to the current ASEC instrument. Thus, our analysis focused on assessing response errors associated with the instrument as designed for these two rounds of testing.

In general, across rounds, the redesign efforts focused on addressing:

comprehension errors resulting from differences in the specific language used to define income sources in the ASEC relative to the language naturally used by sampled participants, across geographic areas
recall errors associated with income sources received at irregular or low frequencies, or received in only small amounts
reporting errors associated with income sources perceived as having an attached social stigma with receipt, such as public assistance and food stamps
item nonresponse, either as a result of proxy reporting (i.e., a household respondent reporting for all adult household members), or lack of knowledge about income source.
errors resulting from respondent fatigue
errors reflective of changes in the structure of income sources, such as TANF eligibility, retirement accounts and other assets, since the last ASEC redesign in the early 1990s.

These two rounds of cognitive testing, as well as the literature review and data analysis report, serve as the initial components of the Census Bureau’s larger testing and development effort for the CPS ASEC instrument. Since this represents the earliest stages of testing, all modifications were developed and tested using a paper interviewer-administered instrument. Future testing will develop alternative versions of the ASEC instrument in the Census Bureau’s Blaise environment.

Methods

High-Level Description of Methods Common to Both Rounds

Both rounds of cognitive testing used the same general interviewing approach, though each round used a slightly different version of an alternate ASEC interview. (As noted above, the second round instrument reflected updates based on the results of the first round, as well as results of the data analysis effort not available prior to the first round of testing.) All interviews started with a review of the consent form, which included the request to audio record the interview (see Attachment A). After discussion of any questions raised by the respondent and getting the respondent’s signature, the interviewer proceeded to explain the task to the respondent, noting that Westat is evaluating some possible changes to the Current Population Survey - Annual Social and Economic Supplement interview for the U.S. Census Bureau. Respondents were told that at different points in the interview, the interviewer would stop and ask questions to get his/her feedback on some of the ASEC questions, explaining that their feedback gives a lot of valuable information about how well the questions work. Interviewers also emphasized that respondents did not have to answer any questions they didn’t want to, noting a greater interest in how they interpret the questions and how they come up with an answer rather than the actual answer. All interviews in each round started with the work experience questions before moving into the questions about income sources. All participants received $40 in appreciation for their time.

The same cognitive interviewers conducted the interviews in each round. Each interviewer participated in a detailed training session prior to each round. In addition to training on the instrument and procedures for cognitive testing, interviewer training in the first round also covered the current ASEC interview and data collection protocol to provide a better general understanding of the ASEC survey environment. Prior to starting their cognitive interviews, each interviewer was instructed to conduct a least one practice interview to gain familiarity with the various components of the interview and associated materials (these are discussed in more detail in design and procedures section for each round).

All interviews were conducted in focus group facilities or similar types of facilities, with the recruited respondents traveling to the facilities to complete the interview. Conducting the interviews in a central location allows completion of several interviews in the course of the day since interviewers do not need to commute from one location to another. However, the disadvantage for the ASEC testing is that respondents did not have access to their financial records, another departure from the actual ASEC interview situation. Interviewers attempted to address this by probing at the end of the interview on the types of records respondent had at home and whether or not they felt they would use records to respond to this interview. However, these probes are hypothetical in nature, and responses may not reflect actual behaviors in a production ASEC interview situation.

Interviews were conducted in the Spring of 2011, in a total of 9 different states across two rounds. In the first round, we completed 28 interviews in the following 4 locations:

San Francisco, CA
Columbia, SC
New York, NY
Jacksonville, FL

In the second round, we completed 29 interviews across the following 5 locations:

Minneapolis, MN
Columbus, OH
Nashville, TN
Alexandria, VA
Edison, NJ

Recruiting Cognitive Interview Participants

BOC and Westat agreed that three income source areas would be the primary focus for the cognitive interviews: 1) TANF and food stamps, 2) asset income, including assets held within both retirement and non-retirement accounts, and 3) Social Security, including receipt for non-retirement reasons and partial year receipt. Thus, we focused our recruiting efforts on finding and scheduling persons with characteristics deemed relevant for testing the ASEC items on these areas.

For Round 1, we focused on recruiting persons in the following four groups:

Persons who lived in subsidized (public) housing or received Section 8 housing assistance in 2010. We believed such persons would be especially likely to have received TANF and food stamps;
Persons who had investments in the stock or bond markets in 2010, including assets in both retirement and non-retirement savings);
Persons who are age 62 or older and retired sometime in 2010. Spouses of such persons were eligible as well. We expected such persons to be partial-year recipients of Social Security;
Persons who received disability-related government financial support for at least part of 2010. Spouses of such persons were eligible as well.

Some of the 28 Round 1 participants met more than one of these characteristics. Altogether, we interviewed 8 persons receiving public housing assistance, 15 persons with stock/bond investments, 4 persons who retired in the previous year, and 13 persons who received disability payments for at least part of the year.

Round 2 had a very similar focus. Specifically, we targeted:

Low income persons with children. Since none of the Round 1 participants appeared to have received TANF in 2010, we tried to recruit low income persons by contacting local TANF offices in each of the five interview locations, with mixed success. TANF offices in some locations would not assist us whereas those other locations did. If persons could not be obtained through local TANF offices, recruiters targeted persons receiving Section 8 public housing in 2010;
Persons who had a retirement account (such as a 401(k) or IRA) during 2010, including a mix of persons currently retired and persons of high income who are still working;
Persons who received disability-related government support in 2010, including some who received it for only part of the year.

Some of our 29 participants in this round met more than one characteristic. Altogether, we interviewed 15 low-income persons, 12 persons with retirement assets, and 13 who had received disability payments last year.

Table 1 summarizes the demographic characteristics of the cognitive interview participants, by round.

Table 1: Demographic Characteristics of Cognitive Interview Participants

		Round 1	Round 2
Gender
Male	12			9
Female	16			20

Education
High school or less	5			9
Some college	12			10
College graduate	6			7
Advanced degree	5			3

Age
20 to 39	5			8
40 to 59	7			12
60 or older	16			9

Race/Ethnicity
White	16			17
Black	11			8
Hispanic	-			3
Asian	1
Other				1

First Round Cognitive Interview Design

The first round of cognitive interviewing assessed several high-level structural changes to the design of the ASEC instrument. The structural changes, in general, reflect increased usage of the potential functionality provided by Computer Assisted Interviewing technology. The main objectives of these structural changes were to: 1) reduce under-reporting of income sources for self and proxies, 2) reduce the effect of fatigue on response quality and, 3) reduce the level of item missing data particularly in regards to reports of specific income amounts.

Specifically, the first round cognitive interview instrument examined the following structural changes:

Reduce underreporting by collecting all sources of income for a household prior to asking for any amounts. Conventional wisdom in survey research purports that using an interleafed design (e.g., income source 1, amounts for source 1, income source 2, amounts for source 2, and so on) provides respondents the opportunity to learn the level of follow-up asked about each identified income source early in the interview. As a result, this increased awareness of the ‘consequence’ of reporting receipt of an income source may negatively affect reporting of sources of income that fall later in the interview. In fact, work by Kreuter et al (2010) found evidence of this occurring for some survey topics, though the finding did not universally apply to all topics included in the study. With this in mind, the Round one cognitive interview instrument restructured the collection of all income sources first before proceeding to collect the amounts received by household members for each source. This structural change aims to reduce under-reporting of income, particularly for those sources that fall later in the interview.

To assess this change, probes were built into the cognitive interview to identify if any sources of income were missed. Additionally, interviewers were instructed to observe and probe as necessary any respondents who reacted in any way to the transition from income sources to collecting detailed amounts, or if they observed any reporting difficulty in connecting the earlier reported source of income to the questions about the amount of income received for that source.

Reduce fatigue effects by using a tailored order of presentation of income sources, depending on known characteristics of the household either from previous waves of data collection or from the core items asked earlier in the interview. For cognitive testing, Westat used screening information collected as part of participant recruitment to determine the presentation order for income sources. The goal of this manipulation was to reduce potential fatigue effects on reporting, moving the questions about the most likely source of income for that respondent earlier in the interview. Round one used three different tailored orders of income presentation.

As with the previous change, probes were built into the cognitive interview to identify if any sources of income were missed under this tailored approach. In addition, interviewers paid attention to the occurrence of reports of income sources shifted to later in the interview for a specific tailored presentation order.

Reduce item missing data by using bracketed amounts as follow-up to initial “don’t know” responses to the income amounts questions, with brackets defined by geography and income type. The objective of this manipulation was to reduce item nonresponse by allowing respondents to instead provide a less precise response to questions about the amount of income received for a particular income type. Respondents may simply choose to respond with a “don’t know” response rather than provide a point estimate that they do not feel confident about. In addition, as discussed by Tourangeau et al (2000), changing from an open recall response task to a recognition task by selecting from a provided dollar range can facilitate recall and reporting. Westat evaluated this modification by noting instances in which respondents selected a bracketed range, after an initial “don’t know” response. Similarly, interviewers noted and probed on instances when respondents did not provide an answer to either the point estimate request nor from the bracketed ranges.

In addition, the round one cognitive interview instrument also included several changes to the collection of asset income identified by the Census Bureau as a concern in terms of data quality.

Added questions to collect the asset value (e.g., the account balance for a mutual fund or savings account). While the CPS is not necessarily interested in the total value of an asset, having the total value would allow for more informed imputation of dividends and interest amounts. In round one, about half the interviews asked for the total asset value prior to asking for interest or dividend amount. The other half maintained the current ASEC approach and asked for the interest or dividend amount, but then followed up with a request for the account balance.

The analysis of this change focused on identifying the respondents’ relative awareness of their account balance as compared to the dividend amounts or interest earned. In addition, interviewers observed and probed as necessary any sensitivity to reporting the asset value. The varying order of total value versus interest/dividend amount provided a way to assess the relative sensitivity of the total value request.

Separated reporting of retirement accounts/assets (e.g., IRA, 401k, 403k accounts, etc) from other assets. This change was introduced to better facilitate reporting of retirement income sources, other than pensions, and better reflect of the changes in retirement planning and savings approaches of today. Currently, ASEC makes no distinction between investment income received in a retirement account and investment income received outside of a retirement account. It may be easier for people to consider and report them separately. Interviewers assessed respondents’ ability to separate retirement and non-retirement accounts, and for those with both types, probed on how respondents maintain records of that information, how they think about the accounts and the easiest way for them to report such income.

Round one testing also addressed BOC concerns with reporting of Social Security and SSI income, as well as under-reporting of means tested assistance programs (public assistance or TANF and SNAP).

Issues with respect to reporting back payments for Social Security and SSI Income. People who receive either Social Security or SSI may experience a delay in receiving their initial payment. It is not uncommon for the first payment in either of these programs to cover several months prior to the first month of receipt, while subsequent payments cover one month each. In this first round, recruitment targeted respondents whose first payment for either of these programs occurred at some point in the previous calendar year (i.e., the ASEC reference period) in order to understand how respondents naturally think about and report these initial ‘back payments’ within the current ASEC language for collecting this income information. While the ASEC question wording remained unchanged, interviewers asked probes that identified if a back payment occurred, if respondents reported that back payment or not, and if not, why not.

Issues with respect to reporting public assistance and food stamps. As identified by the BOC in their RFP, in the literature review, and later in the data analysis report, public assistance and food stamp income is chronically under-reported in the ASEC. The literature review results suggest that under-reporting is exacerbated among partial year recipients within either of these programs. In round one, recruiting efforts targeted those receiving public housing as likely recipients of one or both of these programs, but the question wording and structure for collecting these income sources remained the same as currently in the ASEC instrument. The only modification made was to drop the use of the family income questions used to determine which households get asked about public assistance, food stamps and other means tested programs. (In fact, the Data Analysis report provided evidence that this question likely inappropriately screens out households eligible for and participating in one or more of these programs¹.) All respondents, regardless of what we knew about their household income level from recruitment and screening, received the recipiency question for these programs.

Interviewers asked in-depth probes to understand whether in fact respondents or anyone in their households actually received income from either of these sources, the language or terms the respondents naturally use in reference to these programs, the types of cues salient to respondents in thinking about and discussing these programs and knowledge of the amounts received, if any. Westat planned on using the results of this probing to inform any needed language or cueing changes for the second round of interviewing.

Round one interviews included 5 different interview versions. The five versions reflect the three different orders of presentation of income types, and the two different orders for asking total asset value – either before or after asking for interest/dividend amount. (Attachments B1 through B5 show each interview version.) Table 2 below shows the three different orders of income. Cognitive interview respondents fell into one of the three orders based on the following criteria.

Low-income: Respondents recruited as receiving housing assistance of any type, including living in section 8 housing, received this version of the instrument. Housing assistance was used as a proxy indicator for public assistance as we did not want to ‘preview’ the ASEC questions and concepts of interest in recruiting participants. The actual ASEC can make use of employment information from prior waves, household size, the presence of dependent children in the household and address/geography to determine whether a household should receive this presentation order.
Seniors: Respondents who identified themselves as 62 or older in screening received this order. The actual ASEC can make use of employment information from prior waves (e.g, retired) as well as age as a determinant of eligibility for this order.
All others: This was the default condition if the respondent did not meet either of the above criteria. The order used here closely reflected the existing ASEC income order. However, since the cognitive testing instruments did not include the Family Income screener, all income sources were asked of all respondents.

The protocol that included both the global and specific probes for Round One interviews is included as Attachment C.

Table 2: Round One Alternate Orders of Income Presentation.

Low Income	Seniors	All Others
Unemployment and Workers Compensation	Social Security	Unemployment and Workers Compensation
Public Assistance / TANF	Supplemental Security Income (SSI)	Social Security
Food Stamps (SNAP)	Veterans	Supplemental Security Income (SSI)
WIC	Survivors	Public Assistance
School Lunches	Retirement and Pensions	Food Stamps (SNAP)
Social Security	Interest	WIC
Supplemental Security Income (SSI)	Dividends	School Lunches
Veterans	Property	Veterans
Survivor Benefits	Unemployment and Workers Compensation	Survivor Benefits
Disability Income	Public Assistance / TANF	Disability
Retirement and Pensions	Food Stamps (SNAP)	Retirement and Pensions
Interest	WIC	Interest
Dividends	School Lunches	Dividends
Property Income	Disability	Property Income
Education Assistance	Education Assistance	Education Assistance
Child Support	Child Support	Child Support
Alimony	Alimony	Alimony
Financial Assistance from friends or relatives	Financial Assistance from friends or relatives	Financial Assistance from friends or relatives
Other Income I (as in ASEC now)	Other Income I (as in ASEC now)	Other Income I (as in ASEC now)
Other Income II (as in ASEC now)	Other Income II (as in ASEC now)	Other Income II (as in ASEC now)

In order to better reflect the capabilities of an automated instrument while actually using a paper interview instrument, interviewers used a structured notes page to record each type of income received by household member (See attachment B6). Interviewers referred to this note page to determine which income amount questions to ask for which household members. In addition, interviewers had a calculator available to sum non-annual amounts of income (e.g., weekly, monthly, or quarterly) reported in order to feed back a total amount for respondent verification. The note page and the calculator will not be necessary in future stages of testing with an automated version of the instrument.

Second Round of Cognitive Interview Design

Updates to the structural changes used in Round One

In Round 2, we retained and continued testing the same three structural changes included in round 1 to get a larger set of responses. Specifically, Round Two interviews continued collecting all sources of income first prior to collecting amounts as a method to reduce under reporting of income types; used a tailored order of presentation to reduce potential fatigue effects on reporting, and; included bracketed amounts as follow-up questions to don’t know responses as a way to reduce item missing data.

Round One results (discussed in detail in the Findings section) did however suggest some additional modifications in the implementation of the source first approach and the specific ordering of income types. In general, the link between the source of income received and the amount of that income received was not always maintained with the collection of all sources of income first. This primarily reflects the fact that interviewers did not have the benefit of an automated instrument to store and reference an income source name or label once he/she began asking about the amounts received for each income source. Interviewers needed to maintain this link themselves, which proved difficult when respondents identified multiple sources of income or different sources for different household members. In addition, without the immediate context of the source question itself, the labels or terms captured in the ASEC for a source were often not specific enough to cue unambiguously the correct source of income when returning to that income type in responding to the amount questions. This was particularly problematic in the Interest and Dividends sections. In order to address these issues, we designed the Round 2 instruments to use more specific cuing within the asset reporting, but also for all other income source that include sub-sources (Disability, Pensions, Survivor Benefits).

Using these detailed cues for each source meant that respondents receiving a given income type would need to respond to a few more individual questions within that income type. As a way to reduce the perceived burden with this approach, we re-ordered the presentation of the detailed cue questions within each income type to better reflect the proportional distribution of these income sources, at a national level, from highest to lowest. (However, with an automated instrument, these orders can potentially be tailored to the state level distribution.) We also limited the specific questions for a detailed source within each income type to only the top 5 dollar amounts (ranked at a national level) as a way to minimize burden.

As examples of the changes between rounds in the level of detailed cues within an income source, table 3a shows the differences between the source questions for disability income by round, and table 3b shows the difference by round in the survivor benefits source questions.

Table 3a: Disability income ‘source’ questions by cognitive interviewing round

Round 1	Round 2
Did you receive any income in 2010 as a result of (your/his/her) health problem (other than Social Security/other than VA benefits/ Other than Social Security or VA benefits)?	Did (you/name) receive any disability income in 2010 as a result of (your/his/her) disability or health problem?
(If YES) What was the source of that income? (Response categories not read) Workers compensation Company / union disability Federal government disability U.S. Military retirement disability State or Local government employment disability U.S. Railroad retirement disability Accident or disability insurance Black Lung miner’s disability State Temporary sickness Other or don’t know (specify)	(If YES) Was this disability income from… Social Security Disability Insurance, for people who are eligible based on years of work? (Y/N) Supplemental Security Income, which provides payments to low-income aged, blind and disabled persons? (Y/N) Company or union disability payments? (Company/Union/None) Federal, State or local government employee disability? (Federal/State/Local/None) Accident or disability insurance? (Y/N) Did you receive disability income from any other source?
	(If YES to ‘f’) What was this source of disability income? Workers compensation U.S. Military retirement disability U.S. Railroad retirement disability State temporary sickness Black Lung miner’s disability Other or don’t know (specify)

Table 3b: Survivor benefit ‘source’ questions by cognitive interviewing round

Round 1	Round 2
Did (you/anyone in this household) receive any survivor benefits in 2010 such as widow’s pensions, estates, trusts, insurance annuities, o rany other survivor benefits (other than Social Security/ other than VA benefits/ other than Social Security or VA benefits)?	Did (you/anyone in this household) receive any survivor benefits in 2010 such as widow’s pensions, estates, trusts, insurance annuities, o rany other survivor benefits (other than Social Security/ other than VA benefits/ other than Social Security or VA benefits)?
(If YES) What was the source of that income? (Response categories not read) Company or union survivor pension (INCLUDE PROFIT SHARING) Federal government survivor pension U.S. Military retirement survivor pension State or Local government survivor pension U.S. Railroad retirement survivor pension Worker’s compensation survivor pension Black Lung survivor pension Regular payments from estates or trusts Regular payments from annuities or paid-up insurance policies Other or don’t know (specify)	(If YES) Was (your/name’s) income from… A company or union survivor pension (include profit sharing)? (Company/Union/None) Federal, State or local government survivor (civil service) pension? (Federal/State/Local/None) U.S. Military retirement survivor pension? (Y/N) Regular payments from estates or trusts? (Y/N) Regular payments from annuities or paid-up insurance policies? (Y/N) Did you receive survivor benefits from any other source?
	(If YES to ‘f’) What was the source of this income? Social Security survivor payments Black Lung survivor pension U.S. Railroad retirement survivor pension Worker’s compensation survivor pension Other or don’t know (specify)

The specific cues captured for each income type were then used as fills in the amount section for the income type. For example, if a respondent indicated that she received a U.S. Military retirement survivor pension as well as payments from annuities, the survivor income amounts section asked for the amount she received from the U.S. Military retirement survivor pension, followed by a separate question for the amount received from the annuity.

With the addition of these detailed level cues within each income source, it became apparent that several of the income types shared the same or very similarly named detailed sources of income. For example , Disability income includes a probe for Social Security Disability Insurance, which could easily be confused with the higher level screening question for Social Security as a source of Income. Similarly, both Pensions and Survivor Benefits include a probe for company or union (survivor) pensions. In order to reduce the potential confusion and possible duplicate or misreporting across income types, Round Two also changed the order of presentation slightly for each of the three categories used in Round one. Table 4 shows the revised source type order groups used in Round Two.

Table 4: Round Two Alternate Orders of Income Presentation.

Low Income	Seniors	All Others
Unemployment and Workers Compensation	Disability	Unemployment and Workers Compensation
Public Assistance / TANF	Social Security	Disability
Food Stamps (SNAP)	Supplemental Security Income (SSI)	Social Security
WIC	Veterans	Supplemental Security Income (SSI)
School Lunches	Survivors	Veterans
Public Housing	Pensions	Survivor Benefits
Energy Assistance	Annuities	Public Assistance
Disability	Retirement Accounts – Withdrawals or distributions	Food Stamps (SNAP)
Social Security	Other Income Earning Assets (outside of retirement)	WIC
Supplemental Security Income (SSI)	Property Income	School Lunches
Veterans	Unemployment and Workers Compensation	Public Housing
Survivor Benefits	Public Assistance / TANF	Energy Assistance
Pensions	Food Stamps (SNAP)	Pensions
Annuities	WIC	Annuities
Retirement Accounts – Withdrawals or distributions	School Lunches	Retirement Accounts – Withdrawals or distributions
Other Income Earning Assets (outside of retirement)	Public Housing	Other Income Earning Assets (outside of retirement)
Property Income	Energy Assistance	Property Income
Education Assistance	Education Assistance	Education Assistance
Child Support	Child Support	Child Support
Alimony	Alimony	Alimony
Financial Assistance from friends or relatives	Financial Assistance from friends or relatives	Financial Assistance from friends or relatives
Other Income I (as in ASEC now)	Other Income I (as in ASEC now)	Other Income I (as in ASEC now)
Other Income II (as in ASEC now)	Other Income II (as in ASEC now)	Other Income II (as in ASEC now)

In addition, we modified the interviewer note sheet which interviewers used to record the reported the income types per household member from round one. We added a space to record the specific term or terms used by respondents in identifying each particular source of income received. The interviewers could then use the specific or detailed income source received as the cue in the subsequent amount section. (See Attachment D4).

Updates to Specific Income Types

In addition to the updates to the structural modifications, the Round One results and the Data Analysis Report both suggested additional updates to specific income types for Round Two testing.

MEANS-TESTED BENEFIT PROGRAMS

As discussed in more detail in the Recruiting Section below, Round One did not result in the successful recruiting of TANF recipients and only a limited number of SNAP recipients. Thus modifications to TANF and SNAP collection in Round Two are based almost entirely on the Urban Institute Data Analysis Report. In particular, the Data Analysis report demonstrated that in regards to Food Stamps (SNAP) and SSI, the weighted caseload totals are about two thirds of the target derived from administrative totals, indicating ASEC misses recipients of each program benefit. However, the total dollar amounts are slightly higher, roughly seven tenths of the targets suggesting that perhaps those who receive higher levels of benefits may be more likely to remember those benefits and report them. This is particularly likely if those who receive a benefit for fewer than 12 months of the year are less likely to report any benefit, particularly in the case where a benefit was received early in the calendar year, in some cases more than 12 months prior to the interview. The Round One cognitive interview findings did identify a potential issue with the wording of the SNAP benefit question that could contribute to missed recipients. Specifically, the question asks if “(you/anyone) gets Food Stamps or a Food Stamp benefit card”. One respondent pointed out that he didn’t ‘get’ the card in the previous year, he received it prior to that but was still receiving benefits on it during 2010.

The Urban Institute data analysis found that the pattern for TANF is just the opposite of that found with SNAP and SSI —the caseload aggregate is 57% of the target value, but the dollar total is only 43% of the total. This suggests that those who with larger TANF benefits may actually be less likely to report any receipt. Or it might be the case that respondents who report receipt of TANF simply systematically underreport amounts received during the calendar year.

With these findings in mind, recruiting efforts again targeted individuals who just started receiving SNAP or TANF benefits during the previous calendar. As with Round One, all versions of the interview instrument excluded the Family Income Screener question currently in the ASEC Interview ( a modification supported by the Data Analysis results that found between 12% and 20% of SNAP and TANF recipients would be screened out if the Family Income Screening criteria were applied to ACS data.) The Round Two wording for the SNAP question replaced “get” with “use” as one approach for addressing missed recipients. However, in the absence of the opportunity to evaluate the TANF question in Round One, no specific wording changes were made for Round Two. Probing to identify knowledge of TANF, language and labels for TANF and any prior receipt of TANF served as the main evaluative information about those income questions.

REPORTING OF ASSET INCOME

Round One cognitive interviews revealed some issues with the identification of withdrawals and required distributions from retirement accounts, based on the wording used in that round. Thus Round Two made additional changes that more specifically cue on both withdrawals and distributions when asking if these are a source of income. The Data Analysis Report supported the need for these updates, identifying that the CPS misses over 90 percent of retirement account withdrawals.

The report also demonstrated that the CPS misses about a quarter of SOI pension recipients and pension income. It misses nearly 40 percent of aggregate interest income. With comparable shares of filers with interest income, the underreporting of CPS interest amounts is driven by underreporting of dollars and not by underreporting of receipt. The CPS is also missing about 75 percent of dividend income.

As a result of these findings, the Round Two interviews further distinguished between retirement assets and those outside of retirements, but for both, the interviewer started by asking for the specific types of assets or accounts the person might have and then probed about interest or dividends for the individual accounts or assets identified. This approach is based on the hypothesis that some of the under-reporting may reflect the need for additional cueing to help respondents think through all such assets and accounts in order to include them in their report of interest earned or dividends received.

In addition, for respondents who refused or said ‘don’t know’ to the probe on interest or dividend amounts, interviewers asked for the total account balance. In reaction to the observed sensitivity of collecting total account values with some respondents (discussed further in the results section), the account value question was prefaced with an explanation that having the value amount would allow the Census Bureau to estimate the amount of interest earned, or dividends received. If respondents could/would not answer the total value question, the interviewer followed –up with the bracketed amount questions to collect income or dividend amounts. This combination of changes - cueing for specific accounts, and then adding two different methods for capturing interest earned or dividends received – targeted reduction of under-reporting as well as decreased item nonresponse.

DISABILITY

In addition to the more specific cuing for types of disability income noted above, Round Two included one more additional change. In Round One, respondents noted confusion in regards to the implied reference period in the first (screener) disability income source question. Specifically, the question asked “(Do you/Does anyone in the household) have a health problem or disability which prevents (you/them) from working or limits the kind or amount of work (you/they) can do?” The question wording implies ‘as of right now’ which for the ASEC interview is in March following the appropriate calendar year reference period. This could screen out people who did have a disability and were receiving disability income in the prior year, but are not at the time of the interview resulting in underreporting. Thus, Round Two changed the screener question to refer to the prior calendar year.

PROPERTY

In Round One, one respondent noted that he did have a rental property, but chose not to report this potential source of income because it did not yield a profit in the prior calendar year. In fact, the Data Analysis report identified that the CPS appears to miss a significant proportion of rental unit losers and the associated income losses. Thus, Round two included specific language to include property as a source of income even in the absence of a profit.

Attachments D1 – D3 show the three different interview instruments used in Round Two. (In this round, the cognitive interview probes were embedded within each interview instrument.)

Findings

Observations Regarding the Structural Changes

Both rounds of cognitive testing included the following three structural changes:

Collecting all sources of income first, before asking for amounts for any income source.
Varying the order of presentation of income types, based on known respondent characteristics, with three general orders used in each round.
Following-up “Don’t Know” or “refused” responses with a series of questions asking bracketed dollar amounts.

As noted above the purpose of each of these structural changes was to make use of automation to minimize under-reporting and missing data. In round two, one additional structural change was made to increase the memory cues for respondents as another method for minimizing under-reporting, as well as to better facilitate the separation of source and amounts collection. Specifically, in the Asset sections (interest and dividends) as well as within other income sources that cover multiple sub-sources (e.g., disability, pensions, survivor benefits) we modified the instrument to collect responses at the sub-source level.

Source first approach

In the first round, the source-first approach resulted in some difficulties for respondents at the point when requesting the amount received for a given income source. However, the difficulty resulted from issues with using a paper rather than an automated instrument. Interviewers did not always record the exact sub-type of income within the source level questions. Thus, when the interviewer asked the corresponding amount question, he or she could not repeat back as part of the question the necessary cue (sub-source name). With automation, CAPI can store the sub-source name and use it as a fill in the corresponding amount question that comes later in the interview. Despite this paper-based limitation, there were some respondents who noted a benefit to the source first approach, saying things such as: “it’s better like you did it – to me –because then that way you didn’t really put nobody on guard…. You could get it out of a person better than asking flat out ‘how much did you get…?”. There were also respondents who stated a preference for the interleafed approach, though in part this preference reflected the interviewer difficulty with maintaining the link between the positive response at the source question and the appropriate (sub) source in the corresponding amount question.

In the second round, interviewers did not probe specifically for respondents’ feedback on the source first approach. Rather interviewers attempted to determine if any income sources were missed, and observed any confusion between the source questions and the request for the associated amount of income from that source. Round two interviews did not result in any of the same confusion or difficulties observed in round one with maintaining the link between the two pieces of information. In fact, in the second round there was some suggestion that the separation of source and amounts could have an additional advantage. In two different interviews, the respondents were aware of and could report the different types of assets they owned. They reported their assets without hesitation. However, when it came to asking for the earned income from those assets, neither respondent knew the amounts, and repeatedly “excused” their lack of knowledge saying that the accountant knew the amounts. Both respondents seemed to get agitated or slightly defensive. However, because this occurred after collecting all sources of income, the structure diminished the possible negative effect on reporting of subsequent incomes types.

We also observed one other potential advantage of separating source and amount questions. The separation allows for a ‘second-pass’ at thinking through the income source. An interviewer encountered a couple of instances in which in the amount section, respondents noted two or more of an income type (e.g., two 401K accounts) that hadn’t been discussed initially in the source section. It seemed the second pass through the income type when requesting amounts could provide an opportunity for more specific recall, perhaps encouraging better reporting. That being said, these interviews did include additional probing between the source and amounts section which could also influence respondent recall, and the analysis cannot separate out that affect. However, there was no direct indication that the source first approach resulted in any negative response characteristics. More specifically, we did not observe any completely missed sources of income that seemed linked to the source first approach.

Tailoring the order of income source presentation

In both Round One and Round Two, interviewers used one of three orders for collecting the sources of income: a low-income order, a senior (62+) order, and the default order. The order presented depended on known characteristics of the respondents. In the cognitive interview situation, the respondent characteristics used for determining the appropriate order came from the screening interview completed during recruitment. In the actual ASEC, automation will allow the use of more detailed information collected in prior waves, or even from earlier in the interview (e.g., roster, work history) to determine the most appropriate order.

The goal for using a tailored order is to reduce the effects of respondent fatigue on reporting, by increasing the likelihood that the most relevant source of income are discussed and identified earlier in the interview. The target outcome is a reduction in missing data, and under-reporting. However, without the benefit of a ‘truth’ source, the cognitive interview environment limits our ability to assess the effect on data quality. As with the source-first assessment, our analysis focused on identifying any missed sources of income, particularly those that came later in the order of presentation. Round one did not reveal any such missed sources. However, in Round two we did observe three cases of missed income (identified by the detailed probing at the end of the interview) from the “other income” source questions. However, in all three cases it seemed the respondent did not report the income for reasons other than the order of presentation of income types. For example, one female respondent did not report two $10 gift cards she received for taking her infant to two well-baby doctor visits. She indicated that she didn’t think of these small gift cards only available to her for this very specific reason, as relevant to this data collection. The other instances also reflected the respondent’s perception that the money did not match the scope of this data collection. One woman neglected to report income received for a few months of the previous year through a special grant that covered the dietary needs of her disabled son. Another respondent’s partner received a one-time payment of $40 for driving a relative some place. These errors reflect small dollar amounts, for very specific instances, and do not likely contribute to a possible explanation of underreporting in the ASEC. Nor do these missed reports seem attributable to the order of presentation; rather this seems to reflect the respondents’ perceived relevance to the survey objectives.

Follow-up Bracket Items for “Don’t Know” Responses

In both Round One and Round Two, interviewers used follow-up brackets for “don’t know” responses or refusals. There was one difference between rounds in the timing of the bracketed response questions, but only for the Asset reporting. In Round Two (described in more detail below), if respondents initially refused or indicated they didn’t know the amount of interest earned or dividends received, the interviewer first attempted to get the total account value. In requesting the account value, the interviewer explained that the Census Bureau can use the account value to estimate interest/dividend amounts. Only if respondents were not able or unwilling to report the account value did interviewers ask the bracketed amounts questions for Asset reporting in Round Two.

In Round One, interviewers used brackets at least once in 13 of the 27 interviews. Interviewers used brackets in nine instances for estimating interest amounts, both for accounts owned by the respondent and those owned by other household members. Brackets were used to estimate earnings for other household members in four interviews. In Round two, brackets were used in only 6 interviews, though this seemed reflective of Round Two respondents’ greater willingness to provide an estimated dollar amount. In Round Two, interviewers used brackets for estimating interest in checking and savings accounts, and with dividends, and in one instance in estimating another householders earnings. Across these six cases, only one respondent went beyond the second level of the bracket, and one respondent indicated that she could not go further than the initial question.

One of the concerns with the use of the bracketed follow-up questions is that they make it easy for an uninformed respondent to guess. However, the self-imposed limitation we observed in Round Two suggests that respondents don’t necessarily ‘acquiesce’ and give responses beyond what they think they can reasonably know. In fact, one respondent in Round Two after going to the third level of brackets for his own interest income, after the initial question the interest on a savings account for someone else in the household said “Oh, you’re just going to keep on, aren’t you. I don’t know.” He then told the interviewer to make the answer ‘don’t know.’ Similarly, in Round One, three of the 13 cases who initially started using brackets ultimately decided to respond with “don’t know” rather than select from the brackets.

We observed one other potential benefit of using follow-up brackets in round one. In three cases, respondents used the brackets to help develop a point estimate for their own interest earned. It seemed the brackets provided an additional opportunity to think about their answer, leading to a point estimate for a response that originally started out as a ‘don’t know’. We did not see this same behavior in Round Two.

As with the evaluation of the other structural modifications, the absence of a “truth” source of the respondents’ income means we limit our data quality assessment to what we observed in the interviews when respondents used the brackets. Our observations suggest that respondents are willing to use the bracketed follow-up questions, but limit the level of precision in their response rather than guessing in subsequent, lower-level brackets. We also don’t have appropriate data to speak to whether respondents might view the brackets as overly burdensome. Thus, we suggest the Census Bureau include the use of brackets into subsequent testing of an automated instrument, but also include measures of perceived burden, as well as a method to assess the quality of the bracketed response.

Observations on the Income Sources of Key Interest for this Project

TANF

Since receipt of TANF is known to be chronically under-reported in ASEC, we put a priority on recruiting low-income persons for this project likely to have received these benefits in 2010, so as to explore possible reasons for erroneous reporting. The strategy for determining whether the household had members who received TANF was essentially the same across the two rounds, though there were some notable differences. In Round 1, we used the current ASEC question wording for determining TANF receipt. In Round 2, we converted a respondent instruction to include food stamps, WIC and other benefits as TANF to an interviewer instruction, and more importantly we attempted to gain the most specific type of assistance possible so as to be able to ask more targeted questions when asking for amounts (consistent with the approach taken in Round 2 for other income sources, as discussed in the Methods section). We also provided flexibility for respondents to report TANF amounts separately for each child – it is possible that children of the same mother within a household receive differing TANF amounts because one, for example, receives child support or disability benefits that another child does not receive. In such cases, it may be easier for respondents to report two separate amounts, rather than a single total amount. In both rounds, and in all locations, interviewers referred to the specific state program name for TANF. Below we show how the questions varied between Round and Round 2.

Current ASEC and Round 1

At any time during 2010, even for one month, did (you/ anyone in this household) receive any CASH assistance from a state or county welfare program such as (STATE PROGRAM NAME)?

Include cash payments from:

Welfare or welfare-to-work programs,

(State Program Name and/or acronyms),

Temporary Assistance for Needy Families program (TANF),

Aid to Families with Dependent Children (AFDC),

General Assistance/Emergency Assistance program,

Diversion Payments,

Refugee Cash and Medical Assistance program,

General Assistance from Bureau of Indian Affairs, or Tribal Administered General Assistance.

Do not include food stamps/Supplemental Nutrition Assistance Program (SNAP) benefits, SSI, energy assistance, WIC, School meals, or transportation, childcare, rental, or education assistance.

-------------------------------

Round 2

At any time during 2010, even for one month, did (you/ anyone in this household) receive any CASH assistance from a state or county welfare program such as (STATE PROGRAM NAME)?

Yes

No (SKIP)

From what type of program did (name/you) receive the CASH assistance? Was it a welfare or welfare-to-work program such as (STATE PROGRAM NAME), General Assistance, Emergency Assistance, or some other program?

(State Program Name)/Temporary Assistance to Needy Families (TANF)/welfare/AFDC
General Assistance
Emergency Assistance/short-term cash assistance
Diversion Payments
Refugee Cash and Medical Assistance program
General Assistance from Bureau of Indian Affairs, or Tribal Administered General Assistance

If respondent mentions any of the following categories 7 through 12, note this, but probe: “Right now we are interested in cash assistance” and seek answer to 59C8-R using categories above.

Food stamps/Supplemental Nutrition Assistance Program (SNAP) benefits/
SSI
Energy assistance
WIC
School meals
Transportation, childcare, rental or education assistance
Some other program What type of program? ______________

Earlier, you reported that [NAME(S)] received [CASH ASSISTANCE PROGRAM NAME (Q59C8)] (on behalf of children) in 2010. What would be the easiest way for you to report the amount received - for everyone in the household combined, or for each person separately?

Combined

Separately

In Round 1, no one among the 8 low income persons we recruited reported having received TANF during the reference year (2010). From what interviewers could gather through probing, none of these respondents seemed to have failed to report TANF receipt in response to the ASEC questions. No respondents seemed to have had difficulty understanding the question. Four of 5 persons probed on the state program name were aware of it. A couple of respondents had received TANF in prior years, but it appeared that most of our low income respondents were in fact not eligible for it (based on income, no children, etc.).

As noted in the Methods section, for Round 2 we made a more concerted effort to recruit persons potentially eligible for TANF. We contacted local TANF offices in each location for assistance, though offices in 3 locations refused to provide it. We also required low income recruits to have dependent children in the household (a strict requirement for receipt of TANF). Out of 15 low-income persons recruited for the second round, 7 reported having received TANF during the previous year. As in Round 1, we observed nothing to suggest that someone may not have received TANF yet did not report it in response to the ASEC questions. However, we believe that 3 of the 7 persons reporting TANF receipt did so incorrectly:

One reported TANF receipt on the basis of having received food stamps. When asked the new question we had devised for this round regarding the type of program providing this assistance, he simply answered “welfare.” He later noted that he has always thought of food stamps as being welfare, recalling as a child being taken to the “welfare office” by his mother in order to get their food stamps;
The other two respondents reported TANF receipt on the basis of assistance that went directly to their landlords to help pay their rent – in other words, they did not receive money to help with their general expenses. One asked the follow-up item regarding the program type, one described it as “Welfare-to-work, the DWP” (i.e., Diversionary Work Program), while the other said it was “Emergency Assistance.”

Although the new question we inserted asking for the type of program failed to catch these incorrect reports of TANF receipt, the question did appear to be effective overall, with respondents offering a variety of programs, including “TANF,” “Welfare,” “General Assistance,” “Emergency Assistance,” and “DWP” or “Diversionary Work Program.” As intended, we referred to these program types later when asking for the amounts. So we think this follow-up question is worth additional consideration and testing for ASEC. Note that to the extent respondents mention assistance programs that should not be reported as TANF (e.g., food stamps, WIC, rental assistance, etc.), this information could be captured and allow for skipping later items on these programs. So the burden associated with this additional question should be very minimal.

We were somewhat surprised by the “false positives” we observed with respect to TANF receipt, given the known problem of serious under-reporting of this income source. But to minimize such errors, the Census Bureau might want to consider referring to “general CASH assistance” in the question (rather than just “CASH assistance.”) Also consider following up “yes” responses with a question such as: “Did this general cash assistance have to be used for a specific purpose, such as _____ ?” It would be useful if the examples referred to in such a question are state-specific.

Food Stamps

Like TANF, Food Stamps are an income source known to be chronically under-reported in ASEC. Thus, we hoped to explore reasons for this among the low-income persons recruited for the cognitive interviews. For the first round, we tested the current ASEC questions on food stamp receipt, which appear as follows:

Did (you/anyone in this household) get food stamps or a food stamp benefit card at any time during 2010?

YES (SKIP)

At any time during 2010, even for one month, did (you/anyone in this household) receive any food assistance from (STATE PROGRAM NAME)?

In all locations, we referred to the unique food stamp program name for that state.

In Round 1, 6 of 8 recruited low-income participants reported receiving food stamps. We saw no reason to believe that anyone had received food stamps yet was not reporting it in response to the ASEC questions. One person did note that asking “Did you get…a food stamp benefit card..” was a bit confusing, since one can receive food stamps benefits on the same card over a considerable period of time. This person did answer the question correctly though, and all 8 low-income persons seemed familiar with the notion of food stamps and a food stamp benefit card. For example, one of the persons not reporting food stamp receipt discussed how he had applied for food stamps yet was told he was ineligible. In addition, almost all seemed to correctly (and quite easily) report the amount of the food stamp benefit they had received. One individual, in a very large and complex household, did likely misreport the amount – it seemed he failed to multiply correctly when generating the total benefit amount per month across all household members covered by this assistance. Furthermore, when making this calculation he used his currently monthly benefit amount, neglecting to consider two occasions in 2010 when the benefit level had been adjusted.

In Round 2, we made a minor adjustment to the initial food stamp item in an effort to avoid confusion regarding when some may have gotten their benefit card:

Did (you/anyone in this household) get food stamps or use a food stamp benefit card at any time during 2010?

As in Round 1, the food stamps items appeared to be very straightforward and work as intended. Sixteen persons in this round reported having received food stamps in 2010, but the questions were always easily answered, regardless of the response. As far as we were able to determine, no one had received food stamps while failing to report it in response to the ASEC questions. As with TANF, we observed nothing to suggest that under-reporting in ASEC is due to problems with comprehension of the questions. Reporting of the food stamp benefit amount was generally easy and straightforward as well. One person could not recall with certainty whether a drastic reduction to her monthly benefit had occurred in late 2009 or early 2010 – she ultimately may have under-reported on the amount since she decided the reduction had occurred in 2009. In addition, after hearing the ASEC item: “What is the (monthly) value of the food assistance received in 2010?” one participant replied with “Meaning how much did she get?” Perhaps the wording of this item is unnecessarily formal in tone, and could be reworded in a more conversational manner, such as: “How much did (you/name) receive in (monthly) food stamp benefits in 2010?”

Disability Income

In this section we describe the respondent difficulties and other issues we observed with respect to reporting income received for reasons of disability, including short-term disability and longer term disability payments from Social Security Disability Income (SSDI) and Supplemental Security Income (SSI). In Round 1 we tested the current ASEC approach for collecting disability income, detecting several problems. In Round 2 we tested major potential revisions to ASEC’s approach to capturing this type of income.

In the first round of interviews, 13 respondents were screened as having disability income. One significant observation from the first round was that three partial-year recipients of SSDI did not report large “back-payments” that covered a lag in time between the determination of their eligibility and the receipt of their first disability payment. Two respondents did not think any ASEC question asked for the back-payment amount since it was something quite different from their regular monthly payment amounts. Neither one even thought of it when responding to the ASEC items collecting the amount of their disability income. Another respondent chose not report the back-payment due to the sensitivity of having received a large amount – she discussed how revealing the amount to her family and friends had been a source of tension among them.

Two respondents did not report receiving short-term disability in the income sources section, as they had trouble determining the appropriate questions at which to report this income. In both cases, the key ASEC items designed to identify persons who might have received such disability income failed to do so. These questions are as follows:

(Do you/Does anyone in the household) have a health problem or disability which prevents (you/them) from working or which limits the kind or amount of work (you/they) can do?
(Did you/Is there anyone in this household who) ever (retire or leave/ retired or left) a job for health reasons?

Note that the first question (ASEC Q59A) is phrased in the present tense. Both respondents answered “no” because they no longer have the health issue that prevented them from working for part of 2010. They both also answered the second item with “no” since they had merely stopped working for a short time, rather than retiring or leaving their job. As one put it: “I didn’t leave the job – I was out for a while, but I was still employed by the same company, so I just went back to work for them, after my recovery.”

Interestingly, both of these respondents wanted to report this short-term disability income – they were just unsure of where in the ASEC interview to report it. One respondent asked if she should report her disability income in response to the ASEC item on Supplemental Unemployment Benefits. The other, upon hearing ASEC’s definition of SSI (“assistance payments to low-income aged, blind and disabled persons, and come from state or local welfare offices, the Federal government, or both”) wondered if his short-term disability might fit here. However, he ultimately reported the amount of his short-term disability when asked about the financial assistance he had received. Here he neglected to report actual financial assistance he should have reported, replacing it with his short-term disability payments. The use of a more effective cue when asking for the amount of financial assistance would have avoided this confusion.

At least two respondents had difficulty distinguishing SSDI from SSI benefits. One respondent reported getting both forms of disability pay at different times in 2010. When he was asked ASEC Q57a (“During 2010 did [you/ anyone in this household] receive: any SSI payments, that is, Supplemental Security Income?”), he asked “That’s the same thing as Social Security, right?” When he was asked “As you understand it, which do you get: regular Social Security or SSI?” the respondent said he received both sources, although it was clear he was unfamiliar with the SSI program name: “I got Social Security part of the year then it switched over to Social Security Disability.” Another respondent who received disability income, likely through Social Security, also had trouble distinguishing the two programs. This respondent was asked ASEC Q57a, as well as the interviewer note: “SSI are assistance payments to low-income aged, blind and disabled persons, and come from state or local welfare offices, the Federal government, or both.” In response, he said: “Well they say I didn't qualify for welfare, but disability and social security are the same thing. It would fall under social security, yes” and he went on to report receiving SSI. However, later, when he was at the disability source section, he reported “From what I understood, I was supposed to qualify for both ... but they only qualify me for one ... that would be under SSD.” He suggested what he received should most accurately be reported as ‘disability income’ rather than Social Security payments or SSI.

For the Round 2 interviews, a number of changes were made in the instrument after first round to address the difficulties we observed.

To serve as a cue for respondents to recall back-payments received, we added a question to the disability amounts section for those who reported receiving Social Security for less than a full year:

During 2010, did (name/you) receive an initial disability payment that was larger than the usual payment? (This is sometimes done to make up for a delay in the start of payments)?

To minimize confusion about whether to report disability income in response to early or later questions, we changed the ordering of income sources to have the disability section immediately precede the sections asking about SSI and SSDI payments, and furthermore added follow-up items into the disability income section designed to determine the specific source(s) of disability pay. This includes items on SSI and SSDI, since we expected many receiving these benefits would now report them in the disability section.

In addition to moving the disability source section before sections on SSI and Social Security, we added descriptions of the programs into the sub-items that referred to those programs (at Q61B_R, item a and Q61B_R, item b), to make it easier for respondents distinguish them:

a. Social Security Disability Insurance, for people who are eligible based on years of work;

b. Supplemental Security Income, which provides payments to low-income aged, blind and disabled persons;

To help prevent underreporting among those who experienced only short-term interruptions in their income due to a disability, we revised ASEC Q59A (one of the ASEC disability screener items) from its current present tense form to the following:

At any time in 2010 (did you/did anyone in the household) have a disability or health problem which prevented (you/them) from working, even for a short time, or which limited the work (you/they) could do?

In Round 2, we observed less evidence that respondents were misreporting receipt of disability income. One respondent did under-report the amount of disability benefits he received due to an unreported back-payment. A total of 21 of the 29 Round 2 respondents answered yes to one or both of the ASEC screener questions in the disability section. Of those 21 respondents, 14 reported that someone in their household received disability income in 2010. For one of the 14, it was undetermined if this (Social Security) income was actually due to disability or was traditional retirement, though the amount was reported without difficulty.

Three respondents had some difficulty responding to the follow-up items we added to the disability section that were designed to determine the specific program/source of disability pay:

One respondent had difficulty recalling the source of his disability income. He qualified his ‘Yes’ answer to an item asking if it was ‘Social Security Disability Insurance, for people who are eligible based on years of work,” by indicating he received income from a state temporary disability program, though he thought the source was Social Security. It seemed likely that he was reporting his disability source prematurely, as a moment later he indicated that it came from “Federal, State or Local government employee disability” and he knew that he received disability income from only one source;
One respondent had difficulty understanding the intent of the follow-up items we had added to determine the source of disability pay. She was administered the question stem (‘Was this disability income from…’) but as the sub-items were read, she appeared to think the items were asking whether she had access to these sources, and not whether she actually received income from each of them in 2010. At the item on ‘Company or union disability payments’ she seemed unsure if her employer’s long-term disability benefit plan should be reported. She demonstrated the same problem at an item meant to determine if the disability pay was ‘Accident or disability insurance’ - she paraphrased the question incorrectly as ‘Do I have accident or disability insurance…?’ As a separate issue, she was uncertain if she should include Social Security-Disability that she had already reported in this series again at an item determining whether she had gotten ‘Federal, State or Local government employee disability.’ In our view, this respondent’s difficulties could best be handled by simple clarifications provided by an interviewer;
Another respondent appeared unsure of who the follow-up items were pertaining to. She indicated confusion at the item ‘Supplemental Security Income, which provides payments to low-income aged, blind and disabled persons;’], saying “I don’t understand that question. My kids get Social Security…uh, SSI…Social Security income, because I’m disabled and they’re in school.” She did not seem to realize the questions were asking about her own disability income. This is another instance that could easily be handled by an experienced interviewer.

Some confusion might be avoided by revising the order of follow-up items (currently ordered from programs with the highest number of recipients to the fewest), rephrasing the stem wording to refer explicitly refer to the respondent or household member who received the disability pay.

Three respondents had difficulty distinguishing SSDI and SSI. Two of these reported (correctly, we believe) household members receiving income from both programs. One respondent reported both SSI and SSDI as sources in the disability section but when asked at the next section if she received Social Security aside from what was already counted, she asked if SSI should be reported. She referred to it as ‘Supplemental Social Security Income,’ suggesting a reason for her confusion between programs is her limited knowledge of the acronym, and not knowing the program name that it represents. The other respondent demonstrated a similar confusion, as she referred to SSI as “Social Security…uh, SSI…Social Security income” when reporting about her children’s benefits. Such confusion is likely to persist due to the similarities in program names, though it may be minimized by referring to programs consistently using the same wording. In addition, respondents indicated that SSI payments are electronically deposited on the 1^st of each month and SSDI payments are deposited on the 3^rd of the month – if this can be established as true for virtually all persons receiving such disability payments, this information should be included on CATI/CAPI screens as interviewer help text.

Collecting income amounts from our interview participants with disability income was generally very straightforward and almost no problems or issues were observed. Only one of the 5 respondents who were asked the new question regarding back-payments had difficulty reporting them correctly. This respondent initially referenced receiving a back-payment when he was deciding whether to report it in the Social Security section despite already having reported receiving Social Security benefits in the disability section. He omitted a sizeable amount of disability income ($9,000-$10,000) received from back-payments. The respondent seemed to interpret the back-payment question as referring only to the payments he received on a monthly basis, noting he received portions of his back-payment at various times during the year. Our question, as worded, did assume that recipients receive one-time back-payments. So we would now recommend a wording that does not make this assumption. Furthermore, the term “back-payment” is one we heard respondents repeatedly use in these interviews, so we suggest this term be used in the question. Consider the following wording, for example:

During 2010, did (name/you) receive any back-payments to make up for a delay in the start of (your/his/her) disability payments?

Asset Income

Based on the work and recommendations of the Urban Institute, we explored the feasibility of two major refinements to the ASEC data collection of asset income. First, as a supplement to the current questions on dividend and interest income, we developed questions asking for the values of the assets generating this income. People may be more knowledgeable about the balances of accounts holding stocks and mutual funds, certificates of deposits, and other forms of savings, relative to income received within these accounts. Second, we explored ways of collecting asset income (and values) separately for tax-advantaged retirement accounts, apart from other forms of investments. The Urban Institute argued that collecting information on retirement accounts would yield an improved picture of the wealth of U.S. households. With the asset income questions in the current ASEC, it is unclear to what extent the questions are intended to collect income from retirement accounts.

Table 5 illustrates the approach we tested in Round 1 for determining whether household members have retirement accounts, and whether payments or withdrawals were received from these accounts. Note that the current ASEC relies upon the respondent to recall and report income from retirement accounts in response to rather broad questions that do not specifically cue retirement accounts.

Table 5. Determining Sources of Retirement income: ASEC versus Round 1 Questionnaire

Current ASEC

Round 1

During 2010 did (you/anyone in HH) receive any pension or retirement income from a previous employer or union (other than Social Security/VA benefits)?
What was the source of (name’s) income?

At any time during 2010 did (you/anyone in HH) have any retirement accounts such as a 401(k), 403(b), KEOGH, or IRA?
During 2010 did (you/anyone in HH) receive any pension or retirement income from a previous employer or union (other than VA benefits)?
During 2010 did (you/anyone in HH) receive any other retirement income (other than Social Security/VA benefits) (including payments or lump sum withdrawals from a retirement account)?
What was the source of (name) income?

After asking the standard ASEC questions on whether household members had interest-earning assets (e.g., money-market funds, savings accounts, savings bonds, certificates of deposits) during the reference year, we added questions to determine if these assets were held in a retirement account (if applicable), outside the retirement account, or in both:

[IF PERSON HOLDS A RETIREMENT ACCOUNT] Did (you/NAME) have (this asset / any these assets) within a retirement account?

Yes

No (SKIP)

Did (you/NAME) (also) have (this asset / any these assets) outside of a retirement account?

Yes

Similarly, if someone in the household was reported to have dividend-earning assets (i.e., shares of stock in corporations or mutual funds), we sought to determine if they were held in a retirement account (if applicable), outside the retirement account, or in both:

[IF PERSONS HOLDS A RETIREMENT ACCOUNT] Did (you/NAME) own any of these shares within a retirement account?

Yes

No (SKIP)

Did (you/NAME) (also) own shares of stock or mutual funds outside of a retirement account?

Yes

When collecting amounts of income from interest and dividend-earning assets, we sought to obtain the income separately for retirement and non-retirement accounts (as applicable), and to obtain asset values in addition to the income generated. Table 6 illustrates this approach. It should be noted that we varied the order of collecting asset income and values across interviews.

Table 6. Collecting Asset Income: Current ASEC versus Round 1 Questionnaire

Current ASEC

Round 1

How much did (name/you) receive in interest from these sources during 2010, including even small amounts reinvested or credited to accounts?

Within (your/NAME’s) retirement account(s) what was the value of the (interest earning accounts or money market funds/savings bonds/treasury notes, CDs or other investments which pay interest) at the end of 2010?

Within (your/NAME’s) retirement account(s) how much did (name/you) receive in interest from these sources during 2010, including even small amounts reinvested or credited to accounts?

(Outside of (your/NAME’s) retirement account(s)), what was the value…

(Outside of (your/NAME’s) retirement account(s)), how much did you receive…

We observed a myriad of respondent difficulties and problems in the first round of cognitive interviews. Some of the observations stemmed from issues with the current ASEC items, while others were due to issues with the additional measures we had developed for this section. The most notable problem was that respondents, as well as interviewers, often struggled to determine precisely what asset was being targeted by a question asking whether the asset was held within a retirement account, for the amount of interest income earned, or the value of the asset. Many of our respondents had assets in both retirement and non-retirement accounts, at least two persons had more than one retirement account (e.g., both an IRA and a 401(k), or more than one IRA). Some also reported having retirement pensions from previous employers. While the new questions we developed attempted to address assets held in retirement accounts separately from those in non-retirement assets, they did not distinguish among different types of retirement accounts, or adequately distinguish between retirement savings accounts and pensions. One respondent, when asked if any of “these assets” (i.e., interest-earning assets she had just reported having) summed up the problem in this way:

“I don’t know how to classify different things. I think when you [ask a question] and then you don’t really list the thing that I’m looking for, it’s like ‘what category did I put it under?”

Furthermore, the current ASEC questions on interest-earning assets ask about multiple asset types in a single question. For example, one item asks if anyone in the household had a money-market fund interest-earning checking account, or a savings account during the reference year. Another asks if anyone had “any treasury notes, certificates of deposit, or any other investments which pay interest.” One respondent neglected to report having a CD worth $165,000 in response to this latter item – he seemed distracted by the reference to “treasury notes” in the item. Otherwise, since these questions do not determine precisely what kind of account is being reported, the latter questions asking for the amount of interest earned were frequently awkward and confusing, since much of the wording of these questions referred to accounts the respondent does not have. On the other hand, In at least two cases, the respondent had more than one of the account types mentioned in the question, and in such cases it can be unclear whether the respondent should report for the multiple accounts combined, or one by one. Thus, considerable unscripted probing and clarifications between the respondent and cognitive interviewers were necessary when collecting the amount of interest income.

Several other observations from the first round of interviews are worth mentioning:

We learned that the notion of a “retirement account” needs greater specificity. For one respondent, the retirement account was simply an account in which his Social Security check gets deposited, rather than a tax-advantaged account designed for retirement savings. This caused confusion at the new question we had created to determine If certain interest-earning assets were within his retirement account;
Referring to a “lump sum withdrawal” can be confusing. When asked “During 2010, did anyone…receive any other type of retirement income, including payments or lump sum withdrawals from a retirement account?” one person said “They weren’t lump sums, I took the minimum amount.” In retrospect, we should have not used this wording to refer to a retirement account distribution, as it could be taken to mean withdrawing the entire amount of the account, as noted by this respondent;
Not everyone with stock and mutual fund investments understands the difference between dividends and interest. For example, when asked how much dividend income he had received, one person said the question was identical to one we had previously asked about interest income. In fact, he had reported dividend income in response to the item seeking the amount of interest income. As another example, a respondent who owned stock/mutual fund shares, when asking how much interest she had earned from an interest-earning asset, reminded the interviewer that “some stocks pay you interest too.” She apparently believed she should be reporting dividends from the stock/mutual fund shares in combination with interest from her money-market fund and savings account;
At least two respondents referred to having received a “required minimum distribution” from their retirement account. None of the questions we tested included this wording, but since this seems to be standard, commonly understood language, we decided it was worth using in revised questions tested in the second round of cognitive interviews;
Asking for the “value” of an asset (such as a retirement account) was not immediately clear to at least two respondents. For example, one person thought about it briefly and sought clarification by asking: “You mean ‘what is the balance?’”
Asking for the value of an asset can be viewed as intrusive. In a few instances, respondents refused to divulge asset values, such as the amount in a retirement account. With the exception of one person who did not want to reveal the amount of a back-payment for disability benefits, we encountered no refusals to divulge income information requested by ASEC in these interviews.
Respondents did seem to possess a greater level of knowledge of the values of their assets as compared to the amount of interest or dividend income these assets generated. In the vast majority of cases, respondents were able to report an approximate value of their assets (e.g., amount held in stocks and mutual funds) at the end of the previous year, but we observed frequent difficulties with generating the income earned from these assets. This pattern was particularly true for retirement accounts – a couple of persons did not even know whether their retirement account assets were invested dividend-earning assets (i.e., stock and mutual fund shares) or interest-earning assets (e.g., money-market funds). As one person explained: “I have investments in Vanguard, I don’t know what they are.”

For Round 2, the sections of the ASEC instrument determining sources of interest and dividend income, including that within retirement accounts, were substantially revised. The revisions were driven directly by the notable problems we observed in Round 1.

First, we developed a short, straightforward section for retirement accounts. This section determined who in the household held retirement accounts and what types of accounts (e.g., 401(k), Roth IRA, SEP, etc.). This was done so as to allow for greater specificity (thus making questions more clear) and more effectively cue respondent recall when interviewers are asking the amount of interest or dividends earned from a retirement asset – for example, by referring to “your 401(k) account” rather than “your retirement account.” In addition, we added a question that specifically addresses whether the individual withdrew any money (received a distribution) from these retirement accounts. It became clear to us after Round 1 that ASEC is intended to capture this as income, yet the current instrument lacks a question directly asking about retirement account withdrawals.

Second, we revised the current ASEC items determining interest and dividend-earning assets to a somewhat longer, but simpler series of items. For example, rather than asking if anyone in the household had money in “any kind of money market fund, interest-earning checking account, or savings account“ (Item Q63A1 in the ASEC items booklet), we sought to determine whether anyone had money in each of these accounts, by obtaining a “yes” or “no” response for each. While this may initially appear to increase respondent burden by increasing the number of questions, asking short, easier-to-understand items will serve to decrease respondent burden. Also note that by obtaining the specific types of interest-earning assets held, the latter items collecting the amounts of interest earned can be asked with greater specificity (by referring to a single account type, rather than multiple account types, some of which do not apply), which should further aid in increasing respondent comprehension and cueing recall.

Finally, rather than ask all persons with these assets for both the value of the assets (which Round 1revealed to be perceived as quite intrusive) and the interest or dividend income generated by these assets, we instead relied upon first asking for the interest or dividend income, and only if the respondent indicating not knowing the amount (or refused), we followed up by asking for value. We attempted to “soften” the request for the asset value by offering a brief justification for the question, noting that the Census Bureau can estimate the amount of income generated if the asset value is known.

The questions on retirement accounts and interest/dividend-earning assets that we tested in Round 2 are shown below.

At any time during 2010 did (you/ anyone in this household) have any retirement accounts such as a 401(k), 403(b), IRA, or other account designed specifically for retirement savings?

YES

NO (SKIP)

Which members of this household ages 15 and over had such a retirement account?

What type of retirement account (did you/ NAME) have? Did (you/he/she) have…

(MARK ALL THAT APPLY)

A 401(k)?

A 403(b)?

A Roth IRA?

A Regular IRA?

A KEOGH plan?

A SEP plan?

Another type of retirement account? (SPECIFY): ______________________

Did (you/NAME) withdraw any money or receive a distribution from (your/his/her) [ACCOUNT TYPE] account in 2010? IF AGE 70+ ADD: including distributions you may have been required to take?

YES

IF ANY RETIREMENT ACCOUNTS IN HH, READ TRANSITION:

(Now I will ask about assets that may have paid interest or dividends in 2010 outside of the [ACCOUNT TYPES).

At anytime during 2010, did (you/anyone in this household) have money in:

An interest-earning checking account?