Attachment J Report Cognitive Testing Results

Attachment J
Census Report “Cognitive Testing Results for the Current
Population Survey Food Security Supplement”

RESEARCH REPORT SERIES
(Survey Methodology #2021-06)
Cognitive Testing Results for the Current Population Survey
Food Security Supplement
Kathleen Kephart
Jonathan Katz
Matthew Virgile
Rodney Terry
Jessica Holzberg

Center for Behavioral Science Methods
Research and Methodology Directorate
U.S. Census Bureau
Washington, D.C. 20233

Report issued: November, 2021
Disclaimer: This report is released to inform interested parties of research and to encourage
discussion of work in progress. Any views expressed on the methodological issues are those of
the authors and not necessarily those of the U.S. Census Bureau. The paper has been reviewed for
disclosure avoidance and approved under CBDRB-FY21-CBSM002-003.

Abstract
This report documents the results from cognitive testing of the Current Population Survey Food
Security Supplement (CPS FSS). Testing was conducted from June 2019 to October 2019 by the
Center for Behavioral Science Methods at the U.S. Census Bureau (CBSM). The purpose of this
testing was to: 1) ascertain how participants interpreted new questions and modifications to
existing questions about food insecurity and food purchases, 2) make sure the language in existing
survey questions was still relevant to respondents, and 3) test the feasibility of changing the order
of two topic sections in the survey. Overall, we found that new questions performed well, minor
updates to language helped ensure the questions were understood as intended, and changing the
order did not introduce any issues. This report contains the results from three rounds of testing as
well as CBSM’s final recommendations for wording for the instrument.
Keywords: cognitive testing, cognitive interviews, food insecurity, food security, Current
Population Survey
Suggested Citation: Kathleen Kephart, Jonathan Katz, Matthew Virgile, Rodney Terry,
Jessica Holzberg (2020). Cognitive Testing Results for the Current Population Survey Food
Security Supplement. Research and Methodology Directorate, Center for Behavioral Science
Methods Research Report Series (Survey Methodology #YYYY-##). U.S. Census Bureau.
Available online at:

i

TABLE OF CONTENTS
EXECUTIVE SUMMARY ................................................................................................... iv
1.

INTRODUCTION ........................................................................................................... 1

2.

METHODOLOGY .......................................................................................................... 1
2.1
2.2
2.3

Data Collection............................................................................................................ 1
Analysis....................................................................................................................... 2
Recruiting Methods and Respondent Characteristics ................................................... 2

Table 1: Respondent Characteristics ..................................................................................... 4
3.

RESULTS ........................................................................................................................ 5
3.1

Section I Food Expenditures........................................................................................ 5
3.1.1
3.1.2
3.1.3
3.1.4

3.2

Section II Minimum Spending Need to Have Enough Food ...................................... 13
3.2.1

3.3

3.3.4
3.3.5
3.3.6
3.3.7
3.3.8

Food Stamp Participation (SP1) .................................................................. 22
Free and Reduced Price Breakfast, Lunch, and Snacks (SP6, SP7, NEW
1) ................................................................................................................. 23

Section V Community Food Assistance .................................................................... 24
3.5.1
3.5.2
3.5.3
3.5.4

4.

Balanced vs Healthy .................................................................................... 15
Enough Money for Food vs Money for Enough Food ................................. 16
Table 9 Enough Money for Food vs Money for Enough Food (SH2,
SH3, SH4, SH5, SSH1, SSH2, SSH4, SSH5).............................................. 16
Couldn’t Afford vs There Wasn’t Enough Money (SH1, SSH3) ................. 17
Food Security Screeners (S9) ...................................................................... 18
Food Security Screeners (SS1) .................................................................... 19
Cutting the Size of Meals and Skipping Meals (SH2) ................................. 20
Low Cost Food for Children (SS5).............................................................. 21

Section IV Food Program Participation ..................................................................... 21
3.4.1
3.4.2

3.5

Buying Enough Food to Meet Needs........................................................... 13

Section III Food Sufficiency and Food Security ........................................................ 14
3.3.1
3.3.2
3.3.3

3.4

Food Purchase Places .................................................................................... 5
Food Place Purchase Amounts ...................................................................... 8
Non-Food Spending .................................................................................... 11
Food Purchase Check .................................................................................. 12

Senior Free Meal Programs (SC1, SC2) ...................................................... 24
Emergency Food vs. Free Groceries (SC3, SC3A) ...................................... 26
Free Meal Program Usage (SC4)................................................................. 28
Free Meal Program Community Availability (NEW 4) ............................... 29

CONCLUSIONS ........................................................................................................... 30
ii

5.

REFERENCES .............................................................................................................. 30

LIST OF TABLES

Table 1: Respondent Characteristics………………………………………………………..4
Table 2: Food Purchase Places (S1A, S1B, S1C, S1D) ................................................................. 6
Table 3: Food Place Purchase Amounts (SLEAD, S2, S4, S6) ..................................................... 9
Table 4 Non-Food Spending (S3A, S5A) .................................................................................... 11
Table 5 Food Purchase Check (S8A, S8ZA) ............................................................................... 12
Table 6 Buying Enough Food to Meet Needs (S8B, S8C, S8D).................................................. 13
Table 7 Section Introduction (SS1_LEAD) ................................................................................. 14
Table 8 Balanced vs Healthy (SS4, SS6) .................................................................................... 15
3.3.3

Table 9 Enough Money for Food vs Money for Enough Food (SH2, SH3, SH4,
SH5, SSH1, SSH2, SSH4, SSH5) ............................................................................... 16

Table 10 Couldn’t Afford vs There Wasn’t Enough Money (SH1, SSH3) .................................. 17
Table 11 Food Security Screeners (S9) ....................................................................................... 18
Table 12 Food Security Screeners (SS1) ..................................................................................... 19
Table 13 Cutting the Size of Meals and Skipping Meals (SH2) .................................................. 20
Table 14 Low Cost Food for Children (SS5)............................................................................... 21
Table 15 Food Stamp Participation (SP1) ................................................................................... 22
Table 16 Free and Reduced Price Breakfast, Lunch, and Snacks (SP6, SP7, NEW 1) ................ 23
Table 17 Senior Free Meal Programs (SC1, SC2) ....................................................................... 24
Table 18 Emergency Food vs. Free Groceries (SC3, SC3A) ....................................................... 26
Table 19 Free Meal Program Usage (SC4).................................................................................. 28
Table 20 Free Meal Program Community Availability (NEW 4) ................................................ 29

iii

EXECUTIVE SUMMARY
This report presents findings from cognitive testing of the 2018 Food Security Supplement (FSS)
which is administered on an annual basis in December as part of the Current Population Survey
(CPS). This testing was conducted because the U.S. Department of Agriculture’s Economic
Research Service (ERS) wanted to ensure all question wording remains relevant, since most of the
questions were written over 25 years ago. ERS also wanted to conduct cognitive testing of several
new questions, as well as switching the order of two sections, before fielding them in a split panel
test. Modifications included changes to the Food Purchase section, more inclusive items in the
Ways of Coping with not Having Enough Food section, and moving the Food Sufficiency and Food
Security section before the Food Program Participation section.
Overall, the findings from cognitive testing indicate that most questions in the FSS are being
interpreted as intended and only minor changes, documented in this report, are needed to update
the question wording. We found that the few modifications ERS and the Center for Behavioral
Science Methods (CBSM) recommended seemed to improve the quality of data collected,
especially in the Food Expenditure and Ways of Coping with not Having Enough Food sections.
Switching the order of the Food Sufficiency and Food Security section and the Food Program
Participation section questions also seemed to not introduce any issues.
All the recommended changes in this report will be tested in a split-panel experiment that will be
conducted in the CPS and is planned for September 2020.

iv

1. INTRODUCTION
At the request of the U.S. Department of Agriculture’s Economic Research Service (ERS), the
Center for Behavioral Science and Methods (CBSM) conducted an expert review and cognitive
testing of the 2018 Food Security Supplement (FSS). The FSS has been administered as a
supplemental survey in the Current Population Survey (CPS) in December of every year since
2001. Section I of the FSS, Food Expenditures, is asked of every household in the CPS population
as it is a data source on the amount of money Americans spend on food on a weekly basis. Section
II, Minimum Spending Needed to Have Enough Food, is also asked of everyone. Some questions
on food hardship and food insufficiency are used as screener questions for the remaining sections
of the survey. If the respondent indicates they have experienced food insufficiency, or if the
household income is less than 185 percent above the federal poverty line, they are then asked
Sections III, Food Sufficiency and Security (formerly Section IV), Section IV Food Program
Participation (formerly Section III), and finally Section V Ways of Coping with Not Having
Enough Food.
Before cognitive testing, changes were made to the existing questions based on suggestions from
ERS and an expert review by CBSM. One of the biggest changes to the survey was switching the
order of the sections on Food Sufficiency and Security and Food Program Participation. In the
current survey, Food Program Participation is asked before Food Sufficiency and Security; in
cognitive testing, it was asked fourth. The order was changed because ERS was concerned that
asking about food program participation before asking about food insecurity may have been
influencing the answers to the food security items. In particular, ERS was concerned that
respondents may feel the need to report food insecurity after they have reported participation in
nutrition assistance. Additionally, FSS was written over 25 years ago so ERS was concerned that
some of the questions in the FSS contained dated language that was no longer widely used or
understood. Several new questions proposed by ERS were also included in cognitive testing. Once
the questionnaire was adapted based on the expert review process, three rounds of cognitive testing
were conducted in the greater Washington, D.C. area. During testing, terminology was updated
throughout the survey based on an expert review and a Google Trends search, where applicable.

2. METHODOLOGY
2.1

Data Collection

Thirty interviews over three rounds of iterative cognitive testing were conducted between June and
October 2019 in the greater Washington, D.C. area. Four experienced cognitive interviewers in
CBSM were trained on the cognitive interview protocol and conducted the cognitive interviews.
Participants received $40 cash after the interview. The goals of cognitive interviewing were to
determine:
1) Are the revised and new questions being interpreted consistently as intended?
2) Are the current items ERS or CBSM identified as potentially problematic being correctly
interpreted?
3) Is switching Sections III and IV introducing any unintended issues?
1

U.S. Census Bureau

In each interview, cognitive interviewers administered the full CPS FSS with the new section order
for III and IV, and asked retrospective probes at the end of each section. The CPS FSS is a
Computer Assisted Personal Interview/Computer Assisted Telephone Interview mode, but for
cognitive interviewing we used a modified paper version of the instrument that also contained
follow up probes. In order to test the full instrument with revised and new questions in under 60
minutes and allow time for retrospective probing, we decided to omit a think aloud protocol.
Participants were asked retrospective probes on all revised and new items, as well as any items
that ERS was concerned may be outdated or confusing. All thirty participants received the full
CPS FSS survey, despite the fact most participants screen out of Sections III-V if they do not
indicate food insufficiency or meet the income guidelines when the survey is administered
formally in the CPS. Only age restrictions were used to exclude questions, since these items would
not apply or make sense to households that did not contain the targeted populations.
Upon arrival participants were told that the Census Bureau was working with the US Department
of Agriculture to develop questions for a survey we would conduct for the USDA. They were
asked to sign a consent form and informed that all of their information would be kept confidential.
They were also asked for their permission to audio record and when applicable informed there was
an observer.
Participants were told to treat the survey questions as if the interviewer had come to their home.
They were prompted throughout the survey when the “real interview” was being conducted and
when we were asking some additional background questions.
There were three rounds of testing. The first round of testing had 8 participants, followed by 11 in
round 2 and 11 in round 3. The protocol and questions were modified based on the previous round’s
findings for Rounds 2 and 3. All of these changes are detailed in this report.
2.2

Analysis

Interviewers listened to the audio recording of their interviews and completed summaries in a
standard template for all 30 interviews. Personally identifiable information (PII) such as names
were not included in the summaries and participants were referred to using ID numbers.
Summaries included the participant’s response to each survey or scripted probe question as well
as interviewer observations for each question and probe. After all three rounds a consensus meeting
was held to discuss major findings and any issues that came up during interviews. All summaries
were compiled into a briefing document. Recommendations for the next round were based on these
findings and included in the document. All interviewers independently reviewed the briefing
document before it was sent to ERS.
2.3

Recruiting Methods and Respondent Characteristics

Across all rounds, participants were recruited using posts on Craigslist.com, NEXTdoor.com and
a Census Headquarters blast e-mail. In Rounds 2 and 3, recruitment was also conducted at a
community college and food pantry in the greater Washington DC area. The requirements for
recruitment were that the person had to live with at least one other person and have a household

2

U.S. Census Bureau

income below $50,000 1. This threshold was chosen because the high cost of living in the DC area
means that food insecurity can be a problem even at $50,000. Lowering the income threshold
below this might have limited the pool of potential participants too much. Ideally, participants also
lived with at least one child so that the questions related to children’s food insecurity and school
food programs could be tested. The majority of participants lived with children. Household sizes
ranged from one person to 11, but 23 participants lived with three or more people and seven
participants lived with over five people. Participants also represented a diversity of ages from 18
to over 65 as well as a diversity of education levels. The majority of participants identified as black
(17 people).

1

In Round 2, the income threshold was increased to $75,000 to help increase the number of potential participants.
For Round 3, it was set back at $50,000 or less.

3

U.S. Census Bureau

Table 1: Respondent Characteristics
Participant Characteristics
Household Size
12
2
3
4
5-11
Presence of Children
At least one person under 18 in household
No one under 18 in household
Age
18-40
41-60
61+
Not asked 3
Race
White
Black
Asian
Not asked
Education
High school diploma or less
Some college or Associate’s Degree
Bachelor’s degree or higher
Not asked
Household Income
Less than $15,000
$15,000-$25,000
$25,000-$50,000
Only asked if below $75,000 4
Gender
Male
Female
Not asked

Number of Participants
1
6
11
5
7
20
10
10
2
8
10
1
17
2
10
7
7
6
10
4
6
10
10
7
13
10

One participant lived alone due to a recruitment error. They still met the income guidelines.
The participants who were recruited at a food pantry or community college were not given the full demographic
screener. Due to time and staff constraints, they were interviewed on the spot after confirming they met the basic
recruitment criteria.
4
For the community college recruitment, participants were asked a “yes or no” question if the household income was
below $75,000. For the food pantry recruitment participants were asked if their household income was below
$50,000.
2
3

4

U.S. Census Bureau

3. RESULTS
This report contains only questions that were 1) modified from the original production 2018 FSS
instrument, 2) identified as problematic during cognitive testing, and/or 3) had a scripted probe
question during at least one round of interviewing. Any item from the 2018 CPS FSS that is not
mentioned in this report was not found to be problematic during cognitive testing and we are not
recommending any changes at this time.
Throughout the results section when a change is made to a question that contains follow-up
questions that ask about more specific time frames (“12 months” vs “in the last 30 days”) the
changes were made and recommended for the follow-up questions, but are not discussed below to
avoid redundancy and for length.
We use tables to show the current production question wording in the FSS, the wording tested in
Rounds 1, 2, and 3, and the final recommended wording. Changes made to the questions between
rounds are shown in red.
3.1
3.1.1

Section I Food Expenditures
Food Purchase Places

There were two major changes for Section I. First, we updated the types of places listed as
examples in questions to more accurately reflect current consumer behavior. Second, we tested
grouping questions together to ask about food purchased for consumption at home versus away
from home. See Table 1 for a display of the question wording across rounds.
Round 1 Findings
The first item in Section I asked about food purchased at grocery stores then the second question
asks about more specific food stores such as produce stands, meat markets, warehouse clubs, etc.
The third question asks participants about their food purchases from places for food eaten away
from home, such as at restaurants, cafeterias, or vending machines. Finally participants are asked
a catch all about any food purchases from other (unspecified) places. Participants are then asked
four follow-up questions about how much they spent at each of these places. The ultimate goal of
this series of questions is to determine the total amount spent on food in a typical week. The
individual questions are designed to remind participants about places they may have bought food,
to find out how much they spent, and to determine how much of that was spent on non-food items.
In Round 1, CBSM experimented with combining three items into two items, one of which asked
about food purchases to be eaten at home and the other which asked about food to be eaten away
from home. This was done to reduce the number of questions asked and potentially improve recall
by grouping the types of food purchases by where they were typically consumed.

5

U.S. Census Bureau

Item
Number
S1A

S1B

S1C

S1D

Table 1: Food Purchase Places (S1A, S1B, S1C, S1D)
Current CPS FSS Survey
Wording
First did (you/anyone in your
household) shop for food at a
supermarket or grocery store
last week?

Think about other places where
people buy food, such as meat
markets, produce stands,
bakeries, warehouse clubs, and
convenience stores. Did
(you/anyone in your household)
buy food from any stores such
as these LAST WEEK?
LAST WEEK, did (you/anyone
in your household) buy food at a
restaurant, fast food place,
cafeteria, or vending machine?
(Include any children who may
have bought food at the school
cafeteria).
Did (you/anyone in your
household) buy food from any
other kind of place LAST
WEEK?

Round 1 Wording

Round 2 Wording

First, did (anyone in your household)
shop for food at a supermarket,
grocery store, superstore (for example
Walmart or Target), dollar store,
butcher shop, farmers market or club
store LAST WEEK?

First, did (anyone in your
household) shop for food at a
supermarket, grocery store,
superstore (for example
Walmart or Target)?
Think about other places where
people buy food, such as dollar
stores, pharmacies, club stores,
farmers markets, or butcher
shops. Did (you/anyone in
your household) buy food from
any stores such as these LAST
WEEK?
No changes

LAST WEEK, did (you/anyone in
your household) buy food at a
restaurant, fast food place, cafeteria,
deli, convenience store, or vending
machine?

No changes

No changes

6

Round 3 Wording/Final
Recommended Wording
First, did (you/anyone in your
household) shop for food at a
supermarket, grocery store,
Walmart or Target LAST WEEK?
Think about other places where
people buy food, such as dollar
stores, pharmacies, club stores,
farmers markets, or online. Did
(you/anyone in your household)
buy food from any stores such as
these LAST WEEK?
LAST WEEK, did (you/anyone in
your household) buy food at a
restaurant, fast food place,
cafeteria, deli, convenience store,
or vending machine?

Did (you/anyone in your
household) buy food from any
other kind of place LAST WEEK?

U.S. Census Bureau

For Round 1, the wording to ask about food purchases for consumption at home combined the first
and second questions into one item. In addition to combining the items, the term “supercenter (for
example Walmart or Target)” was added to the question because these stores account for a
significant share of food expenditure, according to ERS. ERS also requested the replacement of
“meat market” with “butcher shop,” “produce stands” with “farmers market,” and “warehouse
clubs” with “club stores” because a Google Trends search showed the latter terms were more
common than the former. This wording was also more consistent with other food surveys such as
the National Household Food Acquisition and Purchase Survey, which gathers data about where
American households buy food.
For purchases to be consumed away from home, the modification to the existing item was minor
and included the addition of “deli” and “convenience store” to the list of examples.
The list of potential places in the new combined item proved to be overwhelming for interviewers
to read and difficult for participants to remember all the choices mentioned. There did not seem to
be any reduction in burden gained by combining the two items. Participants seemed to forget the
full list of stores, and CBSM was concerned it would be problematic to get interviewers to read
the full list during production data collection. Based on spontaneous comments made by
participants, it appeared that the idea of grouping stores by where the food might be consumed was
not helpful in reminding them about purchases. Both grocery stores and convenience stores had a
lot of ready to eat foods and participants mentioned fast food was often taken to be eaten at home.
The two items did not seem to provide the intended stimulus of food to be eaten at home vs away
from home. Given there is a gray area as far as where food is ultimately consumed, this
methodology was not helpful to participants to remind them about food purchases.
One participant mentioned buying food at CVS, but said she thought it did not really fit in any of
the questions asked so she did not include the cost of her purchases at the CVS when she was later
asked to report the amount spent.
Round 1 Recommendations
CBSM recommended splitting the two items back into three items but keeping the modifications
to the types of places listed. CBSM also recommended adding “pharmacy” to the second question
about other places to buy food because a significant proportion of food stamps are redeemed at
CVS and drug stores according to ERS.
Round 2 and 3 Findings
In Rounds 2 and 3 it worked better to separate the two items about food purchases at grocery stores
(or superstores) and food purchases at other stores that sell groceries, such as butcher shops, and
club stores. This was easier for interviewers to read and for participants to recall the full list of
places mentioned before they answered.

7

U.S. Census Bureau

In Round 2, one participant mentioned buying food online and was not sure where a person would
report this because no item specifically mentioned this option. They also mentioned that there has
been a recent rise in online grocery purchases.
The fourth question in Section 1 asks about “any other kind of place.” ERS was curious whether
this item was actually acting as a catch-all question and what other kinds of places participants
were including. This question worked as a catch-all question for most participants, but several
participants across all three rounds said they were thinking of grocery stores or another place that
was specifically listed by a preceding question when they said “yes” to this question. Most of these
participants reported the money spent under the grocery store amount question, discussed below
in 3.1.2, and they then reported $0 when asked about the amount spent at the “other place”. A few
participants incorrectly reported the amount spent under the “other place” question instead, but
still ended up ultimately reporting the correct total since the amount is summed across all
questions. No participants incorrectly included the amount twice under different questions. The
“other places” mentioned included by participants included a fall festival, a home food catering
service, an online shopping service like Amazon (mentioned before the word “online” was added
in Round 3), and food trucks.
Round 2 and 3 Recommendations
After Round 2, CBSM wanted to shorten the first question (S1A) because it continued to be
difficult for interviewers to read and participants to remember. A Google Trends search showed
that “supercenter” is a relatively unused term and people more commonly search for Walmart or
Target. In fiscal year 2019, 53 percent of all SNAP benefits were redeemed at super stores, such
as Walmart (USDA Food and Nutrition Service, 2020). Therefore, we thought it was important
to include it in the question. However, most respondents were less familiar with “supercenter” than
the brand names of Walmart or Target. For this reason, CBSM recommended dropping the word
“supercenter” and just saying “Walmart or Target.”
CBSM also recommended adding the word “online” to the second question in place of “butcher
shops.” We recommended this because “butcher shops” are less common than the other types of
stores mentioned. This also allows for the addition of “online” without increasing the length of the
question.
3.1.2

Food Place Purchase Amounts

Round 1 to 3 Findings
These follow-up questions are intended to capture the amount of money spent at each of the places
the respondent indicates they spent money. ERS expressed concern about the use of the word
“actually” in the lead and questions S2, S4, and S6. Prior to cognitive testing, we changed the
wording of the question to “How much did (you/anyone in your household) spend in total at …”
which we thought would be more polite and correctly reflect the intent of the question. Across
three rounds of testing, the modified question worked well.

8

U.S. Census Bureau

Table 2: Food Place Purchase Amounts (SLEAD, S2, S4, S6)
Item
number

Lead

S2

S4

5

Current Survey Wording

Round 1 Wording

Round 2-3 Wording

Final Recommended Wording

Now I'm going to ask you about
the ACTUAL amount you spent
on food LAST WEEK in all the
places where you bought food.
Then, since LAST WEEK may
have been unusual for you, I will
ask about the amount you
USUALLY spend.

Now I'm going to ask you about the
TOTAL amount you spent on food
LAST WEEK in all the places
where you bought food. Then,
since LAST WEEK may have been
unusual for you, I will ask about
the amount you USUALLY spend.

N/A

Now I'm going to ask you about the
TOTAL amount you spent on food
LAST WEEK in all the places where
you bought food. Then, since LAST
WEEK may have been unusual for
you, I will ask about the amount you
USUALLY spend.

How much did (you/anyone in
your household) ACTUALLY
spend at supermarkets and grocery
stores LAST WEEK (including
any purchases made with
or food stamp
benefits)?

How much did (anyone in your
household) spend in total at
supermarkets, grocery stores,
superstores (like Walmart or
Target), dollar stores, butcher
shops, farmers markets or club
stores LAST WEEK (including any
purchases made with (SNAP/ Food
Supplement Program) or food
stamp benefits)?

How much did (anyone in
your household) spend in total
at supermarkets, grocery
stores, or superstores (like
Walmart or Target) LAST
WEEK (including any
purchases made with (SNAP/
Food Supplement Program) or
food stamp benefits)?

How much did (you/anyone in your
household) spend in total at
supermarkets, grocery stores,
Walmart or Target LAST WEEK
(including any purchases made with
or food stamp
benefits)?

How much did your
household spend at stores
such as dollar stores, drug
stores 5, club stores, farmers
markets, or butcher shops
LAST WEEK (including any
purchases made with (SNAP/
Food Supplement Program) or
food stamp benefits)?

How much did (you/your household)
spend at stores such as dollar stores,
pharmacies, club stores, farmers
markets, or online LAST WEEK
(including any purchases made with
or food stamp
benefits)?

How much did (you/your
household) spend at stores such as
meat markets, produce stands,
bakeries, warehouse clubs, and
convenience stores LAST WEEK
(including any purchases made
with or food
stamp benefits)?

In Round 2 there was a mistake in the protocol and “pharmacy” was mistakenly replaced with “drug stores” in item S4. No known issues came up in testing
from this and the mistake was corrected before Round 3.

9

U.S. Census Bureau

Item
number

S6

Current Survey Wording
How much did (you/your
household) spend for food at
restaurants, fast food places,
cafeterias, and vending machines
LAST WEEK, not including
alcohol purchases?

Round 1 Wording

Round 2-3 Wording

How much did (your household)
spend for food at restaurants, fast
food places, cafeterias, delis,
convenience stores, and vending
machines LAST WEEK, not
including alcohol purchases
(including any purchases made
with (SNAP/ Food Supplement
Program) or food stamp benefits)?

10

N/A

Final Recommended Wording
How much did (you/your household)
spend for food at restaurants, fast
food places, cafeterias, delis,
convenience stores, and vending
machines LAST WEEK, not
including alcohol purchases
(including any purchases made with
(SNAP/ Food Supplement Program)
or food stamp benefits)?

U.S. Census Bureau

ERS was wondering whether participants were including children buying school lunch in the
amount reported under the appropriate follow-up question that mentions cafeterias. When probed,
participants with children said they were thinking about them in their answer; however, the price
of their lunch was free or a reduced amount, and they were not sure how much it was. Parents who
sent a lunch with their child or purchased a full price lunch did include this amount in what they
reported.
When asked if they included food stamps in the amounts reported throughout Section I,
participants who had indicated they received food assistance said ‘yes’.
Round 1 to 3 Recommendations
Changes to the lead in and items S2, S4, and S6 included changes in the places asked about and
asking about “the total amount spent” vs the “actual amount.” These changes seemed to work well
in cognitive testing and no further changes were recommended.
3.1.3

Non-Food Spending

Table 3 Non-Food Spending (S3A, S5A)
Item
number

S3A& S5A

Current Survey Wording/ Round 1-3 Wording/Final Recommended Wording
How much of the $(fill) was for non-food items, such as pet food, paper products, alcohol,
detergents, or cleaning supplies?

Round 1 to 3 Findings
After asking about the amount spent at each of the places for items S2 and S4 (3.1.2), participants
are then asked to report how much of the amount was for non-food purchases. ERS was concerned
that the list of examples may not be comprehensive enough and participants may therefore be
underreporting. We cautioned against adding additional examples before verifying that
participants were excluding non-food purchases that were not explicitly listed. We recommended
testing the question as it was and then during probing we reminded participants how much they
reported and asked about the types of non-food items they purchased. Across all three rounds, most
participants gave an example of something that was not explicitly listed or something more specific
than the examples listed (for example, for “paper products” they reported paper towels or plates.).
Examples of other items reported included toiletries, medicine, makeup, clothing, and shoes. This
suggested participants were not underreporting in response to this question and no further
modification was needed.
Round 1 to 3 Recommendations
Given participants were interpreting the item as intended and including all non-food items, we did
not recommend any changes to this item.

11

U.S. Census Bureau

3.1.4
Food Purchase Check
Table 4 Food Purchase Check (S8A, S8ZA)
Item
Number

S8A

S8ZA

Current Survey Wording
Let's see, (your household) spent
about (SUM OF 1B+2B+3B-1C) on
food LAST WEEK. Now think
about how much (anyone in your
household) USUALLY spends.
How much (does your household)
USUALLY spend on food at all the
different places we've been talking
about IN A WEEK? (Please
include any purchases made with
(SNAP/ Food Supplement Program)
or food stamp benefits). Do not
include non-food items such as pet
food, paper products, detergent or
cleaning supplies.
Let's see, it seems that (your
household) did not buy any food
LAST WEEK. If that is because
you shop for food infrequently, how
much would the weekly average be
over several weeks?

Round 1-3 Wording

No changes

Let's see, it seems that
(your household) did
not buy any food
LAST WEEK. If that
is because you shop
for food only once in a
while, how much
would you say you
spend in total in a
month?

Final Recommended Wording
Let's see, (your household) spent
about (SUM OF 1B+2B+3B-1C) on
food LAST WEEK. Now think
about how much (anyone in your
household) USUALLY spends.
How much (does your household)
USUALLY spend on food at all the
different places we've been talking
about IN A WEEK? (Please include
any purchases made with (SNAP/
Food Supplement Program) or food
stamp benefits). Do not include
non-food items such as pet food,
paper products, detergent or cleaning
supplies.
Let's see, it seems that (your
household) did not buy any food
LAST WEEK. If that is because you
shop for food only once in a while,
how much would you say you spend
in total in a month?

Round 1 to 3 Findings
After asking about the types of places and amount spent on food, Section I branches off into two
paths depending on whether the participant reported any food expense in the past week.
If a participant reports any amount, they are asked S8A which is a check that the week was
representative of a current typical week, and that there was not an event that changed food
purchasing habits. ERS requested that CBSM probe on how participants were interpreting the word
“usually” in the question. Across three rounds, the synonyms mentioned to interpret “usually”
included: normally, consistently, in general, or “on average spend on food in a week,” as opposed
to a week that is a “splurge” or special occasion. The question appeared to be interpreted as
intended, it worked as a check on the weekly total and whether the last week was outside of their
normal spending pattern.
The other question is for participants who reported no food expenses in the prior week, they are
asked S8ZA which asks them to calculate an average weekly purchase amount over several weeks.
CBSM was concerned that the cognitive burden of this question was too high because
“infrequently” is a higher register word and the cognitive task of calculating a weekly average over
“several” weeks is difficult and ambiguous. It is generally better to not ask participants to calculate
a number when we can ask a simpler question and then do the computations ourselves. We did not
12

U.S. Census Bureau

have any participants across the three rounds who reported no food purchases in the preceding
week. However, starting in Round 2, we asked participants during probing if they would be able
to report a monthly total of their food expenses. Participants reported they would be able to do this
and mentioned using a variety of tactics, including thinking back to an unusual month where they
only shopped once all month, but most said they would multiply the weekly amount by four.
Round 1 to 3 Recommendations
No further changes were recommended to S8A, the question for participants who reported any
weekly food expense.
Given most participants can report a monthly amount, CBSM recommended changing S8ZA, the
question for participants with no weekly food expenses, to the proposed wording.
3.2

Section II Minimum Spending Need to Have Enough Food

3.2.1

Buying Enough Food to Meet Needs

Table 5 Buying Enough Food to Meet Needs (S8B, S8C, S8D)
Item
Number

Current Survey Wording/ Round 1-3 Wording/Final Recommended Wording

S8B

In order to buy just enough food to meet (your needs/the needs of your household), would you need
to spend more than you do now, or could you spend less?

S8C

[If more] About how much MORE would you need to spend each week to buy just enough food to
meet the needs of your household?

S8D

[If less] About how much LESS could you spend each week and still buy enough food to meet the
needs of your household?

Round 1-3 Findings
Section II contains only two questions, but they are useful for understanding respondent’s food
spending. Participants are asked if they need to spend more than they do now or if they could spend
less to meet the food needs of their household. They are then asked a follow-up on how much more
or less they could spend. ERS was concerned that the phrase “the needs of your household” was
too ambiguous and that they may need to add a more specific definition for this term and mention
being able to afford “healthy foods” in the question. They also raised the issue that it was unclear
if participants were including the amount they receive in food aid in their answer. CBSM did not
recommend any changes to the item, but we did ask participants several probes about the question.
Across all three rounds, participants interpreted “the needs of your household” to mean enough
food to meet the nutritional needs of everyone in their household. When asked to elaborate they
said that the question was asking if they could afford to spend less than they do now and still buy
enough food for everyone, or alternatively if they weren’t meeting the needs of their household
and needed more money. CBSM did not recommend any changes to the item since it was being
consistently interpreted as intended and a reference to “healthy foods” could introduce more
confusion or variation in interpretation.
13

U.S. Census Bureau

A few participants in each round said that they currently spend just the right amount and that
neither more nor less fits their circumstances. In the production FSS, interviewers can accept
“neither” as an answer and move on without a follow-up question. ERS said they did not want to
offer neither as a third option that is read aloud.
CBSM did not directly ask participants if they included food stamps in their answer, but several
participants indicated through spontaneous comments, that they were including their food stamp
allocation when they answered the first item. In fact, one participant interpreted the item to be
asking if her food stamp money was enough to meet her needs and she said she needed more
because she had to spend additional money out of her own pocket.
Round 1 to 3 Recommendations
CBSM did not recommend changes to this item because it appears to be interpreted as intended by
the majority of participants. ERS should be aware that participants are probably including their
food stamp benefits when they respond to this question.
3.3

Section III Food Sufficiency and Food Security

Section III: Food Sufficiency and Security used to be Section IV what is now Food Program
Participation. Swapping the order of these questions did not seem to cause any confusion.
Table 6 Section Introduction (SS1_LEAD)
Item Number

Current Survey Wording

SS1_LEAD

People do different things when they are
running out of money for food to make
their food or their food money go
further.

Round 1 to 3 Wording/ Final Recommended
Wording
The next questions are about the food eaten in
your household in the last 12 months since
December of last year, and whether you were
able to afford the food you need.

Round 1 to 3 Findings
After the section order was switched CBSM recommended replacing the lead-in text with just the
existing introductory text for the section. ERS was concerned how “afford the food you need” was
being interpreted so we probed on this in Rounds 1 and 2. All participants seemed to have a clear
and consistent interpretation of the phrase and they indicated it was asking if they could afford to
buy enough food to get by. Some example quotes:
• “I can purchase it and it’s enough to feed the family”
• "If you have the money to actually buy the things you need, and if you don't, then you go
without. The difference between that is I don't need to go without but there's food there.
It doesn't matter whether I want to eat it or not.”
Round 1 to 3 Recommendations
We recommend using the lead in text for Section III as it is presented above. “Afford the food you
need” is consistently interpreted.
14

U.S. Census Bureau

3.3.1

Balanced vs Healthy

Table 7 Balanced vs Healthy (SS4, SS6)
Item
Number
SS4

SS6

Current Survey Wording

Round 1-3 Wording

"(I/we) couldn't afford to eat balanced
meals."
"(I/we) couldn't feed (the child in
(my/our) household/the children) a
balanced meal, because (I/we) couldn't
afford that."

"(I/we) couldn't afford to
eat healthy meals."
"(I/we) couldn't feed (the
child in (my/our)
household/the children) a
healthy meal, because
(I/we) couldn't afford that."

Final Recommended
Wording
"(I/we) couldn't afford to eat
balanced meals."
"(I/we) couldn't feed (the
child in (my/our)
household/the children) a
balanced meal, because
(I/we) couldn't afford that."

Round 1 to 3 Findings
ERS was concerned that the term “balanced” used in two questions in Section III may be dated
and ambiguous. ERS had previously tested a version of item SS4 that replaced the term “balanced”
with the term “nutritious.” They found this did not work well and cautioned CBSM to not use the
term “nutritious.” After our expert review, we recommended replacing the term “balanced” with
“healthy,” because Google Trends revealed the term “healthy meals” was used more frequently
than “balanced meals.” However, after three rounds of cognitive testing we recommended
reverting back to the original wording and not using the word “healthy.” The term “healthy” was
confounded with unintended constructs including, but not limited to, “organic,” “more expensive
name brands,” and “fresh vs. canned produce.” Further, the terms “balanced” and “healthy” do not
appear to be measuring the same construct, which may be problematic for data trends. When
probed, almost all participants could interpret what “balanced” meant and generally viewed it as
meaning there was a variety of foods in a meal. The handful of participants who struggled to
interpret “balanced” did not appear to have a clearer understanding of “healthy.” Starting in Round
2, we probed participants to see if they thought “balanced” and “healthy” meant the same thing.
The majority of participants said it did, but most then appeared to contradict this by saying that a
meal could be balanced without being healthy. Some participants said that balanced means
“meat/vegetable/starch” and that a balanced meal could become unhealthy if it was fried. Other
participants said that “balanced” ties back to the “chart/pyramid.” Despite the fact “balanced” can
still mean “unhealthy” to participants, it does convey the idea that a person can afford a variety of
foods such as meats, starch, and a vegetable.
ERS also wanted to know if participants thought that healthy or balanced is different for children
and adults. For item SS6 we asked participants with children if what is healthy for a child is also
healthy for adults. All participants said that what is healthy for adults is the same as for children.
However, the examples given were slightly different and included more processed foods like
pudding, Jell-O, and processed lunch snacks. One participant mentioned that kids need to avoid
food dyes; they seemed to imply this was less of a concern for adults.
Round 1 to 3 Recommendations

15

U.S. Census Bureau

CBSM recommended reverting back to the current instrument wording using the term “balanced”
instead of “healthy” in items SS4 and SS6.
3.3.2
3.3.3
Item
Number

SSH2

SSH3

SSH4

SSH5

SSH1

SSH2

SSH4

SSH5

Enough Money for Food vs Money for Enough Food
Table 8 Enough Money for Food vs Money for Enough Food (SH2, SH3, SH4,
SH5, SSH1, SSH2, SSH4, SSH5)
Current Survey Wording

Round 1 Wording

In the last 12 months, did (you/ you
or other adults in your household)
ever cut the size of your meals or
skip meals because there wasn't
enough money for food?

In the last 12 months, did (you/
you or other adults in your
household) ever cut the size of
your meals or skip meals
because there wasn't money for
enough food?

In the last 12 months, did you ever
eat less than you felt you should
because there wasn't enough money
for food?

In the last 12 months, did you
ever eat less than you felt you
should because there wasn't
money for enough food?

In the last 12 months, were you ever
hungry but didn't eat because there
wasn't enough money for food?
In the last 12 months, did you lose
weight because there wasn't enough
money for food?

In the last 12 months, were you
ever hungry but didn't eat
because there wasn't money for
enough food?
In the last 12 months, did you
lose weight because there
wasn't money for enough food?

In the last 12 months, did (you/you
or other adults in your household)
ever not eat for a whole day because
there wasn't enough money for
food?

In the last 12 months, did
(you/you or other adults in
your household) ever not eat
for a whole day because there
wasn't money for enough food?

In the last 12 months, did you ever
cut the size of (the child's/any of the
children's) meals because there
wasn't enough money for food?

In the last 12 months, did you
ever cut the size of (the
child's/any of the children's)
meals because there wasn't
money for enough food?

In the last 12 months, did (the child/
any of the children) ever skip a meal
because there wasn't enough money
for food?

In the last 12 months, did (the
child/ any of the children) ever
skip a meal because there
wasn't money for enough food?

In the last 12 months, did (the
child/any of the children) ever not
eat for a whole day because there
wasn't enough money for food?

In the last 12 months, did (the
child/any of the children) ever
not eat for a whole day because
there wasn't money for enough
food?

16

Round 2-3 Wording/Final
Recommended Wording
In the last 12 months, did
(you/ you or other adults in
your household) ever cut
the size of your meals or
skip meals because there
wasn't enough money for
food?
In the last 12 months, did
you ever eat less than you
felt you should because
there wasn't enough money
for food?
In the last 12 months, were
you ever hungry but didn't
eat because there wasn't
enough money for food?
In the last 12 months, did
you lose weight because
there wasn't enough money
for food?
In the last 12 months, did
(you/you or other adults in
your household) ever not
eat for a whole day because
there wasn't enough money
for food?
In the last 12 months, did
you ever cut the size of (the
child's/any of the
children's) meals because
there wasn't enough money
for food?
In the last 12 months, did
(the child/ any of the
children) ever skip a meal
because there wasn't
enough money for food?
In the last 12 months, did
(the child/any of the
children) ever not eat for a
whole day because there
wasn't enough money for
food?

U.S. Census Bureau

Round 1 Findings
CBSM recommended a change throughout Section III in questions that use the phrase “there
wasn’t enough money for food” because we were concerned that participants may interpret this to
mean “there was no money for any food” rather than “there wasn’t money for enough food.” For
Round 1, the phrase was changed to “there wasn’t money for enough food.” During probing, we
gave participants a handout and asked them to compare the phrases. Participants said they did not
see much difference, and most said “enough money for food” is the way they and the people they
know talk about it. We reverted to the production wording for Rounds 2 and 3; the current survey
wording worked well in these rounds.
Round 1 Recommendation
CBSM recommended reverting back to the original wording of “there wasn’t enough money for
food” after Round 1.
3.3.4

Couldn’t Afford vs There Wasn’t Enough Money (SH1, SSH3)

Table 9 Couldn’t Afford vs There Wasn’t Enough Money (SH1, SSH3)
Item
Number

SH1

SSH3

Current Survey Wording
(The child in (my/our) household
was/The children were) not eating
enough because (I/we) just couldn't
afford enough food.
In the last 12 months, (was the
child/were the children) ever hungry
but you just couldn't afford more
food?

Round 1-3 Wording
(The child in (my/our)
household was/The
children were) not eating
enough because there
wasn’t enough money for
food.
In the last 12 months, (was
the child/were the
children) ever hungry
because there wasn’t
enough money for food?

Final Recommended
Wording
(The child in (my/our)
household was/The children
were) not eating enough
because there wasn’t enough
money for food.
In the last 12 months, (was
the child/were the children)
ever hungry because there
wasn’t enough money for
food?

Round 1 to 3 Findings
Several of the questions in Section III that ask about food for children used the language of “but
you just couldn’t afford more food?” rather than the more neutral phrasing of “there wasn’t enough
money for food,” which is the default phrasing in Section III used when asking about adult food
insecurity. In order to try to increase consistency across the survey and reduce the potential for
social desirability bias, CBSM recommended replacing the phrase “but you just couldn’t afford
more food” with “there wasn’t enough money for food.” This affected items SH1 and SSH3. No
issues came up with this wording across all three rounds of testing.
Round 1 to 3 Recommendations
The new wording worked well across three rounds of testing and this wording was recommended
as the final wording.
17

U.S. Census Bureau

3.3.5

Food Security Screeners (S9)

Table 10 Food Security Screeners (S9)
Item
Number

S9

Current Survey Wording

Round 1-3 Wording

Final Recommended Wording

In the last 12 months, since
December of last year, did
you ever run short of money
and try to make your food or
your food money go further?

In the last 12 months, since
[Month of interview] of last
year, did you ever run short
of money and try to make
your food or your food
money go further?

In the last 12 months, since [Month
of interview] of last year, did you
ever run short of money and try to
make your food or your food money
go further?

Round 1 to 3 Findings
Section III contains a screener question that is used to determine if additional questions about food
insecurity and assistance will be asked. The first item, S9, was kept at the beginning of the new
Section III to still be used as a screener for Sections IV and V. No issues came up with the question
being asked at the beginning of Food Sufficiency and Security.
Round 1 to 3 Recommendation
CBSM recommends keeping S9 at the beginning of the new Section III Food Sufficiency and
Security.

18

U.S. Census Bureau

3.3.6

Food Security Screeners (SS1)

Table 11 Food Security Screeners (SS1)
Item
Number

Current Survey Wording
Which of these statements best describes the food eaten in
your household-- enough of the kinds of food (we) want to
eat, enough but not always the kinds of food (we) want to
eat, sometimes not enough to eat, or often not enough to
eat?

Item
Number

SS1A

SS1

SS1B

19

Round 1-3 Wording
Which of these statements
best describes the food eaten
in your household-- (We)
could always afford to eat
healthy meals. (We) could
always afford enough to eat
but not always healthy meals.
Sometimes (we) could not
afford enough to eat. Often we
could not afford enough to
eat.
Which of these statements
best describes the food eaten
in your household-- enough of
the kinds of food (we) want to
eat, enough but not always the
kinds of food (we) want to eat,
sometimes not enough to eat,
or often not enough to eat?

Final Recommended
Wording
Which of these statements best
describes the food eaten in your
household-- enough of the
kinds of food (we) want to eat,
enough but not always the
kinds of food (we) want to eat,
sometimes not enough to eat, or
often not enough to eat?

U.S. Census Bureau

Round 1 to 3 Findings
ERS wanted to test a new version of the second screener question SS1. During our expert review,
CBSM noted that the phrase “the kinds of food we want to eat” may be problematic because it is
ambiguous and may be interpreted to mean “healthy” or “expensive junk food.” The intention of
the question is to measure food insufficiency by asking participants to identify whether they can
afford enough to eat, including a variety of foods. CBSM proposed a second version be tested as
a mini “split ballot,” where half of the cognitive testing participants hear the current instrument
version, SS1A during the interview, and then hear the alternative version, SS1B during probing.
The other half of participants heard SS1B during the interview and SS1A during probing. SS1A
and SS1B can be seen in the table above.
In Rounds 1 and 2, some participants chose different answers for each version, indicating the items
are not measuring the same construct. When asked if the questions were asking the same thing,
some participants said they felt the two items were the same while others said they were asking
different things. We probed participants to see what they think each version is asking, and the
predominant theme was that the “kinds” version is asking if they can afford whatever they want to
eat that may not necessarily be healthy.
As one participant put it: "[Kinds of food”] just makes a little bit more sense because…we just
want to eat the food we want to eat. It doesn't necessarily have to be healthy meals."
However, participants thought the “healthy” version was asking if they can afford to eat healthy
independent of whether that food is what they want to eat or not. Further, throughout the survey
the word “healthy” was found to be ambiguous and problematic for participants to consistently
define. While we hoped to improve upon SS1A, SS1B was measuring a very different construct
and SS1A, the current instrument wording was not difficult for participants to interpret.
Round 1 to 3 Recommendations
CBSM recommends keeping the current survey wording of “kinds.”
3.3.7

Cutting the Size of Meals and Skipping Meals (SH2)

Table 12 Cutting the Size of Meals and Skipping Meals (SH2)
Item
Number
SH2

Current Survey Wording/Round 1-3 Wording/Final Recommended Wording
In the last 12 months did (you/you or other adults in your household) ever cut the size of your meals
or skip meals because there wasn’t enough money for food?”

Round 1 to 3 Findings
ERS was concerned that item SH2 was confusing to participants because it is asking about two
different behaviors and therefore may be difficult for participants to know which they are saying
“yes” to, “skipping meals” or “cutting the size of your meals.” They also wanted to be sure that
participants correctly interpreted the phrase “cutting the size of your meals.” We probed
20

U.S. Census Bureau

specifically on this item in Rounds 1 and 2. When probed, participants felt that “cut the size of
meals” meant having smaller portions. They were also clear that the question was asking if they
had to do this for cost reasons versus voluntary reasons, such as being on a diet to lose weight or
feeding an unexpected guest. Several participants indicated they had cut the size of meals for these
voluntary reasons, but they all correctly said “no” for this item. Some participants said that “cut
the size of meals” meant having small portions and eating just enough to prevent serious hunger.
In general, participants did not seem to have any issues with the question asking about both cutting
the size and skipping meals. The item seems to be measuring the same underlying theme of eating
less food because there is not enough, despite asking about two behaviors. One participant even
said cutting the size of meals and skipping an entire meal were the same thing.
Round 1 to 3 Recommendations
CBSM recommended keeping the current survey wording for SH2.
3.3.8

Low Cost Food for Children (SS5)

Table 13 Low Cost Food for Children (SS5)
Item
Number

Current Survey Wording/Round 1-3 Wording/ Final Recommended Wording

SS5

(I/we) relied on only a few kinds of low-cost food to feed (the child/children) in (my/our) household
because (I was/we were) running out of money to buy food.

Round 1 to 3 Findings
ERS wanted to know more about what kind of low-cost food people are feeding children in item
SS5. Participants who lived with children were asked about what kind of low-cost food they had
used or had heard of people using. The types of food mentioned included hot dogs and rice, ramen,
noodles, canned food (such as spaghetti O’s), Peanut Butter and Jelly, cereal, macaroni and cheese,
and generic brand frozen foods like pizzas, french fries, fish sticks, and chicken nuggets.
Participants also mentioned tactics like buying food in bulk at Costco or buying it at discount stores
like Aldi, buying whatever was on sale that week, and buying pre-packaged lunch meat instead of
meat from the deli. A major theme among participants’ responses was that processed food is
cheaper than fresh food. One participant spontaneously lamented they felt bad they could not
afford organic food for their children as they seemed to think it was healthier. For both parents
who had used low cost food and those who had not, there was a general sense that low cost food
is less healthy, but it is what you have to eat to feel full.
Round 1 to 3 Recommendations
The current item is working as intended and CBSM does not recommend any changes.
3.4

Section IV Food Program Participation

New introductory text from Feeding America’s survey was added at ERS’s request. This text
smoothed the transition from the food-insecurity items to the food program participation items:
“Sometimes people need help getting food for their household. There are many programs that can
21

U.S. Census Bureau

help.” CBSM agreed the text was an effective transition and lead in to Section IV, it worked well
during cognitive testing.
3.4.1

Food Stamp Participation (SP1)

Table 14 Food Stamp Participation (SP1)
Item
Number
SP1

Current Survey Wording/ Round 1 to 3 Wording/Final Recommended Wording
In the past 12 months, since December of last year, did (you/anyone in this household) get or food stamp benefits?

Round 1 to 3 Findings
ERS was considering adding “EBT cards” to the list of benefits in item SP1, but CBSM instead
recommended probing on the current item to see if the current terminology is being recognized.
The current format fills in the specific state program name based on which state the participant
lives in (for DC and VA “SNAP” for MD “Food Supplement Program”) and also mentions “food
stamp benefits.” CBSM probed on this question in Rounds 1 and 2 and found that while not many
MD participants readily knew what “Food Supplement Program” meant, they were familiar with
Food Stamp Benefits. When probed on what they had heard food stamp benefits called there was
a split in whether participants knew the benefits as SNAP, food stamps, or EBT. No one who lived
in MD readily volunteered “Food Supplement Program” as the more common name. All
participants were familiar with food stamp benefits and anyone who was familiar with EBT also
knew what either SNAP or Food Stamps were. ERS mentioned a potential complication of using
“EBT,” that additional aid for TANF and the Women Infant and Children program can be loaded
on EBT cards, which may lead to incorrect “Yes” responses. CBSM therefore did not recommend
any changes to this item. The item seemed to work as intended across all three rounds of testing,
for both participants who had and had not received food stamp benefits.
Round 1 to 3 Recommendation
CBSM recommended keeping the current survey wording with no modifications.

22

U.S. Census Bureau

3.4.2

Free and Reduced Price Breakfast, Lunch, and Snacks (SP6, SP7, NEW 1)

Table 15 Free and Reduced Price Breakfast, Lunch, and Snacks (SP6, SP7, NEW 1)
Item
Number
SP6

SP7

NEW 1

Current Survey Wording

Round 1- 3 Wording

Final Recommended Wording

During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive free or reduced cost
lunches at school?
During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive free or reduced cost
breakfasts at school?

During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive free or reduced-price
lunches at school?
During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive free or reduced-price
breakfasts at school?
During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive a free or reduced-price
meal or snack at an afterschool
program or daycare?

During the past 30 days, did any
children in the household
(between 5 and 18 years old)
receive free or reduced-price
lunches at school?
During the past 30 days, did any
children in the household
(between 5 and 18 years old)
receive free or reduce- price
breakfasts at school?
During the past 30 days, did any
children in the household
(between 5 and 18 years old)
receive a free or reduced-price
meal or snack at an afterschool
program or daycare?

New item

Round 1 to 3 Findings
ERS requested that the questions about “free and reduced-cost breakfast” SP7 and “lunch” SP6 be
changed respectively to “free and reduced-price breakfast” and “free and reduced-price lunch.”
Given this language is consistent with how the USDA Food and Nutrition Service (FNS), who
oversees the program, refers to the free and reduced-price lunch and breakfast program, CBSM
agreed with this change. There was no issue with this change across all three rounds of testing.
ERS expressed concern that the current item, SP7, which asks about free and reduced-price
breakfast may not be capturing a new program in which children were fed breakfast on the bus
rather than literally at school. Rather than changing the item and possibly introducing confusion
for most participants who are not enrolled in such programs, we instead asked during probing if
they would include free or reduce price breakfast that was given on the way to school. Participants
were confused by the idea that kids could receive breakfast on the bus. They were unable to answer
the probe because they imagined the only way kids would get breakfast on the bus is if parents
sent something with the child. After one round, ERS agreed to stop probing on the breakfast on
the bus program and decided to not further modify item SP6.
A new item, NEW1, was added at ERS’s request to Section IV which asked if any children between
5-18 years old had received a free or reduced-price snack at an afterschool program or daycare.
This question was added immediately after the questions about “free and reduced-price
breakfast/lunch” for 5-18-year-old children, and right before item SP7A which asks about free and
reduced-price food for children 6 years and under. The new item, NEW1, was intended to capture
free or reduced-price food availability for school-age children as part of an afterschool program.
During cognitive testing, no participants had children who had ever received any kind of free or
reduced-price food at an afterschool program. Several participants seemed to mistakenly think this
question was asking about kids age 6 and under who are in daycare.
23

U.S. Census Bureau

Round 1 to 3 Recommendations
CBSM recommended incorporating the word “price” instead of “cost” into SP6 and SP7 because
this wording worked well during cognitive testing and it is consistent with the terminology used
by the program.
CBSM recommended dropping the words “or daycare” from NEW 1 to make it clearer it was
asking about after care for school-age children. The new item read, “During the past 30 days, did
any children in the household (between 5 and 18 years old) receive a free or reduced-price meal
or snack at an afterschool program?”
3.5

Section V Community Food Assistance

ERS requested substantial changes be tested in Section V on Community Food Assistance because
the data from these questions on food assistance program use have not been consistent with that
of other organizations that collect similar data. Given this discrepancy, ERS wanted to test
questions that were similar to those used by other organizations, particularly Feeding America.
During data collection, Section V is only asked of participants who indicate food insecurity in the
preceding questions or meet certain income guidelines. In order to test the new question wording
and how it was being interpreted, Section V was administered to all 30 participants and only age
restrictions were used to determine which questions were asked. The new items were tested in a
split ballot format, in which half of cognitive interview participants received one version during
the interview and then the other version during probing, and the other half of participants received
the opposite order for the versions.
3.5.1

Senior Free Meal Programs (SC1, SC2)

Table 16 Senior Free Meal Programs (SC1, SC2)
Item number

SC1 and SC2

Current Instrument Wording/Round 1-3
Wording
SC1: During the past 30 days, did (you/anyone in
this household) receive any meals delivered to the
home from community programs, “Meals on
wheels,” or any other programs?
SC2: During the past 30 days, did (you/anyone in
this household) go to a community program or
senior center to eat prepared meals?
[Only asked if HHD has someone 60 or older]

Final Recommended Wording
These items will be replaced by the
NEW 3 item that is asked of
everyone who receives Section V,
regardless of age.

Round 1 to 3 Findings
During cognitive testing, Section V began with SC1 and SC2 which ask about receiving free
prepared meals; these questions are only asked if the household has a member 60 or older.
In Rounds 1 to 3, eleven participants had a household member 60 or older and were asked these
two questions. While these items were easily understood during testing, the questions were
replaced by the NEW 3 item below that asks about free meals or food obtained by any household
24

U.S. Census Bureau

member, regardless of age. This was done to reduce the burden and repetitiveness of the survey
while also more accurately capturing the usage of these programs across all age groups.
Recommendations
ERS decided to replace items SC3 and SC4 with the NEW 3 6.5.3. CBSM agreed with this decision
since the new items worked well for participants 60 and over and encompass more people who
may be utilizing these programs.

25

U.S. Census Bureau

3.5.2

Emergency Food vs. Free Groceries (SC3, SC3A)

Table 17 Emergency Food vs. Free Groceries (SC3, SC3A)
Item
Number

SC3

SC3A

Current Instrument
Wording
In the last 12 months,
did (you/you or other
adults in your
household) ever get
emergency food from
a church, a food
pantry, food bank, or
other place that helps
with free food?

Is there a church, food
pantry, food bank or
other place in your
community where you
could get emergency
food if you needed it?

Round 1 and 2 Wording
During the past 12 months,
have you or anyone in your
household gotten free
groceries from a food pantry,
food bank, church, or other
place that helps with free
food?
In the last 12 months, did
(you/you or other adults in
your household) ever get
emergency food from a
church, a food pantry, food
bank, or other place that helps
with free food?
Is there a food pantry, food
bank, church or other place in
your community where you
could get free groceries if you
needed it?
Is there a church, food pantry,
food bank or other place in
your community where you
could get emergency food if
you needed it?

Item
Number

SC3-Free
Groceries

SC3Emergnecy
Food

SC3A-Free
Groceries

SC3AEmergnecy
Food

26

Round 3 Wording
During the past 12
months, did you or other
adults ever get free
groceries from a food
pantry, food bank,
church, or other place
that helps with free
food?
In the last 12 months, did
(you/you or other adults
in your household) ever
get emergency food from
a food pantry, food bank,
church or other place
that helps with free
food?
Is there a food pantry,
food bank, church or
other place in your
community where you
could get free groceries
if you needed it?
Is there a food pantry,
food bank, church or
other place in your
community where you
could get emergency
food if you needed it?”

Final Recommended
Wording
In the last 12 months, have
you or anyone in your
household ever gotten free
groceries from a food
pantry, food bank, church,
or other place that helps
with free food?

N/A

Is there a food pantry, food
bank, church, or other place
in your community where
you could get free groceries
if you needed it?

N/A

U.S. Census Bureau

Round 1 to 2 Findings
All participants in Rounds 1 and 2 were asked both the current instrument wording, SC3 & SC3Aemergency food and the new proposed wordings, SC3 and SC3A-free groceries. The order was
randomized so half of participants received the current instrument wording during the interview
and the new proposed wording during probing and the other half received the opposite order.
The difference most participants noticed between the two versions was the use of the phrase “free
groceries” versus the phrase “emergency food”. Participants were almost evenly split on which
version they preferred. One participant who preferred “free groceries” said it was “more hip, more
regular, less dramatic”. Conversely, another participant said “emergency food” is a phrase used by
people facing food shortages use, as it is a true emergency. One participant said “no” to SC3emergency food but then said “yes” to SC3-free groceries because they specifically said the phrase
“emergency food” was too dramatic.
Another participant in Round 1 spontaneously noticed the difference between the phrases “you or
other adults in your household” versus “you or anyone in your household.” To her these were
different, as the former included homeless people you let sleep on your couch or let use your
address for food stamps. She mentioned that she had let other adults sleep on her couch in the past
and at one point slept on other people’s couches. She preferred a question that combined “you or
other adults in your household” with emergency food.
Round 1 and 2 Recommendations
CBSM recommended using the new proposed wording of “free groceries” for both questions
during Round 3 of cognitive testing and only asking about “emergency food” during probing. “Free
groceries” seemed to be more inclusive and participants were more likely to use it. CBSM also
recommended the order of places in the emergency food version be re-ordered to match the free
groceries version. The verb in the free groceries version was changed to “did” instead of “have”
so the versions were consistent. CBSM also recommended changing the free groceries version to
“you or other adults in your household,” as this was consistent with the emergency food version.
Round 3 Findings
Starting in Round 3, every participant was asked the free groceries version during the interview
and the emergency food version was only asked during probing.
Two participants said “yes” to the free groceries version and then “no” to the emergency food
version. Both said they saw “free groceries” as more of a regular routine thing they did as opposed
to a dire situation where you need food on a sudden and temporary basis. Other participants agreed
that “free groceries” conveys less urgency to get food:
“We don't get emergency food, just food we get every single month to help out if we need it. We
go most months.”

27

U.S. Census Bureau

For this round, probing questions focused less on participant preference and more on their
interpretations of the difference between “free groceries” and “emergency food.” It appears that
these two terms are measuring different constructs, with “free groceries” being interpreted more
broadly and participants being more likely to say that they received free groceries.
Round 3 Recommendations
We recommend the new proposed wording that utilizes “free groceries.” In addition to being
consistent with what other surveys use, this item was easily understood and interpreted to mean
anyone who received “free groceries” as opposed to only people who needed groceries on an
emergency basis. Using terminology that is more inclusive may help bring the reported program
usage in line with other surveys.
ERS requested that SC3 revert to the “you or anyone in your household” language instead of “you
or other adults.” This change was made was for consistency with the question asked by Feed
America, which is trying to capture teens and children who may receive free groceries through
their school to bring home in a backpack or other similar programs.
3.5.3

Free Meal Program Usage (SC4)

Table 18 Free Meal Program Usage (SC4)
Item
Number

Current Instrument
Wording
In the last 12 months, did
(you or other adults in your
household) ever eat any
meals at a soup kitchen or
shelter?

Item
Number

SC4-Free
meals

SC4
SC4Soup
Kitchen

Round 1 to 3 Wording
During the past 12 months,
have you or anyone in your
household received a free
meal from a church, shelter,
home-delivered meal
service like Meals on
Wheels, or other place that
helps with free meals?
In the last 12 months, did
(you or other adults in your
household) ever eat any
meals at a soup kitchen or
shelter?

Final Recommended
Wording
In the last 12 months,
have you or anyone in
your household received
a free meal from a
church, shelter, homedelivered meal service
like Meals on Wheels,
or other place that helps
with free meals?

Round 1 to 3 Findings
The order was randomized for all three rounds so half of participants received the current
instrument wording, SC4- soup kitchen, during the interview and the other half received the new
proposed wording SC4-free meals during probing. The other half of participants heard SC4-free
meals during the interview and then SC4-soup kitchen during probing.
The majority of participants said that SC4A- free meal version was broader, less dire, and they
thought more people would say “yes” to it. One participant said:
“I'm gonna say, like 'soup kitchens' or 'shelters' means really a lot of homeless people usually go
to soup kitchens and shelters. The first [version about free meal] is, say, like a home delivery meal
28

U.S. Census Bureau

service, maybe for people like senior citizens, who can't go out grocery shopping. And a free meal
from a Church is if like they have a free event or concert or a community day, they may have meals
for the community ... Those are definitely two different scenarios."
Multiple participants also said it was good it included more options like “Meals on Wheels.”
Several participants said that the term “soup kitchen,” is not as commonly used anymore but
everyone was familiar with the term. Participants easily interpreted both versions, but SC4 -free
meal version appears to encompass more situations and ways in which people can obtain a free
meal.
Round 1 to 3 Recommendations
CBSM recommends replacing the current SC4- soup kitchen with the SC4-free meal version. In
addition to being easily understood by participants, the wording in SC4-free meal is more inclusive
than SC4- soup kitchen. It implies a “less dire” situation in which someone would obtain a free
meal and lists more options for the types of places a person could get a free meal.
3.5.4

Free Meal Program Community Availability (NEW 4)

Table 19 Free Meal Program Community Availability (NEW 4)
Item
Number

Current Instrument
Wording

NEW4

N/A

Proposed New Wording/ Round 1-3
Wording
Is there a church, shelter, home-delivery
meal service like meals on Wheels, or other
place in your community where you could
get free meals if you needed it?

Final Recommended
Wording
Is there a church, shelter,
home-delivery meal service
like meals on Wheels, or
other place in your
community where you
could get free meals if you
needed it?

Round 1 to 3 Findings
Item NEW 4 was the last item asked in the survey and has no existing counterpart in the current
survey. Participants across all three rounds easily understood this item as intended. Many said they
were unsure if there was a free meal program in their community.
In Rounds 2 and 3, a probe was added to see if participants could see the difference between this
item and SC 3A, which asks if there is a program to provide free groceries in their community.
When probed about the difference, most participants said that a “free meal” is something that is
already prepared whereas “free groceries” is something that they would have to cook or prepare
themselves.
Round 1 to 3 Recommendations
CBSM recommends adding this question to the survey as it was easily understood across three
rounds of testing.

29

U.S. Census Bureau

4. CONCLUSIONS
Overall, results from cognitive testing showed that the CPS FSS appears to be working well for
the majority of participants. The changes made to Section I were easily understood and should
help better reflect current shopping behaviors of participants than the previous question wording.
Switching Sections III and IV was not problematic for the flow of the survey or participants’
understanding of the questions. The new items in Section V encompassed more food assistance
programs than the current survey wording and this was reiterated by participants who agreed the
new items were “less dire” and had more options.
All recommended changes to the FSS survey will be pretested in a split panel test in the CPS in
2020. Pending the results of the split panel test and upon approval, a final revised survey instrument
will be implemented in a future CPS-FSS collection.

5. REFERENCES
USDA Food and Nutrition Service (2020). Fiscal Year 2019 Year End Summary. USDA Food and
Nutrition Service. Retrieved from: https://fns-prod.azureedge.net/sites/default/files/resourcefiles/2019-SNAP-Retailer-Management-Year-End-Summary.pdf.

30

U.S. Census Bureau