Attachment D Cognitive Testing Recommendations

Attachment D
Cognitive Testing Findings and Recommendations

Final Briefing Report Contains Extra Column with Final wording December 3 2019

2

CPS FOOD SECURITY SUPPLEMENT Final Findings and Recommendations
DRB Approval No. CBDRB-FY19-245
Identifying information has been removed where appropriate.
This memo presents the results from three rounds of cognitive testing of the Current Population Survey Food Security Supplement (CPS FSS). From June to October of 2019, thirty
participants were interviewed at four locations including the Census Bureau Headquarters in Suitland, Maryland, a community college in the greater DC area, a library in the DC area, and a
food pantry in a rural area outside of DC. We recruited participants from multi-person households with a household income of less than $50,000. The recruitment was challenging and we
had many last minute cancellations.
Interviewers administered the full CPS FSS and then asked retrospective probes at the end of each section. This document contains only questions that were 1) modified from the original
production FSS instrument, 2) identified as problematic during cognitive testing, and/or 3) had a scripted probe question during at least one round of interviewing.
All 11 participants in Round 3 were currently experiencing food insecurity or indicated they had in the past. Nine participants out of 11 were currently on food stamps or had been in the
past. All 11 participants lived with at least one child.
General Issues
One of the main modifications to the instrument at the beginning of testing was switching sections III and IV; this did not seem to introduce any issues or disrupt the flow of the instrument
across all three rounds.
Adding the words “online” to 2A and 2B in Round 3 did not seem to introduce any issues for participants.
After our expert review, we recommended replacing the term “balanced” with “healthy,” because Google Trends revealed the term “healthy meals” was used more than “balanced meals.”
However, after three rounds of cognitive testing we recommend reverting back to the original wording and not using the word “healthy.” The term “healthy” was confounded with
unintended constructs including but not limited to “organic,” “more expensive name brands,” and “fresh vs. canned produce.” Further, the terms “balanced” and “healthy” do not appear to
be measuring the same construct, which may be problematic for data trends. Almost all participants could interpret what “balanced” meant and the main theme of their interpretation was a
variety of foods in a meal. The handful of participants who struggled to interpret “balanced” did not appear to have a clearer understanding of “healthy.”
We recommend moving forward with the new language in section IV including “Free groceries” and “Meals on Wheels.” Cognitive testing across all three rounds revealed that the new
items encompass more types of free food services that people use than the current items do. While participants mentioned that anyone can obtain “free groceries” or “free meals,” no matter
how much money they have, these questions are only asked of respondents who indicate food insecurity. The danger of “false positives” or people who do not have food insecurity saying
“yes” is minimal. This change will help make the items more consistent with other surveys of food insecurity and may more accurately reflect the services people are utilizing.

Final Briefing Report Contains Extra Column with Final wording December 3 2019

Qx #

Current Production
Version
1A
First, did (you/anyone
(S1A) in your household)
shop for food at a
supermarket or
grocery store LAST
WEEK?

Round 3 Version
First, did (anyone in your
household) shop for food
at a supermarket, grocery
store, Walmart or Target
superstore (for example
Walmart or Target)
LAST WEEK?

3

Findings and Recommendations
Findings: Dropping the word “superstore” did not seem to introduce
any issues.
One participant mentioned Dollar General and was uncertain whether
they should include it or not; they decided to say “No” to item 1A, but
then correctly reported their purchases at Dollar General under 2A and
2B.
Breaking item 2A back out into a separate item worked well in Round 2
and 3.
Recommendation: We recommend the following text move forward
to split panel testing against current wording: First, did (anyone in
your household) shop for food at a supermarket, grocery store,
Walmart or Target LAST WEEK?

Final Wording Recommended by CBSM
First, did [fill] shop for food at a
supermarket, grocery store, Walmart or
Target LAST WEEK?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #

Current Production
Version
2A
[Combined into one item
(S1B) with 1A for Round 1]
Think about other
places where people
buy food, such as meat
markets, produce
stands, bakeries,
warehouse clubs, and
convenience stores. Did
(you/anyone in your
household) buy food
from any stores such as
these LAST WEEK?

4

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

Think about other places
where people buy food
such as dollar stores,
pharmacies, club stores,
farmers markets, or
butcher shops online. Did
(anyone in your
household) buy food from
any stores such as these
LAST WEEK?

Findings: Breaking item 2A back out into a separate item worked
well in Round 2 and 3.

Think about other places where people buy
food such as dollar stores, pharmacies, club
stores, farmers markets, or online. Did [fill]
buy food from any stores such as these LAST
WEEK?

No one spontaneously mentioned online purchases, but they did not
seem confused by the inclusion of the term.
One participant struggled with whether she should include an
independent health food store and decided not to. She reported this
purchase correctly under 4A and 4B.
The removal of “butcher shop” which we initially proposed to replace
“meat markets” did not appear to introduce any issues. However under
4A “Any other places” a participant mentioned shopping at a “meat
market” and reported their purchase there.
Similarly a participant mentioned a “produce stand” and reported their
purchases under 4A. We are concerned about the list becoming too
long and burdensome for respondents if “produce stands” and “meat
markets” are added back in addition to the new places of “dollar
stores, pharmacies, and online.” We only had one participant, from a
rural area, mention “produce stands” or “meat markets” across three
rounds and numerous participants mentioned buying food at dollar
stores or CVS.
Round 3 had a more rural sample and it may be that “produce stands”
and “meat markets” are still more common in rural areas. We still
recommend moving forward with the new item below, but recommend
carefully analyzing the split panel for any change in the amount
reported under 2A. It should be noted that “produce stands” are a
seasonal phenomenon and the production survey is in December when
they may be less common.
Recommendation: We recommend the following text move forward
to split panel testing against current wording: Think about other
places where people buy food such as dollar stores, pharmacies,
club stores, farmers markets, or online. Did (anyone in your
household) buy food from any stores such as these LAST WEEK?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #

Current Production
Version
3A
LAST WEEK, did
(S1C) (you/anyone in your
household) buy food at
a restaurant, fast food
place, cafeteria, or
vending machine?
(Include any children
who may have bought
food at the school
cafeteria)

4A
Did (anyone in your
(S1D) household) buy food
from any other kind of
place LAST WEEK?

5

Round 3 Version

Findings and Recommendations

LAST WEEK, did
(you/anyone in your
household) buy food at a
restaurant, fast food
place, cafeteria, deli,
convenience store, or
vending machine?
[ONLY MENTION IF
KIDS 18 AND UNDER]:
(Include any children
who may have bought
food at the school
cafeteria).

Findings: When asked what a convenience store means, the common
themes in responses were similar across rounds in that participants
thought of it as a food store separate from a grocery store that is easier
to get to and buy food at. A few participants mentioned that the food
sold there is more processed food and snacks. When asked about their
purchases, participants described food that was already ready to eat
and only a few took food home to eat. Across three rounds of testing,
the item seemed to work fairly well. The only confusion was
participants including children who receive free lunch and breakfast in
school. However in 3B they did not increase the amount accordingly
since they did not pay for the food. One participant in particular said
“Yes,” but she reported $0 for 3B because she did not buy any food at
the other places and said she was only thinking of the free lunch and
breakfast her son received. ERS, since the point of the initial items is
just to remind participants of all the places they may have bought
food, is it problematic for data estimates if they say “Yes” and then
report $0? Recommendation: We recommend the following text
move forward to split panel testing against current wording: LAST
WEEK, did (you/anyone in your household) buy food at a
restaurant, fast food place, cafeteria, deli, convenience store, or
vending machine? (Include any children who may have bought
food at the school cafeteria).

LAST WEEK, did [fill] buy food at a
restaurant, fast food place, cafeteria, deli,
convenience store, or vending machine?

Did (anyone in your
household) buy food from
any other kind of place
LAST WEEK?

Findings: In Round 3, four participants said “Yes” to item 4A, but
only two actually ended up reporting a dollar amount and the other
two participants reported all their expenses in 1B-3B. When asked
what kind of places they were thinking of, one participant was
thinking of “meat markets”, a second mentioned “produce stands,”
and the third was thinking of a fall festival. The fourth participant
mentioned they had reported the amount under 1B-3B for a store
purchase. Recommendation: Across all three rounds, it appears “any
other place” is being used as the catch all it is intended to be and we
are not seeing evidence of double reporting. No changes
recommended at this time.

Did [fill] buy food from any other kind of
place LAST WEEK?

Final Wording Recommended by CBSM

(Include any children who may have bought
food at the school cafeteria).

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx
#
Intr
o
(SL
EA
D)

Current Production Version

Now I'm going to ask you
about the ACTUAL amount
you spent on food LAST
WEEK in all the places where
you bought food. Then, since
LAST WEEK may have been
unusual for you, I will ask
about the amount you
USUALLY spend. DOLLAR
RANGES OR AMOUNTS
ARE OKAY – JUST
RECORD WHATEVER
INFO THEY GIVE YOU.
1B How much did (you/anyone in
(S2) your household) ACTUALLY
spend at supermarkets and
grocery stores LAST WEEK
(including any purchases made
with or food
stamp benefits)?
How much did (you/your
household) spend at stores
such as meat markets, produce
stands, bakeries, warehouse
clubs, and convenience stores
LAST WEEK (including any
purchases made with
or food
stamp benefits)?

6

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

Now I'm going to ask you
about the TOTAL amount you
spent on food LAST WEEK in
all the places where you
bought food. Then, since
LAST WEEK may have been
unusual for you, I will ask
about the amount you
USUALLY spend.

Findings: We did not observe any issues with modifying
the word “actual” to “total” across three rounds of testing.

Now I'm going to ask you about the TOTAL
amount you spent on food LAST WEEK in all
the places where you bought food. Then, since
LAST WEEK may have been unusual for you,
I will ask about the amount you USUALLY
spend.

How much did (anyone in your
household) spend in total at
supermarkets, grocery stores,
or superstores (like Walmart
or Target LAST WEEK
(including any purchases made
with (SNAP/ Food Supplement
Program) or food stamp
benefits)?

Findings: Participants had no issue answering this item
and those that received food stamps included it in the
amount reported.

Recommendations: We recommend the following text
move forward to split panel testing against current
wording: Now I'm going to ask you about the TOTAL
amount you spent on food LAST WEEK in all the
places where you bought food. Then, since LAST
WEEK may have been unusual for you, I will ask
about the amount you USUALLY spend.

Recommendations: We recommend the following text
move forward to split panel testing against current
wording: How much did (anyone in your household)
spend in total at supermarkets, grocery stores,
Walmart or Target LAST WEEK (including any
purchases made with (SNAP/ Food Supplement
Program) or food stamp benefits)?

How much did [fill] spend in total at
supermarkets, grocery stores, Walmart or
Target LAST WEEK (including any purchases
made with (SNAP/ Food Supplement Program)
or food stamp benefits)?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx
#
1C
(S3
A)

Current Production Version
How much of the (1B
ANSWER) was for non-food
items, such as pet food, paper
products, alcohol, detergents,
or cleaning supplies?

2B [Added back in Round 2]
(S4)

2C
(S5
A)

Round 3 Version

How much of the (1B
ANSWER) was for non-food
items, such as pet food, paper
products, alcohol, detergents,
or cleaning supplies?

How much did your household
spend at stores such as dollar
stores, pharmacies, club stores,
farmers markets, or online
LAST WEEK (including any
purchases made with (SNAP/
Food Supplement Program) or
food stamp benefits)?
How much of the (2B
ANSWER) was for non-food
items, such as pet food, paper
products, alcohol, detergents,
or cleaning supplies?

7
Findings and Recommendations

Findings: As in Round 2, one participant in Round 3
reported what percentage of their food amount was for
non-food items.

Final Wording Recommended by CBSM
How much of the [fill] was for non-food items,
such as pet food, paper products, alcohol,
detergents, or cleaning supplies?

When asked what items are non-food items, participants
across all three rounds listed valid non-food items, such as
hygiene products, toothpaste, and clothes, in addition to
the examples provided. They were very clear what nonfood items means and that the question wants them to
separate that amount out of their food purchases. We do
not recommend adding additional examples to the list.
Recommendations: No changes recommended.
Findings: Participants had no issues answering this item.
Recommendations: We recommend the following text
move forward to split panel testing against current
wording: How much did your household spend at
stores such as dollar stores, pharmacies, club stores,
farmers markets, or online LAST WEEK (including
any purchases made with (SNAP/ Food Supplement
Program) or food stamp benefits)?
Findings: Participants had no issues answering this item.
Recommendations: No changes recommended.

How much did [fill] spend at stores such as
dollar stores, pharmacies, club stores, farmers
markets, or online LAST WEEK (including
any purchases made with (SNAP/ Food
Supplement Program) or food stamp benefits)?

How much of the [fill] was for non-food items,
such as pet food, paper products, alcohol,
detergents, or cleaning supplies?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx
Current Production Version
#
3B How much did (you/your
(S6) household) spend for food at
restaurants, fast food places,
cafeterias, and vending
machines LAST WEEK, not
including alcohol purchases?

1D
(S8
A)

Let's see, (your household)
spent about (SUM OF
1B+2B+3B-1C) on food LAST
WEEK. Now think about how
much (anyone in your
household) USUALLY spends.
How much (does your
household) USUALLY spend
on food at all the different
places we've been talking
about IN A WEEK? (Please
include any purchases made
with (SNAP/ Food Supplement
Program) or food stamp
benefits). Do not include nonfood items such as pet food,

Round 3 Version
How much did (your
household) spend for food at
restaurants, fast food places,
cafeterias, delis, convenience
stores, and vending machines
LAST WEEK, not including
alcohol purchases (including
any purchases made with
(SNAP/ Food Supplement
Program) or food stamp
benefits)?

Let's see, (your household)
spent about (SUM OF
1B+2B+3B-1C) on food LAST
WEEK. Now think about how
much (anyone in your
household) USUALLY spends.
How much (does your
household) USUALLY spend
on food at all the different
places we've been talking
about IN A WEEK? (Please
include any purchases made
with (SNAP/ Food Supplement
Program) or food stamp
benefits). Do not include nonfood items such as pet food,

8
Findings and Recommendations

Final Wording Recommended by CBSM

Findings: Participants with kids said they included the
kids in their answer, but most of them received free and
reduced lunch so it did not seem to change the amount
reported. One participant mentioned she was including
money she gives her kids for vending machines.
ERS, since the point of the initial items is just to remind
participants of all the places they may have bought food,
is it problematic for data estimates if they say “Yes” and
then report $0?Recommendation: We recommend the
following text move forward to split panel testing against
current wording: How much did (your household) spend
for food at restaurants, fast food places, cafeterias,
delis, convenience stores, and vending machines LAST
WEEK, not including alcohol purchases (including
any purchases made with (SNAP/ Food Supplement
Program) or food stamp benefits)?
Findings: Participants interpreted “usually” to mean
“normally,” “consistently,” or “on average” across all
three rounds. Participants’ spontaneous comments made it
apparent they were thinking of their usual purchase
behavior the last few weeks/months when answering this
question.
During probing we asked if participants included any food
stamp money in their totals for this section and they all
said “yes.”Recommendation: No changes recommended.

How much did [fill] spend for food at
restaurants, fast food places, cafeterias, delis,
convenience stores, and vending machines
LAST WEEK, not including alcohol purchases
(including any purchases made with (SNAP/
Food Supplement Program) or food stamp
benefits)?

Let's see, [fill] spent about (SUM OF
1B+2B+3B-1C) on food LAST WEEK. Now
think about how much [fill] USUALLY spends.
How much [fill] USUALLY spend on food at
all the different places we've been talking
about IN A WEEK? (Please include any
purchases made with (SNAP/ Food
Supplement Program) or food stamp benefits).
Do not include non-food items such as pet food,
paper products, detergent or cleaning supplies.

Final Briefing Report Contains Extra Column with Final wording December 3 2019
paper products, detergent or
cleaning supplies.

Qx# Current Production Version
1E
Let's see, it seems that (your
(S8A) household) did not buy any
food LAST WEEK. If that
is because you shop for food
infrequently, how much
would the weekly average
be over several weeks?

9

paper products, detergent or
cleaning supplies.

Round 3 Version
Let's see, it seems that (your
household) did not buy any
food LAST WEEK. If that is
because you shop for food
only once in a while, how
much would you say you
spend in total in a month?

Findings and Recommendations
Findings: Every participant indicated that they had bought
food in the last week. However, we administered some probes
to get feedback from participants on this question. When
asked how they would calculate their monthly food expenses,
they mentioned a variety of tactics, but most mentioned
multiplying their weekly amount by 4. Some participants had
unusual response strategies. For example, one participant
thought about one time she had bought all of her food for the
month at the same time (this seemed to be unusual for her)
and would use that number. Another participant said it was
too difficult to figure out beyond a week.
Recommendation: All participants across three rounds of
testing were skipped past this question since they had bought
food in the last week. When asked probing questions, most
participants could come up with a monthly amount but the
response strategies they used varied and may introduce error.
It is still important to offer a way to report monthly food
expenditures for the population that does not purchase food
weekly. We also agree with ERS’s concerns about the current
wording placing too much burden on respondents to calculate
an average, when we can perform this calculation for them.
For these reasons, we still recommend the following text
move forward to split panel testing against current wording:
Let's see, it seems that (your household) did not buy any
food LAST WEEK. If that is because you shop for food
only once in a while, how much would you say you spend
in total in a month?

Final Wording Recommended by CBSM
Let's see, it seems that [fill] did not buy any
food LAST WEEK. If that is because you shop
for food only once in a while, how much would
you say you spend in total in a month?

Final Briefing Report Contains Extra Column with Final wording December 3 2019

Qx # Current Production Version
5A
In order to buy just enough
(S8B) food to meet (the needs of
your household), would you
need to spend more than
you do now, or could you
spend less?

Round 3 Version
In order to buy just enough
food to meet (the needs of
your household), would you
need to spend more than you
do now, or could you spend
less?

Findings and Recommendations
Findings: Participants in Round 3 appeared to struggle to
paraphrase the question more than in previous rounds.
However, most were able to ultimately correctly paraphrase
the item. Several participants mentioned they could spend less
on food by preparing more at home and eating out less. One
participant volunteered they don’t need to spend more or less.
One participant was confused by the question and seemed to
think it was asking if she needed to spend more of her own
money in addition to food stamps to get enough food for her
family. She said that she usually spends more out of her own
pocket.
ERS brought up a concern that participants are including their
food stamp money in their answer. While we did not directly
ask them if including food stamps in their answer for 5A,
several participants indicated, through spontaneous
comments, that they were including their food stamp
allocation when they answered 5A. (We did ask about
including food stamp money for Section I and participants
indicated they included it in the amounts reported).
Despite the one participant misunderstanding, we do not
recommend modifications to this question. All other
participants were able to correctly interpret and answer the
item.
Recommendation: No changes recommended at this time.

10

Final Wording Recommended by CBSM
In order to buy just enough food to meet (the
needs of your household), would you need to
spend more than you do now, or could you
spend less?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
LEA
D
(SS1_
LEA
D)

7
(SS1)

Current Production Version
The next questions are
about the food eaten in your
household in the last 12
months, since December of
last year, and whether you
were able to afford the food
you need.
V1:Which of these
statements best describes
the food eaten in your
household—a) enough of the
kinds of food (we) want to
eat, b) enough but not
always the kinds of food
(we) want to eat, c)
sometimes not enough to
eat, or d) often not enough
to eat?
V2: Which of these
statements best describes
the food eaten in your
household—a) (We) could
always afford to eat healthy
meals. b) (We) could always
afford enough to eat but not
always healthy meals. c)
Sometimes (we) could not
afford enough to eat.
d) Often we could not afford
enough to eat.

Round 3 Version
The next questions are about
the food eaten in your
household in the last 12
months, since MONTH OF
INTERVIEW of last year,
and whether you were able to
afford the food you need.
V1:Which of these
statements best describes the
food eaten in your
household—a) enough of the
kinds of food (we) want to
eat, b) enough but not always
the kinds of food (we) want
to eat, c) sometimes not
enough to eat, or d) often not
enough to eat?
V2: Which of these
statements best describes the
food eaten in your
household—a) (We) could
always afford to eat healthy
meals. b) (We) could always
afford enough to eat but not
always healthy meals. c)
Sometimes (we) could not
afford enough to eat.
d) Often we could not afford
enough to eat.

11
Findings and Recommendations

Findings: At the sponsor’s request, we probed participants on what
“afford the food you need” means across all three rounds. Almost
every participant mentioned a variation on the theme that it is the
ability to afford the food you need to survive, there was a little bit
of variation if this meant to be healthy, or simply to not go hungry.
Recommendation: No changes recommended at this time.
Findings: As mentioned in previous rounds, V1 and V2 are not
measuring the same constructs. Multiple participants selected
different answers for each version. In Round 3, we still randomized
which version participants received in order to gather more
information about how they are interpreting the items. Most
participants in this round preferred “kinds of food.” (V1:Which of
these statements best describes the food eaten in your household—
a) enough of the kinds of food (we) want to eat, b) enough but not
always the kinds of food (we) want to eat, c) sometimes not enough
to eat, or d) often not enough to eat?)As in previous rounds, the
main theme of participants’ comments was that “kinds of food”
might mean healthy or it might just mean something expensive and
unhealthy because that is what a person wants to eat. Several
participants mentioned that the two versions are not asking the
same thing: "I think depending on exactly what you're looking for,
they are asking two different things. One is specifically asking
whether the person can or can't afford to eat healthy food. The
other one is asking what kinds of food that we want to eat. The
[kinds of food] is more so related to the person and their
preference, the [healthy one] is more so related to 'Can you afford
healthy food consistently… Like in general, are you able to get
enough food, then I would go with the [Kinds of food].'"While we
hoped to improve upon the current version, the alternative is
measuring a very different construct and the original wording was
not difficult for participants to interpret, as we feared during the
expert review.
Recommendation: No changes recommended at this time, we
recommend keeping the current production wording.

Final Wording Recommended by CBSM
The next questions are about the food eaten
in your household in the last 12 months,
since MONTH OF INTERVIEW of last year,
and whether you were able to afford the
food you need.

Which of these statements best describes the
food eaten in your household—a) enough of
the kinds of food (we) want to eat, b) enough
but not always the kinds of food (we) want to
eat, c) sometimes not enough to eat, or d)
often not enough to eat?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
10A
(SS4)

Current Production
Version
"(I/we) couldn't afford to
eat balanced meals." Was
that OFTEN,
SOMETIMES or NEVER
true for (you/ your
household) in the last 12
months?

12

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

"(We) couldn't afford to eat
healthy meals." Was that
OFTEN, SOMETIMES or
NEVER true for (your
household) in the last 12
months?

Findings: One participant brought up the fact that the “fresh
produce” near her was often old and she was afraid it would
make her kids sick, so she would buy “less healthy” packaged
food to make sure it was safe. Another participant brought up
organic and brand name being healthier, as we heard in Round
1.
There was a consistent theme among participants that healthy
food is more expensive. One participant brought up processed
food being less healthy and gave the example of buying lunch
meat instead of fresh chicken and vegetables.
When asked if “balanced” meant the same thing as “healthy,”
participants generally said yes, but then seemed to contradict
this by giving examples of meals that are “balanced” but not
“healthy”. For instance, they mentioned a meal that may have
a protein and vegetables, but be fried; they said this would be
balanced, but not healthy. This occurred in Round 2 as well.
The themes that come up for “balanced” include the pyramid
or chart, and having a protein and carbohydrate/starch.
Generally the term conveys an idea of a variety of food types.
One participant said they did not know what “balanced” meant.
They also did not seem to know what “healthy” meant but they
thought it was more expensive than not healthy food.

"[Fill] couldn't afford to eat balanced
meals." Was that OFTEN, SOMETIMES or
NEVER true for (you/ your household) in
the last 12 months?

Recommendation: Given the confusion some participants had
thinking that organic and brand name equates to healthier food,
we don’t recommend implementing the change from
“balanced” at this time. The divergence between the meaning
of “balanced” vs “healthy” could lead to the item measuring a
different construct. We initially recommended using the words
“healthy meal” because the term is more commonly used than
“balanced meals,” according to Google trends. All but one
participant seemed to have an idea what “balanced” meant. We
recommend no changes at this time and keeping the original
wording.

Final Briefing Report Contains Extra Column with Final wording December 3 2019

Qx #
11A(
SH2)

17A
(SS5)

Current Production
Version
In the last 12 months, did
(you/ you or other adults in
your household) ever cut
the size of your meals or
skip meals because there
wasn't enough money for
food?
"(I/we) relied on only a few
kinds of low-cost food to
feed (the child in (my/our)
household/the children)
because (I was/we were)
running out of money to
buy food. Was that
OFTEN, SOMETIMES or
NEVER true for (you/ your
household) in the last 12
months?

13

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

In the last 12 months, did
(you/ you or other adults in
your household) ever cut the
size of your meals or skip
meals because there wasn’t
enough money for money
for enough food?
"(I/we) relied on only a few
kinds of low-cost food to
feed (the child in (my/our)
household/the children)
because (I was/we were)
running out of money to
buy food. Was that
OFTEN, SOMETIMES or
NEVER true for (you/ your
household) in the last 12
months?

Findings: We did not probe on this item in Round 3. No issues came
up with reverting to the original wording “there wasn’t enough
money for food.”
Recommendation: Reverting back to the original wording of
“enough money for food” seems to be working well. No changes
recommended at this time.

In the last 12 months, did (you/ you or other
adults in your household) ever cut the size of
your meals or skip meals because there
wasn't enough money for food?

Findings: All 11 participants in Round 3 had at least one child in the
household. Six participants reported it was “sometimes true,” one
reported it was “often true,” and the other four reported it was “never
true.” When asked what kinds of food they relied on, participants
who said “sometimes” or “often” true mentioned examples like
noodles, hot dogs, beans, canned food such as spaghetti o’s, frozen
processed food (generic brand) such as French fries, fish sticks,
chicken nuggets, mini frozen pizzas, rice, tomatoes, peanut butter
and jelly, and macaroni and cheese. Participants also brought up
tactics like buying food in bulk, shopping at Costco and Aldi, buying
whatever was on sale that week, and buying pre-packaged lunch
meat instead of meat from the deli.
A major theme among participants was that processed food is
cheaper than fresh food. One participant spontaneously lamented
they felt bad they could not afford organic food for their children as
they seemed to think it was healthier.
Among the participants who said it was “never true”, one participant
mentioned they had heard of people using things like tuna fish,
oatmeal, and cat food, and having to shop at the dollar store.
Participants who reported it was “never true” mentioned the same
kinds of low cost food and tactics of buying in bulk, on sale, etc.
Across both groups, there was a general theme that low cost food is
less healthy, but it is what you have to eat to feel full.

"(I/we) relied on only a few kinds of low-cost
food to feed (the child in (my/our)
household/the children) because (I was/we
were) running out of money to buy food.
Was that OFTEN, SOMETIMES or NEVER
true for (you/ your household) in the last 12
months?

Recommendation: No changes recommended.

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
18A
(SS6)

24A
(SP1)

25
(SP6)

Current Production
Version
"(I/we) couldn't feed (the
child in (my/our)
household/the children) a
balanced meal, because
(I/we) couldn't afford that."
Was that OFTEN,
SOMETIMES or NEVER
true for (you/your
household) in the last 12
months?

In the past 12 months, since
MONTH OF INTERVIEW
of last year, did
(you/anyone in this
household) get (SNAP/
Food Supplement Program)
or food stamp benefits?
During the past 30 days, did
any children in the
household (between 5 and
18 years old) receive free or
reduced cost lunches at
school?

14

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

"(I/we) couldn't feed (the
child in (my/our)
household/the children) a
healthy meal, because (I/we)
couldn't afford that." Was
that OFTEN, SOMETIMES
or NEVER true for
(you/your household) in the
last 12 months?

Findings: Unlike the previous round, not all participants felt
that healthy was the same for adults and children. Four
participants said it was different. Generally they said kids
needed more nutrients because they are growing. Another
difference mentioned was in the portion sizes and the types of
food adults eat versus kids.
Two participants spontaneously mentioned that it was more
important their children eat healthy than that they eat healthy
as adults. Recommendation: We recommend reverting to the
original survey wording. “Healthy” seems to be confounded
with organic, more expensive, etc. Our initial fear that people
are no longer familiar with the term “balanced” does not
appear to be an issue. We recommend keeping the current
production wording: "(I/we) couldn't feed (the child in
(my/our) household/the children) a balanced meal,
because (I/we) couldn't afford that." Was that OFTEN,
SOMETIMES or NEVER true for (you/your household)
in the last 12 months?
Findings: Five participants reported they are currently on
food stamps. Four participants said they have been on food
stamps in the past. Two participants had never been on food
stamps.

"(I/we) couldn't feed (the child in (my/our)
household/the children) a balanced meal,
because (I/we) couldn't afford that." Was
that OFTEN, SOMETIMES or NEVER
true for (you/your household) in the last 12
months?

In the past 12 months, since
MONTH OF INTERVIEW of
last year, did (you/anyone in
this household) get (SNAP/
Food Supplement Program)
or food stamp benefits?
During the past 30 days, did
any children in the
household (between 5 and
18 years old) receive free or
reduced-price lunches at
school?

Recommendation: No changes recommended.
Findings: Eight of the 11 participants with kids were
currently receiving free or reduced lunch. They had no issues
answering the reduced-price version. They all seemed very
familiar with the program even if they were not currently
receiving it.Recommendation: We recommend the following
text move forward to split panel testing against current
wording: During the past 30 days, did any children in the
household (between 5 and 18 years old) receive free or
reduced-price lunches at school?

In the past 12 months, since MONTH OF
INTERVIEW of last year, did (you/anyone
in this household) get (SNAP/ Food
Supplement Program) or food stamp
benefits?

During the past 30 days, did any children in
the household (between 5 and 18 years old)
receive free or reduced-price lunches at
school?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
26
(SP7)

27
(New)

Current Production
Version
During the past 30 days, did
any children in the
household (between 5 and
18 years old) receive free or
reduced cost breakfasts at
school?
New question.

Round 3 Version
During the past 30 days, did
any children in the household
(between 5 and 18 years old)
receive free or reduced-price
breakfasts at school?

During the past 30 days,
did any children in the
household (between 5 and
18 years old) receive a free
or reduced-price meal or
snack at an afterschool
program or day care?

15
Findings and Recommendations

Final Wording Recommended by CBSM

Findings: Seven of the eight participants whose children receive
free lunch said “Yes” to free breakfast. The eighth participant was
not sure if her child got free breakfast or not since they did not eat
breakfast by their own personal choice.
Recommendation: We recommend the following text move
forward to split panel testing against current wording: During the
past 30 days, did any children in the household (between 5 and
18 years old) receive free or reduced-price breakfasts at school?
Findings: Only one participant across all three rounds said “Yes”
and they incorrectly thought the question was asking about Head
Start or some kind of preschool. They then qualified their answer
and said their child doesn’t go to Head Start, but they go there for
the parenting classes and because there is a snack for the child.
Another parent also thought the question was asking about daycare
for young children, but she said “No” since she pays for the daycare
and snacks are included in that price.
In Round 2, one participant also mistakenly thought the question
was asking about day care or Head Start for young kids; she
correctly said “No”.
In Round 1, we did not probe and all participants said “No.”
Despite qualifying that the question is asking about children
between 5 and 18 years old, parents of young children as still
assuming it is asking about daycare or Head Start. It may be that
the word “day care” being the last thing they hear is confusing
them, especially since this question is before the survey asks about
snacks at Head Start and daycare.
Recommendation: If ERS wants to continue using the new item,
consider dropping the words “day care” after “After school
program”, to help further differentiate that the question is asking
about school age children. We recommend the following text move
forward to split panel testing against current wording: During the
past 30 days, did any children in the household (between 5 and
18 years old) receive a free or reduced-price meal or snack at
an afterschool program?

During the past 30 days, did any children in the
household (between 5 and 18 years old) receive
free or reduced-price breakfasts at school?

During the past 30 days, did any children in the
household (between 5 and 18 years old) receive
a free or reduced-price meal or snack at an
afterschool program?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #

Current Production
Version
28
During the past 30 days,
(SP7A) did any children in the
household receive free or
reduced-cost food at a
day-care or Head Start
program?

16

Round 3 Version

Findings and Recommendations

Final Wording Recommended by CBSM

During the past 30 days, did
any children in the household
(under 6 years old) receive free
or reduced-price food at a daycare or Head Start program?

Findings: The participant who said “Yes” to 27 also said
“Yes” to 28. She was thinking of a snack they give her
daughter while she takes a parenting class. The other
participants who had children under 6 said they either
provide the snacks themselves or it’s included in the tuition
they pay for daycare.
No issue came up with using the word “price” instead of
“cost.” However no one had kids under 6 in Rounds 1 and
2.

During the past 30 days, did any children in
the household (under 6 years old) receive
free or reduced-price food at a day-care or
Head Start program?

Recommendation: We recommend the following text
move forward to split panel testing against current wording:
During the past 30 days, did any children in the
household (under 6 years old) receive free or reducedprice food at a day-care or Head Start program?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
32A
(SC3)

Current Production
Version
V1: In the last 12 months,
did (you/you or other
adults in your household)
ever get emergency food
from a church, a food
pantry, food bank, or
other place that helps
with free food?
V2: During the past 12
months, have you or
anyone in your household
gotten free groceries from
a food pantry, food bank,
church, or other place
that helps with free food?

Round 3 Version

17
Findings and Recommendations

Final Wording Recommended by CBSM

V1: In the last 12 months, did
you or other adults in your
household ever get emergency
food from a food pantry, food
bank, church, or other place
that helps with free food?

Findings: Starting in Round 3, every participant was asked the
“free groceries” version during the interview and the “emergency
food” version was only asked during probing. The order of
options in the “emergency food” version was re-ordered to match
the “free groceries” version. The verb in the “free groceries”
version was changed to “did” instead of “have” so the versions
were consistent.

In the last 12 months, did (you/you or other
adults in your household) ever get free
groceries from a food pantry, food bank,
church, or other place that helps with free
food?

V2: In the last 12 months, did
(you/you or other adults in your
household) ever get free
groceries from a food pantry,
food bank, church, or other
place that helps with free food?

Two participants said “yes” to the “free groceries” version and
then “no” to the “emergency food” version. Both said they saw
“free groceries” as more of a regular routine thing they did, as
opposed to a dire situation where you need food on a sudden and
temporary basis.
Other participants agreed with the theme that “free groceries”
conveys less urgency to get food:
“We don't get emergency food, just food we get every single
month to help out if we need it. We go most months.”
For this round, probing questions focused less on participant
preference and more on their interpretations of the difference
between “free groceries” and “emergency food.” It appears that
these two terms are measuring different constructs, with “free
groceries” being interpreted more broadly and participants being
more likely to say that they received free groceries.
ERS has expressed a desire to bring the question more in line
with other surveys that ask a similar question, even if it increases
the number of affirmative responses.
Recommendation: We recommend the following text move
forward to split panel testing against current wording: In the last
12 months, did (you/you or other adults in your household)
ever get free groceries from a food pantry, food bank, church,
or other place that helps with free food?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #

Current Production
Version
33
V1- Is there a church,
(SC3A) food pantry, food bank or
other place in your
community where you
could get emergency food
if you needed it?
V2- Is there a food
pantry, food bank,
church or other place in
your community where
you could get free
groceries if you needed it?

18

Round 3 Version

Findings and Recommendations

V1- Is there a food pantry, food
bank, church or other place in
your community where you
could get emergency food if you
needed it?

Findings: As with question 32 in Round 3, all participants
were asked this question with the “free groceries” version
first and then the “emergency food” version during probing.
The order of options in the “emergency food” version was
re-ordered to match the “free groceries” version.

V2- Is there a food pantry, food
bank, church or other place in
your community where you
could get free groceries if you
needed it?

There was no issue with this item other than a few
participants mentioning they were not sure there was such a
place in their community. This included at least one
participant who was taking the survey at a food pantry and
said they were “not sure.” This also happened in Rounds 1
and 2.
Participants reiterated that the “free groceries” was less of a
dire thing than the “emergency food” version.
One participant thought “free groceries” was a better
wording because it was a prideful thing and that some
people would not want to look like someone who could not
afford groceries for their home. They may be more likely to
say “no” even though they did go to a food pantry if it said
“emergency food.” Recommendation: We recommend the
following text move forward to split panel testing against
current wording: Is there a food pantry, food bank,
church or other place in your community where you
could get free groceries if you needed it? We recommend
the wording to be consistent with question 32.

Final Wording Recommended by CBSM
Is there a food pantry, food bank, church
or other place in your community where
you could get free groceries if you needed
it?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Qx #
34A
(SC4)

Current Production Version
V1- In the last 12 months, did
(you or other adults in your
household) ever eat any meals
at a soup kitchen or shelter?
V2- During the past 12
months, have you or anyone in
your household received a free
meal from a church, shelter,
home-delivered meal service
like Meals on Wheels, or other
place that helps with free
meals?

35
(new)

Is there a church, shelter,
home-delivery meal service
like meals on Wheels, or other
place in your community
where you could get free
meals if you needed it?

Round 3 Version
V1- In the last 12 months,
did (you or other adults in
your household) ever eat
any meals at a soup kitchen
or shelter?
V2- During the past 12
months, have you or
anyone in your household
received a free meal from a
church, shelter, homedelivered meal service like
Meals on Wheels, or other
place that helps with free
meals?

Is there a church, shelter,
home-delivery meal service
like Meals on Wheels, or
other place in your
community where you
could get free meals if you
needed it?

Findings and Recommendations
Findings: The two versions were still randomized in Round
3. As in Round 1 and 2, all participants said “No” to both
versions. Nine participants preferred the “free meal”
version, one preferred the “soup kitchen” version, and one
saw no difference. Ten participants said that the “free meal”
version was broader and more people would say “yes” to it.
They also said it was good it had more options like “Meals
on Wheels.” Several participants said that “soup kitchen” is
not as commonly used anymore, but everyone was familiar
with the term. Participants easily interpreted both versions,
but the “free meal” version appears to encompass more
situations and ways people can obtain a free meal. If ERS
wants to capture more types of free meals that people are
obtaining, we recommend that version.
Recommendation: We recommend the following text
move forward to split panel testing against current wording:
During the past 12 months, have you or anyone in your
household received a free meal from a church, shelter,
home-delivered meal service like Meals on Wheels, or
other place that helps with free meals?
Findings: Seven participants reported they were “not sure”
but all 11 participants easily understood what the question
was asking. When probed about the difference between this
question and question 33, they were able to say that a “free
meal” is something that is already prepared, whereas
question 33 is about “free groceries” that they would have
to cook or prepare.
Recommendation: We recommend the following text
move forward to split panel testing against current wording:
Is there a church, shelter, home-delivery meal service
like Meals on Wheels, or other place in your community
where you could get free meals if you needed it?

19
Final Wording Recommended by CBSM
In the last 12 months, have you or anyone
in your household received a free meal
from a church, shelter, home-delivered
meal service like Meals on Wheels, or
other place that helps with free meals?

Is there a church, shelter, home-delivery
meal service like Meals on Wheels, or
other place in your community where you
could get free meals if you needed it?

Final Briefing Report Contains Extra Column with Final wording December 3 2019
Debriefing Question
1. [IF THEY SAID THEY ARE NOT CURRENTLY ON FOOD STAMPS] Have you ever
been on food stamps? Have you ever struggled to buy enough food for you or your
household? [IF YES] What did you do to buy food or cope?

20
Round 3 Findings
Findings: Tactics used to buy enough food or cope included buying nothing else
but basic food in bulk. This included no snacks or “goodies,” and buying only
needs, no wants. Participants also mentioned cutting down on food intake, working
extra side jobs, using food pantries, asking family for help, and praying. One
participant said the following about how they coped:
"Spread apart the food we currently have, and don't eat as much for lunch, save it
for dinner. Or go to cheaper grocery stores, use coupons, wait for deals, fast."
Recommendations: No changes recommended.

The questions I asked you today were about you and other members of your household.
2. Who do you consider to be a part of your household? [SHOW THEM ROSTER] Do
you consider everyone on this list to be part of your household why or why not?

Findings: Most participants only lived with immediate family so they considered
them to be part of the household. One participant mentioned that everyone they
live with and share food with is part of their household. All participants considered
everyone on the roster to be part of their household. Across all three rounds
participants generally did not include purchases unrelated household members
made, because the person bought food independently of the household.
Recommendations: No changes recommended.

3. Would you consider a relative staying temporarily to be part of your household? What
about a roommate?

Findings: Participants were split on whether they would consider someone staying
temporarily or a long-term roommate to be part of the household. This was true
across all three rounds.
Recommendations: No changes recommended.

4. Do you think other members of your household would answer the questions the same
way you did? [IF NO]: Tell me more about that.

Findings: Participants generally said other members of the household would
answer differently. The most common reason was because the participant was the
one who did the grocery shopping. Only one participant mentioned they were not
the person who bought groceries and someone in their household would know
more.
Recommendations: No changes recommended.