Report of Cognitive Test for recruitment training video

MEMORANDUM

955 Massachusetts Avenue, Suite 801
Cambridge, MA 02139
Telephone (617) 491-7900
Fax (617) 491-8044
www.mathematica-mpr.com

TO:

Mark Denbaly, USDA Economic Research Service

FROM:

Nancy Cole

DATE: 4/16/2012

Revised: 4/19/2012
SUBJECT:

Cognitive Tests of FoodAPS Household Training Video

This memorandum provides a summary of the cognitive tests conducted for the FoodAPS
Household Training protocols, which were required by OMB. FoodAPS will collect food
acquisition data from households over a 7-day period. For households that are eligible and agree
to participate, the primary respondent is trained to use food books and a hand-held scanner to
track their food acquisitions. During the field test, conducted in Spring 2011, field interviewers
trained primary respondents using a script that they could read verbatim, although field
interviewers were told that they did not need to read verbatim once they were comfortable with
the protocol. The field interviewer explained how to complete pages in the book, and they led the
primary respondent through completion of one practice page in each section: Daily List, Red
page, and Blue page.
For the full-scale study, the IRB was concerned about ensuring consistent, comprehensive
training of primary respondents across interviewers and over time. Mathematica proposed, and
ERS accepted, the use of a training video in conjunction with hands-on practice led by field
interviewers. We prepared the training video to include content on the purpose of the study, the
purpose of the food books, a detailed explanation of each section of the food books, and a
description of completed sample pages.
To incorporate the household training video in the field protocols, we revised the script for
field interviewers. The household training video is available at:1
http://www.usdafoodstudy.org/index-9.html
The revised Initial Visit Script does not include material contained in the video; the script
begins with a description of the consent form, includes instructions to start the training video,

1

The Spanish language video is available at: http://www.usdafoodstudy.org/espanol/index-9.html#selected
The English language script was translated to Spanish by Mathematica’s in-house Spanish translator (in-house
translations go through a quality review process). The script was reviewed by a bilingual interviewer, who made
minor revisions to improve the conversational tone prior to recording the video. The bilingual interviewer worked in
the telephone center during the FoodAPS field test, receiving telephone calls from households who reported their
food acquisitions by phone. She was therefore thoroughly familiar with the content of both the food books and the
training script prior to working on the video.

An Affirmative Action/Equal Opportunity Employer

PAGE:

2

and includes scripted practice with respondents for each section of the food book. Cognitive tests
of this revised protocol were conducted in March 2012.
COGNITIVE TESTS
Mathematica recruited five respondents for cognitive testing. Characteristics of respondents
are shown in Table 1. We recruited persons from a low-income housing complex, a senior center,
and a YMCA. Cognitive tests were designed to mimic the primary respondent training that will
be done in the field. Testers read from the “Script for Initial Visit” (Appendix W) beginning with
the start of the training video, skipping over the Initial Interview section, and completing the
Daily List, Red Page, and Blue Page practice sections. The training video has five sections: (1)
purpose of the study, (2) overview of food books and description of how to use Daily List pages,
(3) description of how to use Red pages, (4) description of Blue pages, (5) description of Adult
and Youth Food Books. The video prompts the interviewer to pause and lead the respondent
through practice pages in their Primary Respondent Food Book.
Table 1. Respondents to Cognitive Tests
1
2
3
4
5

Respondent
Gender
Male
Female
Female
Male
Female

Respondent
Age
30 to 40
24
80
34
52

Children in Household
No, single
No, lives with partner
No
No, lives with 2 other adults
Yes

Duration of Cognitive
Test
80 minutes
55 minutes
65 minutes
60 minutes
55 minutes

Cognitive tests were conducted by two survey staff members in the Mathematica Cambridge
office. One of these staff members conducted cognitive tests prior to the FoodAPS field test, but
had no other role on the study. The second staff member provided periodic support to the study
but does not have an integral or ongoing role.
The training video is 28 minutes in length. The cognitive tests lasted from 55 to 80 minutes,
and averaged 63 minutes, which includes time for the video and interactive training. The first
cognitive test lasted the longest, largely due to language difficulties experienced by a native
Spanish speaker. The language difficulties were not apparent when the respondent was recruited
and we did not have a bilingual tester and Spanish language materials available when the
language difficulty was encountered.
Interactive training has three parts. To practice the Daily List, respondents are asked to write
places where they obtained food the day before the cognitive test. If no food was acquired on that
day, respondents are asked to write places where they typically get food. To practice the Red
page, respondents are asked to use a place that they wrote on the Daily List (where they recently
obtained food away from home), and fill out a Red page. To practice the Blue page, respondents
are provided a grocery bag of items and a receipt; they are asked to scan the items and complete
the Blue page using information on the receipt.

PAGE:

3

Figure 1. Cognitive Test Report Form

PAGE:

4

RESULTS OF COGNITIVE TESTS
Cognitive test results were recorded on a standard form, shown in Figure 1 (the actual form
contained more space for writing responses to each item). This form has a section to record
observations of the practice sessions, and a list of debriefing questions for respondents.
Practice Page Observations
Practice page observations showed that respondents had a better initial understanding of the
detailed Red and Blue pages, compared with Daily List pages, based on the video alone.
However, this may simply reveal “test anxiety” since the Daily List was the first of the three
practice sessions. All respondents asked questions about the Daily List prior to filling it out; two
of the five respondents started to write food items on the Daily List, but understood the mistake
when they received guidance from the tester.
All respondents began to complete the practice Red page when prompted by the video.
Respondents referred back to their Daily List for an example to use on this page; one respondent
initially used a “Section B place” and realized their mistake on their own. Respondents asked for
additional explanation of specific sections of the Red page: who to include in “who ate the meal”
(two respondents used an example that included someone outside the household); how to write
the food items at the bottom of the page (needing guidance to put each item on a separate line).
When the video was paused for Blue page practice, all respondents began to scan without
any prompting from the tester. Two respondents needed help with the scanner but were able to
do it correctly after a demonstration by the tester. One respondent scanned food items without
scanning “Begin”, a second respondent failed to scan the Place code. The Begin and place codes
are delimiters in the scanner file and omission is not critical since the time/date stamp on the
scanner is sufficient to identify separate transactions in nearly all cases. All respondents knew
that they should look up fruit with no barcode in the food book, and all did so without prompting.
One respondent had to be prompted to fill the top of the Blue page (store name and
location), and a second respondent did not check the day of the week even after prompting.
During the field period, this missing information will be caught in the field when interviewers
review Blue pages during the Final Visit at the end of the survey week. Overall, respondents
were most successful with Blue page practice, perhaps because it was the last of three practice
sessions.
One respondent commented on the “duplicate” effort of scanning items that are listed on the
receipt. The respondent scanned all the items without prompting, but with each scan, they
commented that scanning should not be necessary.
Three of the five respondents followed along attentively and said that they would be willing
to participate in the study. One respondent was distracted during the cognitive test – their cell
phone rang and they looked at their watch or out the window. This respondent seemed to
understand the concepts best when told by the “interviewer”, though the respondent said it was
too much work and they would not do the study if asked. The final respondent, 80 year-old
female, followed along in the video but wanted to turn the pages of the book along with the
video and had a hard time keeping up. Nonetheless she had no trouble understanding the Daily

PAGE:

5

List and Blue pages, and only slight trouble with the Red Pages, which the tester thought could
be overcome with more practice.
Debriefing Questions
All respondents were asked the same twelve debriefing questions which were designed to
obtain feedback on the training session, and test their understanding of the survey protocols.
Questions 1 and 2: Four of the five respondents had no trouble understanding the video and
thought the speed was “just right”; the fifth respondent had a slight language barrier and said he
had “a little trouble.”
Question 3: One respondent said “not confusing”, two said “confusing but not after we went
over it”, one said “clear after you explained”; one said “confusing to a point, a lot of information
to take in.”
Excluding the one respondent with a language barrier, all other respondents answered the
specific questions (questions 4-11) about protocols correctly. This indicates that the video, along
with the interviewer-led practice was effective in teaching the study protocols. The respondent
with the language barrier did not understand question 4 and responses to other questions were
expressed in general, rather than specific terms, reflecting the language barrier. For example,
when asked “Why do we ask people to scan barcodes?” this respondent said “to find out about
foods,” while other respondents said “so we don’t have to write it down” and “to get product
information.”
In response to question 12, four respondents were confident that they could remember how
to fill out the food book. The respondent with the language problem said that he would ask a
friend to help.
RECOMMENDATIONS FROM COGNITIVE TESTERS
The cognitive testers had four recommendations:
1. Encourage respondents to follow along with the book during the video because that
seemed to be very helpful for those who did it.
2. Emphasize that the Red page section, “Names of people who ate this meal”, should
include only household members who shared the meal. This section caused the most
confusion.
3. Provide a specific example for the Red page practice, as is done for Blue pages. It seemed
difficult for respondents to understand what to write when we ask respondents to come up
with an example on their own and without a receipt.
4. Provide concrete feedback for how the practice pages should be completed.

PAGE:

6

Changes to Instruments as a Result of Cognitive Tests
The cognitive tests demonstrated that respondents could understand the information
provided in the video, found it clear and well-paced, and most could follow along with the video
in their own food book. We will train field interviewers to pause the video, as needed, so that
older respondents do not feel rushed as they follow along in their own food books.
We did not revise the video as a result of cognitive testing. We made minor revisions to the
Initial Visit Script (Appendix X) to introduce additional instruments for the practice sessions:
1. New handout displaying a McDonald’s “meal for two” with receipt and images of
purchased food items. This serves as an example for the practice Red page.
2. New handout with “model” Red page with receipt (front side) and model Red page
without receipt (back side), corresponding to the McDonalds’s meal handout.
3. New handout with model Blue page corresponding to the practice grocery items and
receipt.
These additional instruments are provided as two pages printed doubled-sided and laminated
(Appendix Y). The handouts provide a concrete example for Red page practice, plus “answer
keys” for Red and Blue page practice. All respondents will be asked to complete the Red page
practice for the McDonald’s meal shown on the handout. The McDonald’s meal provides a
scenario that will be familiar to most respondents, and the “Angus Deluxe meal” is an example
of a menu item for which all components are not listed on the receipt and must be recorded on
the Red page. With this new instrument, all respondents complete the same practice scenarios for
a Red and Blue page. The model Red and Blue pages will be shown to respondents after they
complete the practice pages so that they receive feedback on their practice and understand how
we would like them to complete the pages. These additional instruments provide consistent
scenarios and feedback for all respondents.