Spanish Cognitive Test

MEMORANDUM

955 Massachusetts Avenue, Suite 801
Cambridge, MA 02139
Telephone (617) 491-7900
Fax (617) 491-8044
www.mathematica-mpr.com

TO:

Mark Denbaly, USDA Economic Research Service

FROM:

Nancy Cole

SUBJECT:

FoodAPS: Cognitive Tests of Spanish Training Video

DATE: 5/7/2012

This memorandum provides a summary of the cognitive tests conducted for the Spanish
version of the Household Training video for FoodAPS, which were required by OMB. We
previously conducted and reported on cognitive tests of the English-language training video and
protocols. After the English-language tests, we revised the practice portion of household training
to include a concrete example of a McDonald’s meal for Red page practice, and to include
“model” Red and Blue pages as feedback for respondents on how to complete a page. These
revised protocols were used for Spanish language cognitive tests.
The English language video script was translated to Spanish by Mathematica’s in-house
Spanish translator (in-house translations go through a quality review process). The script was
reviewed by a bilingual interviewer, who made minor revisions to improve the conversational
tone prior to recording the video. The bilingual interviewer worked in the telephone center
during the FoodAPS field test, receiving telephone calls from households who reported their
food acquisitions by phone. She was therefore thoroughly familiar with the content of both the
food books and the training script prior to working on the video.
The Spanish version of the household training
http://www.usdafoodstudy.org/espanol/index-9.html#selected

video

is

available

at:

COGNITIVE TESTS
Mathematica recruited five Spanish speaking respondents for cognitive testing.
Characteristics of respondents are shown in Table 1. We recruited the Spanish speaking
custodian of our office building, two residents of an apartment complex occupied by
predominantly Spanish speakers, and two Spanish-speaking spouses of MIT graduate students.
Cognitive tests were designed to mimic the primary respondent training that will be done in
the field. Testers read from the “Script for Initial Visit” to introduce the video and complete the
Daily List, Red Page, and Blue Page practice sections.
Cognitive tests were conducted by three bilingual staff members in the Mathematica
Cambridge office (two staff members each completed two tests; the third staff member
completed on test). These staff had not worked on FoodAPS previously. Prior to administering
the cognitive tests, the testers reviewed both the English and Spanish versions of the household
An Affirmative Action/Equal Opportunity Employer

training video, and discussed the protocols for leading respondents through practice sessions
with a FoodAPS staff member.
Table 1. Respondents to Cognitive Tests
1
2
3
4
5

Respondent
Gender
Male
Female
Female
Female
Female

Respondent Age

Children in Household

50
27
20
30
28

Yes
No
No
Yes
Yes

Duration of Cognitive
Test
65 minutes
70 minutes
75 minutes
55 minutes
60 minutes

The Spanish training video is 44 minutes in length, compared with 28 minutes for English
(Spanish content tends to be 30 percent longer than English). The cognitive tests lasted from 55
to 75 minutes, and averaged 65 minutes (compared with 63 minutes on average for the English
tests, including one outlier at 83 minutes.) Total time included time for the video and interactive
training.
Interactive training has three parts to practice the Daily List, Red page and Blue page. For
Daily List practice, respondents are asked to write places where they obtained food the day
before the cognitive test (or where they typically get food). For Red page practice, respondents
are shown an example of a McDonald’s meal and receipt. For Blue page practice, respondents
are provided a grocery bag of items and a receipt.
RESULTS OF COGNITIVE TESTS
Cognitive test results were recorded on a standard form, shown in Figure 1. This form has a
section to record observations of the practice sessions, and a list of debriefing questions for
respondents.
Practice Page Observations
Similar to the English tests, the practice page observations showed that respondents had an
increasingly easier time with each practice session: they struggled a little with the Daily List, had
a few questions about the Red page, and an easy time with the Blue page. As noted for the
English tests, this observation may simply reveal “test anxiety” since the Daily List was the first
of the three practice sessions.
For the Daily List practice, two respondents needed coaching to complete the page; one
respondent jumped to the Red page and self-corrected and went back to the Daily List; two
respondents completed the Daily List but asked questions as they completed it.
Four of the five respondents began to complete the practice Red page when prompted by the
video; one respondent completed a Red page while watching the video, although she was not
prompted to do so. Respondents asked for additional explanation of specific sections of the Red
page – for example, what to include in the “total paid” field; what to do if they ran out of space at
the bottom of the page. One respondent had some trouble with two questions on the page but
self-corrected and completed the page correctly without coaching.
2

Figure 1. Cognitive Test Report Form

3

Respondents had no trouble with the Blue page. One respondent had trouble scanning barcodes,
and needed coaching because he tried to scan the barcode with the laser pointed vertically across
the barcode, rather than horizontally. One respondent initially forgot to scan the Place code, and
self-corrected; another respondent was not sure how to end the session and needed coaching to
scan the End barcode. All respondents said that they enjoyed scanning.
Debriefing Questions
All respondents were asked the same twelve debriefing questions which were designed to
obtain feedback on the training session and test their understanding of the survey protocols. The
feedback from the Spanish tests were similar to that of the English tests: respondents found the
video clear and easy to understand.


Questions 1 and 2: Respondents did not have any trouble understanding the person who
talked in the video. Four of five respondents found the pace of the video to be “just
right,” while one found it “a little slow.”



Question 3: Three of five respondents found the information in the video to be clear; one
said that it was clear but the Daily List section was somewhat repetitive; one said that
they were initially confused by the extensive list of locations that were provided as
examples for the Daily List.

All respondents answered the specific questions (questions 4-10) about protocols correctly.
One respondent was unsure about a response to question 11, but understood the suggested
answer that he could look for help on the instructions pages in the food book. Respondents’
understanding of protocols indicates that the video, along with the interviewer-led practice was
effective in teaching the study protocols.
In response to question 12, all respondents were confident that they could remember how to
fill out the food book.
COMPARISON OF RESULTS WITH ENGLISH COGNITIVE TESTS
The English-language cognitive tests revealed that respondents had trouble with the Red
page practice because we did not provide a concrete example of a food acquisition to document
on the Red page. The English tests also suggested the following changes in training protocols:
1. Encourage respondents to follow along with the book during the video because that
seemed to be very helpful for those who did it.
2. Provide concrete feedback for how the practice pages should be completed.
These changes were incorporated into training protocols for the Spanish cognitive tests.
Respondents to the Spanish tests followed along in the book while the video played (as noted
above, one respondent independently completed a Red page along with the video). These
respondents also had fewer questions about how to complete Red pages.
4

SUMMARY
The Spanish cognitive tests indicate that the current Spanish-language version of the training
video is clear and easy to understand. Respondents had questions about specific protocols, but
that is expected and interviewers are trained to answer their questions and coach respondents as
needed. The training protocols, which include interviewer-led practice, were effective in guiding
respondents through practice pages in the food book. The tests do not indicate any need for
revision of current protocols.

5