Generic Information Collection Request
Usability and Cognitive Evaluation for
the Data Collection Instrument of FoodAPS-2 Pilot Study
The U.S. Census Bureau plans to conduct a usability and cognitive evaluation of the data collection instrument for the Second National Food Acquisition and Purchase Survey (FoodAPS-2) Pilot Test, under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). In this request, we are seeking clearance for the aforementioned evaluation study that will be conducted by the Census Bureau staff.
INTRODUCTION
FoodAPS is a nationally representative survey designed to collect information on the food purchase and acquisition patterns of U.S. households, sponsored by the U.S. Department of Agriculture (USDA). The U.S. Census Bureau is conducting a pilot test of FoodAPS-2 on behalf of USDA. As part of the pilot test, the Census Bureau will develop a web-based data collection instrument running on the Qualtrics platform and conduct a usability and cognitive evaluation of the instrument with the intention to minimize respondents’ burden and maximize data quality.
Purpose: The purpose of this study is to evaluate respondents’ effectiveness of, efficiency in, and satisfaction with using the Qualtrics-based online instrument to enroll in the FoodAPS-2 Pilot Test survey, report food acquisition, and complete the post-data-reporting debriefing.
Population of Interest: The population of interest for the proposed study is households in U.S.
Timeline: The study will be conducted from February through July in 2024.
Language: The study will be conducted in English.
The present study consists of three rounds: (1) Initial activities evaluation; (2) Food reporting evaluation; and (3) Second food reporting evaluation and post-data-reporting debriefing evaluation. Detailed plans for each round are presented below.
ROUND 1: INITIAL ACTIVITES EVALUATION
Evaluation Objectives
The objective of Round 1 is to evaluate respondents’ effectiveness, efficiency, and satisfaction when (1) accessing the survey, (2) completing primary respondent’s initial interview which includes entering the roster of the eligible household members who will independently or by proxy report their food acquisition information, (3) the primary respondent completing household members’ profile/income questionnaires on their behalf, (4) a household member completing his/her profile/income questionnaire. The text content that are to be programmed into the survey instrument for testing in Round 1 are included in Appendices Q and R.
Six adults (≥ 16 years old) and two children (11-15 years old) from four different households will be recruited for the Round-1 study. Two households will have two adult participants respectively, with one being a “primary respondent” who is the main food shopper or meal planner in the household. The other two households will have one adult participant and one school child participant of 11-15 years old respectively, with the adult being a “primary respondent.” Non-primary-respondent household member participants are referred to as “eligible household member.”
All participants must meet the following inclusion criteria:
Have not participated in any previous studies associated with FoodAPS-2.
Have an iOS or Android smartphone that they have used for at least one year.
Have a data service on their smartphone.
Agree to use their personal smartphone and data service throughout their participation in the study.
The distribution of socio-economic characteristics among participating households:
At least one household is participating in the Supplemental Nutrition Assistance Program (SNAP), and another household is participating in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
Two primary respondents are at least 65 years old.
At least two households have a third household member who is not an “eligible household member.”
Participants will be recruited through advertisements on Craigslist, a recruitment database maintained at the Census Bureau, flyers, word-of-mouth, etc. Appendix B contains the advertisement script to be posted on Craigslist, and Appendix C shows a flyer to be posted in public areas. A screener questionnaire will be used for participant screening and determine eligibility for participation (Appendix D). It is assumed that 50% of primary-food-shopper adults will complete the screener questionnaire but not meet eligibility criteria or decline participation. The estimated burden associated with screening is 30 minutes per participant.
Testing Tasks
The test session will consist of the following four main tasks to be performed by the participants:
Accessing the instrument on the participant’s personal smartphone.
Completing primary respondent’s initial interview, including entering the roster of the eligible household members who will report their food acquisition information.
(The primary respondent) completing household members’ profile/income questionnaires.
(Eligible household member) completing his/her profile/income questionnaire.
The session will be conducted virtually via MS Teams, with one household per session. At the beginning of the session, participants will be introduced to the study and sign a consent form. The participants will share their smartphone screen and perform the three testing tasks (see the section below). At the conclusion of the session, each participant will receive $50 as incentive for his/her participation. This test session is expected to last 60 minutes. Below are the major steps in order during the session:
The participants join MS Teams meeting.
The participants sign a Consent Form for adults. (Appendix F)
The participants complete a demographic questionnaire. (Appendix H)
The primary respondent completes the initial interview with (1) roster entry, his/herself profile/income questionnaires, and other ineligible household members’ profile/income questionnaires on their behalf.
The primary respondent completes a satisfaction questionnaire. (Appendix I)
The eligible household member launches the instrument.
The eligible household member completes his/her profile/income questionnaire.
The eligible household member completes a satisfaction questionnaire.
The test administer (TA) debriefs the participants on their task performance.
Prior to the day of the session, a “tech-check” drill of 5-10 minutes will be performed to ensure that the participants have MS Teams installed on their smartphone (Appendix A) and can communicate with the TA and share screen using MS Teams.
Think aloud: During completion of the task performances, participants will be asked to think aloud (speak out what he/she is thinking).
Probing: The TA may probe participants to elicit additional information about their experience of performing the tasks.
Passive observation: During task performance, the TA will observe participant’s behavior and take notes of participant’s task-performance behaviors.
Debriefing: The TA will conduct a debriefing at the end of the session to gather additional information about testing experience.
Quantitative performance measures: When possible, quantitative performance data (e.g., task completion time) will be collected.
Data Analysis
Quantitative data (e.g., task completion time) will be summarized with descriptive statistics. Qualitative data (e.g., researchers’ notes about participants’ behaviors during task performance) will be analyzed to identify usability issues. Usability issues will be classified as high (H), medium (M), and low (L) priority. High-priority issues are those that prevent a task from being successfully performed; medium-priority issues prolong task completion; and low-priority issues do not impact effectiveness and efficiency of task performance but may affect participant’s satisfaction (e.g., a participant may be dissatisfied with a screen layout or the formatting of text on a screen).
Report
A report on testing will be delivered to the sponsor after the completion of data analysis.
ROUND 2: Food reporting EVALUATION
Evaluation Objectives
The objective of Round 2 is to evaluate respondents’ effectiveness of, efficiency in, and satisfaction with using the instrument to take training in reporting food acquisition and to report food acquisition. The text materials that are to be incorporated into the instrument for testing in Round 2 are included in Appendix S.
Participants and Recruitment
Six adults (≥ 16 years old) and two school children between 11 and 15 years old, for a total of eight participants from six households, will be recruited for Round 2 Testing.
All eight participants must meet the following inclusion criteria:
Have not participated in any previous studies associated with FoodAPS-2.
Be fluent in English (reading, writing).
Have an iOS or Android smartphone that they have used for at least one year.
Have a data service on their smartphone.
Agree to use their personal smartphone and data service throughout their participation in the study.
Additional inclusion criteria for adult participants:
All adult participants must be the main food shopper or meal planner in the household.
Each of the six adults must come from a different household.
Additional inclusion criteria for child participants:
All child participants must be enrolled in a K-12 school, and if possible, are acquiring meals from a school lunch program.
Each child must belong to a different participating household.
The distribution of socio-economic characteristics among participating households:
At least one household is participating in the Supplemental Nutrition Assistance Program (SNAP), and another household is participating in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
At least two adults are 65 years old or older.
Two households have a child between the age of 11-15 years old participating in the study.
Two households have a child under the age of 11 years old with PR proxy reporting for them.
Participants will be recruited through the same methods as those used in Round 1.
Testing Tasks
The test session will consist of the following three main tasks:
Accessing the instrument on the participant’s personal smartphone device.
Taking the online training in using the instrument to report food acquisition information.
Reporting food acquisition information in simulated FAH and FAFH settings. It is unlikely that the food log’s barcode scanning feature, Google Places API feature, and food item database feature will be functional in Round 2, therefore Round 2 will focus on participants manually entering locations (name and address), food items, and the associated detailed information for the food item into the food log.
Food Reporting Use Cases
Three use cases (Appendices K,L,M) were designed to test three food acquisition scenarios: Food at Home (FAH), Food Away from Home (FAFH), and School Meal. The three use cases cover all the critical tasks (Appendix N) in reporting.
Test Session Procedure
The instrument usability/cognitive testing will be conducted in an in-person setting with access for remote observers via MS Teams, one participant per session. The participants will meet with the study team and perform the assigned tasks. The session will proceed in the order below. Cognitive probing may occur within each step. A debrief will be conducted at the end of the session. At the conclusion of the session, each participant will receive $75 as incentive for his/her participation. The session was designed for 90 minutes.
The participants sign a Consent Form. (Appendix F or G)
The participants join MS Teams meeting and share his/her phone screen on the Teams.
TA starts screen recording.
The participants launch the instrument.
The participants complete the online training in using the instrument.
The participants take a quiz on training in using the instrument. (Appendix P)
The primary respondent reports food acquisition based on the food items provided to him/her.
The primary respondent completes a satisfaction questionnaire.
If applicable, the eligible school child reports food acquisition based on the food items provided to him/her.
If applicable, the eligible school child completes a satisfaction questionnaire.
TA ends screen recording.
TA debriefs the participants on their task performance.
The participants complete incentive paperwork.
Data Collection Methods
Data Analysis
Same as Round 1.
Report
Same as Round 1.
ROUND 3: Food reporting AND CLOSING DEBRIEFING EVALUATION
Evaluation Objectives
The objective of Round 3 is to evaluate respondents’ effectiveness of, efficiency in, and satisfaction with using the instrument to take training in reporting food acquisition, to report food acquisition, and to complete the closing interview/debriefing. The text materials that are to be incorporated into the instrument for testing in Round 3 are included in Appendices S and T.
Participants and Recruitment
Same as Round 2.
Testing Tasks
The test session will consist of the following tasks:
Accessing the instrument on the participant’s personal smartphone device.
Taking the online training in using the instrument to report food acquisition information.
Reporting food acquisition information in simulated FAH and FAFH settings. The food log will have complete functionality in Round 3, therefore Round 3 will focus on utilizing the food log’s barcode scanning feature, Google Places API feature, and food item database feature.
Test Session Procedure
The testing procedure is the same as that of Round 2, except for the following: After the primary respondent and household members have reported food acquisition (Steps 9 and 11 in Round 2 procedure), the closing interview/debriefing will be conducted before the TA ends screen recording. At the conclusion of the session, each participant will receive $100 as incentive for his/her participation. The entire test session was designed to last 120 minutes.
Data Collection Methods
Same as Round 1.
Data Analysis
Same as Round 1.
Report
Same as Round 1.
BURDEN ESTIMATE
The estimated total number of participants for this study is 32 individuals. The 32 individuals can be divided into two groups: one group of estimated 8 individuals (nonrespondents) who go through the household recruitment screener and are determined to be either ineligible for the study or eligible but decline to participate, and the other group of 24 individuals (respondents) who are eligible and participate (8 for each of the three rounds). Among the 32 individuals, 24 of them are adult household main food shoppers (16 respondents plus 8 nonrespondents) and participate in recruitment screen of 0.5 hours each, totaling 12 hours of burden; 6 adults and 2 children participate in Round 1 tech check (0.25 hours each) and test session (1 hour each), amounting to 10 hours of burden; 6 adults and 2 children participate in Round 2 session (1.5 hours each), amounting to 12 hours of burden; and 6 adults and 2 children participate in Round 3 session (2 hours each), amounting to 16 hours of burden. The total burden is summed up to 50 hours. The table below provides a detailed breakdown of the response burden for this study.
Activity |
Hours per Participation |
Number of Participants |
Burden Hours |
Recruitment screening |
0.5 |
24 |
12 |
Round 1 tech-check |
0.25 |
8 |
2 |
Round 1 session |
1 |
8 |
8 |
Round 2 session |
1.5 |
8 |
12 |
Round 3 session |
2 |
8 |
16 |
Total burden |
|
|
50 |
INCENTIVES SUMMARY
The incentive is structures as 50 USD per hour of study participation per person. Session 1 is designed to be one hour long, Session 2 1.5 hours, and Session 3 2 hours. The total amount of incentive to be paid for this study is 1800 USD. See the table below for the details.
Activity |
Incentive per Participant (USD) |
Number of Participants |
Sum (USD) |
Round 1 session (1 hour) |
50 |
8 |
400 |
Round 2 session (1.5 hours) |
75 |
8 |
600 |
Round 3 session (2 hours) |
100 |
8 |
800 |
Total amount |
|
|
1800 |
CONTACT INFORMATION
The contact person for questions regarding this request is:
Lin Wang
U.S. Census Bureau
(301) 763-9069
APPENDICES
Appendix A: Instructions for Installing Microsoft Teams
Appendix B: Participant Recruitment Advertisement Script
Appendix D: Screener Questionnaire for Participant Recruitment
Appendix E: Round 1 Protocol
Appendix H: Demographic Questionnaire
Appendix I: Satisfaction Questionnaire
Appendix J: Incentive Voucher
Appendix K: Food-at-Home (FAH) Use Case
Appendix L: Food-away-from-Home (FAFH) Use Case
Appendix M: School-Meal Use Cases
Appendix N: Critical Tasks
Appendix O: Round 2&3 Protocol
Appendix P: Quiz for Training Assessment
Appendix Q Income and Profile Questionnaire
Appendix R Initial Interview
Appendix S Food Log Questionnaire
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Aaron Maitland |
| File Modified | 0000-00-00 |
| File Created | 2025-08-12 |