Addendum to Generic Information Collection Request
(8/02/2024)
Usability and Cognitive Evaluation for
the Data Collection Instrument of FoodAPS-2 Pilot Study
The U.S. Census Bureau would like to submit an addendum to the ICR package for the usability and cognitive evaluation of the data collection instrument for the Second National Food Acquisition and Purchase Survey (FoodAPS-2) Pilot Test, under the generic clearance for questionnaire pretesting research (OMB number 0607-0725, ICR Reference Number: 202209-0607-002). In this request, we are seeking clearance for changes made in this data collection. Explanations of the changes are provided below and sections with changes to the original submission are marked with an asterisk.
INTRODUCTION
FoodAPS is a nationally representative survey designed to collect information on the food purchase and acquisition patterns of U.S. households, sponsored by the U.S. Department of Agriculture (USDA). The U.S. Census Bureau is conducting a pilot test of FoodAPS-2 on behalf of USDA. As part of the pilot test, the Census Bureau will develop a web-based data collection instrument running on the Qualtrics platform and conduct a usability and cognitive evaluation of the instrument with the intention to minimize respondents’ burden and maximize data quality.
Purpose: The purpose of this study is to evaluate effectiveness and efficiency along with respondent satisfaction with using the Qualtrics-based online instrument to enroll in the FoodAPS-2 Pilot Test survey, report food acquisition, and complete the post-data-reporting debriefing.
Population of Interest: The population of interest for the proposed study is households in U.S.
*Timeline: The study will be conducted from February through October in 2024, rather than the original end date of July 2024.
Language: The study will be conducted in English.
The present study consists of three rounds: (1) Initial activities evaluation; (2) Food reporting evaluation; and (3) Second food reporting evaluation and post-data-reporting debriefing evaluation. Detailed plans for each round are presented below.
ROUND 1: INITIAL ACTIVITES EVALUATION
Evaluation Objectives
The objective of Round 1 is to evaluate respondents’ effectiveness, efficiency, and satisfaction when (1) accessing the survey, (2) completing primary respondent’s initial interview which includes entering the roster of the eligible household members who will independently or by proxy report their food acquisition information, (3) the primary respondent completing household members’ profile/income questionnaires on their behalf, (4) a household member completing his/her profile/income questionnaire. The text content that are to be programmed into the survey instrument for testing in Round 1 are included in Appendices Q and R.
Six adults (≥ 18 years old) will be recruited for the Round-1 study. We adjusted the minimum age for adulthood to 18 years old and the participation of children was eliminated due to changes to study scope. Four participants will be from four different households respectively and be “primary respondents” who are the main food shopper or meal planner in the household. The other two participants will be from two of the four households as non-primary-respondent household member participants (aka, “eligible household member”).
All participants must meet the following inclusion criteria:
Have not participated in any previous studies associated with FoodAPS-2.
Have an iOS or Android smartphone that they have used for at least one year.
Have a data service on their smartphone.
Agree to use their personal smartphone and data service throughout their participation in the study.
The distribution of socio-economic characteristics among participating households:
At least one household is participating in the Supplemental Nutrition Assistance Program (SNAP), and another household is participating in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
Two primary respondents are at least 65 years old.
At least two households have a third household member who is not an “eligible household member” (e.g., a child under 14 years old).
Participants will be recruited through advertisements on Craigslist, a recruitment database maintained at the Census Bureau, flyers, word-of-mouth, etc. Appendix B contains the advertisement script to be posted on Craigslist, and Appendix C shows a flyer to be posted in public areas. A screener questionnaire will be used for participant screening to determine eligibility for participation (Appendix D). It is assumed that 50% of primary-food-shopper adults will complete the screener questionnaire but not meet eligibility criteria or decline participation. The estimated burden associated with screening is 30 minutes per participant.
In Appendix D, the Privacy Act Statement was updated, and the minimum age for adulthood in Question 13 was changed to 18 years old.
Testing Tasks
The test session will consist of the following four main tasks to be performed by the participants:
Accessing the instrument on the participant’s personal smartphone.
Completing primary respondent’s initial interview, including entering the roster of the eligible household members who will report their food acquisition information.
(The primary respondent) completing household members’ profile/income questionnaires.
(Eligible household member) completing his/her profile/income questionnaire.
The session will be conducted virtually via MS Teams, with one household per session. At the beginning of the session, participants will be introduced to the study and sign a consent form. The participants will share their smartphone screen and perform the three testing tasks (see the section below). At the conclusion of the session, each participant will receive $50 as incentive for his/her participation. This test session is expected to last 60 minutes. Below are the major steps in order during the session:
The participants join MS Teams meeting.
The participants sign a Consent Form for adults. (Appendix F)
The participants complete a demographic questionnaire. (Appendix H)
Three changes have been made in Appendix H: (1) replacing the birth year/month question with an age question to reduce respondent’s burden, (2) updating the gender question to comply with the current OMB directives, (3) combining the race and ethnicity questions into one to comply with the current OMB directives.
The primary respondent completes the initial interview with (1) roster entry, his/herself profile/income questionnaires, and other ineligible household members’ profile/income questionnaires on their behalf.
The primary respondent completes a satisfaction questionnaire. (Appendix I)
The eligible household member launches the instrument.
The eligible household member completes his/her profile/income questionnaire.
The eligible household member completes a satisfaction questionnaire.
The test administer (TA) debriefs the participants on their task performance.
Prior to the day of the session, a “tech-check” drill of 5-10 minutes will be performed to ensure that the participants have MS Teams installed on their smartphone (Appendix A) and can communicate with the TA and share screen using MS Teams.
Think aloud: During completion of the task performances, participants will be asked to think aloud (speak out what he/she is thinking).
Probing: The TA may probe participants to elicit additional information about their experience of performing the tasks.
Passive observation: During task performance, the TA will observe participant’s behavior and take notes of participant’s task-performance behaviors.
Debriefing: The TA will conduct a debriefing at the end of the session to gather additional information about testing experience.
Quantitative performance measures: When possible, quantitative performance data (e.g., task completion time) will be collected.
Data Analysis
Quantitative data (e.g., task completion time) will be summarized with descriptive statistics. Qualitative data (e.g., researchers’ notes about participants’ behaviors during task performance) will be analyzed to identify usability issues. Usability issues will be classified as high (H), medium (M), and low (L) priority. High-priority issues are those that prevent a task from being successfully performed; medium-priority issues prolong task completion; and low-priority issues do not impact effectiveness and efficiency of task performance but may affect participant’s satisfaction (e.g., a participant may be dissatisfied with a screen layout or the formatting of text on a screen).
Report
A report on testing will be delivered to the sponsor after the completion of data analysis.
ROUND 2: Food reporting EVALUATION
*Evaluation Objectives
The objective of Round 2 is to evaluate respondents’ effectiveness of, efficiency in, and satisfaction with using the instrument to take training in reporting food acquisition and to report food acquisition. The text materials that are to be incorporated into the instrument for testing in Round 2 are included in Appendices S and U.
Appendix U includes the scripts of video training that were previously not included.
*Participants and Recruitment
Eight adults (≥18 years old from eight households will be recruited for Round 2 Testing. [Participation of children was eliminated due to changes to study scope.
All eight participants must meet the following inclusion criteria:
Have not participated in any previous studies associated with FoodAPS-2.
Be fluent in English (reading, writing).
Have an iOS or Android smartphone that they have used for at least one year.
Have a data service on their smartphone.
Agree to use their personal smartphone and data service throughout their participation in the study.
Are the main food shopper or meal planner in the household.
Come from different households.
The distribution of socio-economic characteristics among participating households:
At least one household is participating in the Supplemental Nutrition Assistance Program (SNAP), and another household is participating in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
At least two adults are 65 years old or older.
Participants will be recruited through the same methods as those used in Round 1.
Testing Tasks
The test session will consist of the following three main tasks:
Accessing the instrument on the participant’s personal smartphone device.
Taking the online training in using the instrument to report food acquisition information.
Reporting food acquisition information in simulated FAH and FAFH settings. It is unlikely that the food log’s barcode scanning feature, Google Places API feature, and food item database feature will be functional in Round 2, therefore Round 2 will focus on participants manually entering locations (name and address), food items, and the associated detailed information for the food item into the food log.
*Training Quiz
The training quiz probes participants’ comprehension of the following information:
Who should complete the Profile & Income Survey?
For whom does the primary respondent answer profile-and-income questions?
When does the respondent start to report food and drinks?
What needs to be done on a day when no food and drinks are acquired?
Who will report for those household members who don’t report food and drinks themselves?
Who will complete the Closing Survey?
Both purchased and obtained-for-free food and drinks need to be reported.
Respondent should report all the acquired food and drinks, for themselves or for others.
For online orders, the acquisition date is the date of order being placed.
Non-food or non-drink items are not to be reported.
Do not report items acquired prior to the first day of food reporting.
Keep all the receipts of acquired food and drinks.
Three ways of reporting food and drinking (barcode, PLU, text).
The concept of combo meal.
The section was not included in the previous submission.
*Food Reporting Use Cases
Two use cases (Appendices K2 and L) were designed to test two food acquisition scenarios: Food at Home (FAH) and Food Away from Home (FAFH). The two use cases cover all the critical tasks (Appendix N) in reporting.
We updated the original Appendix K with Appendix K2. The use cases in Appendices K2 and L were updated. The use case of School Meal was eliminated due to changes to study scope.
*Test Session Procedure
The instrument usability/cognitive testing will be conducted in an in-person setting with access for remote observers via MS Teams. The participants will meet with the study team and perform the assigned tasks. The session will proceed in the order below. Cognitive probing may occur within each step. A debrief will be conducted at the end of the session. At the conclusion of the session, each participant will receive $75 as incentive for his/her participation. The session was designed for 90 minutes.
The participants are introduced to the study. (Appendices O2a and O2b).
The original Appendix O was updated with Appendices O2a and O2b due to changes to study design as described in Bullet 6.
The participants sign a Consent Form. (Appendix F)
The participants join MS Teams meeting and share his/her phone screen on the Teams.
TA starts screen recording.
The participants launch the instrument.
Half of the participants complete the online training in using the instrument prior to the usability tasks. The other half complete online training using the instrument after the usability tasks have been completed. (Appendix U, Videos 1-3)
All participants take a quiz on training in using the instrument immediately after completing the training, regardless of whether training was presented before or after the usability tasks. (Appendix P2).
Appendix P2 updates the old quiz to assess the new video training for Round 2.
The primary respondent reports food acquisition based on the food items provided to him/her.
The primary respondent completes a satisfaction questionnaire.
TA debriefs the participants on their performance of video training and usability tasks.
TA ends screen recording.
The participants complete incentive paperwork.
Data Collection Methods
Data Analysis
Same as Round 1.
Report
Same as Round 1.
ROUND 3: Food reporting AND CLOSING DEBRIEFING EVALUATION
*Evaluation Objectives
The objective of Round 3 is to evaluate respondents’ effectiveness of, efficiency in, and satisfaction with (1) training in reporting, (2) using the instrument to report food acquisition, and (3) completing the debriefing interview. The text materials that are to be incorporated into the instrument for testing in Round 3 are included in Appendices S, T, and U. Content in Appendix T was updated.
Participants and Recruitment
Six adults (≥18 years old) from six households will be recruited for Round 3 Testing.
All six participants must meet the following inclusion criteria:
Have not participated in any previous studies associated with FoodAPS-2.
Be fluent in English (reading, writing).
Have an iOS or Android smartphone that they have used for at least one year.
Have a data service on their smartphone.
Agree to use their personal smartphone and data service throughout their participation in the study.
Are the main food shopper or meal planner in the household.
Come from different households.
The distribution of socio-economic characteristics among participating households:
At least one household is participating in the Supplemental Nutrition Assistance Program (SNAP), and another household is participating in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
At least two adults are 65 years old or older.
Participants will be recruited through the same methods as those used in Round 1.
Testing Tasks
The test session will consist of the following tasks:
Accessing the instrument on the participant’s personal smartphone device.
Taking the online training in using the instrument to report food acquisition information.
Reporting food acquisition information in simulated FAH and FAFH settings.
*Training Quiz
The training quiz probes participants’ comprehension of the following information:
Reporting the location of food acquisition.
Reporting the acquisition date of online orders.
Methods of reporting purchased food items.
Uploading paper receipts.
The section was not included in the previous submission.
*Food Reporting Use Cases
Four use cases (Appendices K3a, K3b, L) were designed to test two types of food acquisition scenarios: Food at Home (FAH) and Food Away from Home (FAFH).
Appendix K3b is a new use case for testing proxy reporting.
*Test Session Procedure
The instrument usability/cognitive testing will be conducted in an in-person setting with access for remote observers via MS Teams. The participants will meet with the study team and perform the assigned tasks. The session will proceed in the order below. Cognitive probing may occur within each step. A debrief will be conducted at the end of the session. At the conclusion of the session, each participant will receive $100 as incentive for their participation. The session was designed to last 120 minutes.
The participant is introduced to the study. (Appendices O3).
Appendices O3 is a new protocol designed for Round 3.
The participant signs a Consent Form. (Appendices F)
The participant joins MS Teams meeting and share his/her phone screen on the Teams.
TA starts screen recording.
The participant launches the instrument.
The participant completes the online training in using the instrument. (Appendix U, Videos 4-6)
The participant takes a quiz on training in using the instrument. (Appendix P3).
Appendix P3 is a new quiz to assess the new video training for Round 3.
The participant reports food acquisition based on the food items provided to him/her.
The participant completes a satisfaction questionnaire.
The primary respondent completes the debriefing interview (Appendix T).
TA debriefs the primary respondent on his/her performance of completing the debriefing interview.
TA ends screen recording.
The participant completes incentive paperwork.
Data Collection Methods
Same as Round 1.
Data Analysis
Same as Round 1.
Report
Same as Round 1.
*BURDEN ESTIMATE
This section is updated in accordance with the changes to the study design described above. The updated total of 44.5 burden hours is below the previously approved limit. No additional burden hours are being requested here.
The estimated total number of participants for this study is 28 individuals. The 28 individuals can be divided into two groups: one group of estimated 8 individuals (nonrespondents) who go through the household recruitment screener and are determined to be either ineligible for the study or eligible but decline to participate, and the other group of 20 individuals (respondents) who are eligible and participate (6 for Round 1, 8 for Round 2, 6 for Round 3). Among the 28 individuals, 26 of them are adult household main food shoppers (18 respondents plus 8 nonrespondents) and participate in recruitment screen of 0.5 hours each, totaling 13 hours of burden; 6 adults participate in Round 1 tech check (0.25 hours each) and test session (1 hour each), amounting to 7.5 hours of burden; 8 adults participate in Round 2 session (1.5 hours each), amounting to 12 hours of burden; and 6 adults participate in Round 3 session (2 hours each), amounting to 12 hours of burden. The total burden is summed up to 44.5 hours. The table below provides a detailed breakdown of the response burden for this study.
Activity |
Hours per Participation |
Number of Participants |
Burden Hours |
Recruitment screening |
0.5 |
26 |
13 |
Round 1 tech-check |
0.25 |
6 |
1.5 |
Round 1 session |
1 |
6 |
6 |
Round 2 session |
1.5 |
8 |
12 |
Round 3 session |
2 |
6 |
12 |
Total burden |
|
|
44.5 |
*INCENTIVES SUMMARY
The amount of incentive for each round and the total amount in the table below are updated in accordance with the changes to the number of participants described above.
The incentive is structured as 50 USD per hour of study participation per person. Round 1 is designed to be one hour long, Round 2 1.5 hours, and Round 3 2 hours. The total amount of incentive to be paid for this study is 1500 USD. See the table below for the details.
Activity |
Incentive per Participant (USD) |
Number of Participants |
Sum (USD) |
Round 1 session (1 hour) |
50 |
6 |
300 |
Round 2 session (1.5 hours) |
75 |
8 |
600 |
Round 3 session (2 hours) |
100 |
6 |
600 |
Total amount |
|
|
1500 |
CONTACT INFORMATION
The contact person for questions regarding this request is:
Lin Wang
U.S. Census Bureau
(301) 763-9069
APPENDICES
Appendix A: Instructions for Installing Microsoft Teams [No changes and not included in this addendum]
Appendix B: Participant Recruitment Advertisement Script [No changes and not included in this addendum]
Appendix C: Participant Recruitment Flyer [No changes and not included in this addendum]
Appendix D: Screener Questionnaire for Participant Recruitment
Appendix E: Round 1 Protocol [No changes and not included in this addendum]
Appendix F: Consent Form for Adults [No changes and not included in this addendum]
Appendix G: Consent Form for Minors [No changes and not included in this addendum]
Appendix H: Demographic Questionnaire
Appendix I: Satisfaction Questionnaire [No changes and not included in this addendum]
Appendix J: Incentive Voucher [No changes and not included in this addendum]
Appendix K2: Food-at-Home (FAH) Use Case for R2
Appendix K3a: Food-at-Home (FAH) Use Case for R3a
Appendix K3b: Food-at-Home (FAH) Use Case for R3b
Appendix L: Food-away-from-Home (FAFH) Use Case
Appendix M: School-Meal Use Cases [No changes and not included in this addendum]
Appendix N: Critical Tasks [No changes and not included in this addendum]
Appendix O2a: Round-2 Protocol Odd
Appendix O2b: Round-2 Protocol Even
Appendix P2: Quiz for Training Assessment for R2
Appendix P3: Quiz for Training Assessment for R3
Appendix Q Income and Profile Questionnaire [No changes and not included in this addendum]
Appendix R Initial Interview [No changes and not included in this addendum]
Appendix S Food Log Questionnaire [No changes and not included in this addendum]
Appendix U Scripts of Video Training
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Aaron Maitland |
| File Modified | 0000-00-00 |
| File Created | 2025-08-12 |