Round-2 Protocol Odd

Appendix O2a: Round 2 Protocol – Odd ID

This protocol is to be used for participants with ODD ID number, who will view the training videos prior to usability tasks.

Note: Email P a reminder one day before session, along with the Consent Form.

Thank you for agreeing to participate in this study. My name is XX and I am a member of the study team and I’ll be working with you today. The purpose of this study is to test a web-based survey instrument that will be used for the national survey of Americans’ food acquisitions and purchases. The survey will help us better understand what kind of food Americans get and how they get it, and will help make policies such that all Americans can afford to get healthy food. The survey is to be conducted by the Census Bureau on behalf of U.S. Department of Agriculture.

In today’s session, we will evaluate how easy or difficult it is to use the survey instrument for people like you. Your experiences will help us improve the instrument before it is used by a larger group of respondents. Today, we will ask you to report some food items acquired in differnt scenarios, then we will ask you a few questions about your experience with the instrument when completing these tasks. Upon completion of today’s session, you will receive $75 for your participation. Do you have any questions before we move on?

<Answer participants’ questions if any>

Great. Now, I would like you to read this Consent Form. It explains the purpose of this study and your rights as a participant. It also informs you that we would like to take an audio-and-video recording of our meetings during the study. The purpose for the recording is to get an accurate record of your performance and feedback so that we can best improve the web instrument. Only those of us who are involved in this study will have access to the recording and it will be strictly used for research purposes. Your name will not be associated with the recording or any of the other data collected during the study period.

<Consent Form Link: https://research.rm.census.gov/jfe/form/SV_2adY6VgYUBydbcW >

<Wait until the participant completes reading the Consent Form>

Do you have any questions about the Consent Form?

< Answer participants’ questions if any>

Great, now I would like you to sign the Consent Form if you agree with it.

<After the Consent Forms are sighed>

Thank you. Now I would like you to complete this demographic questionnaire.

<DemoQ Link: https://research.rm.census.gov/jfe/form/SV_0URvN09lccEgsgm >

<Wait until the participant completes the demographic questionnaire>

Thank you.

Before we start the study, I’d like to teach you a method called “Think Aloud.”

I want you to complete the tasks as if you are alone by yourself, but with one major difference. I would like you to “think aloud” as you are performing the tasks: Speak out aloud everything that you are thinking and feeling as you go about working with the web instrument . I am interested in the process you go through in your mind when you are using the web instrument . Let me show you how I would think aloud: For example, if I were to think aloud while I’m grocery shopping, it might sound like this: “I start my shopping in the produce section because that’s the first section I encounter. I check my list to see what I need. After I found the vegetable I want to buy, I move to the deli counter because that’s next to the produce section.”

Let’s do a practice: Please think aloud as you answer this question: How many windows are there in your home?

<Practice Think Aloud>

[HINT: Probe if they fall silent. If they just give a number ask them to do it again but this time so that you can understand how they came up with the number. Probes should be kept to a minimum – trying to not interfere with the timing data, so used sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…]

<Complete Think Aloud practice>

Great. That’s what I want you to do throughout this session. I will remind you to think aloud if you get quiet.

At the end of the session, I will have some questions for you about your experience of using the survey instrument.

<Set up smartphone screen sharing, if P doesn’t have MS Teams installed, help install it>

In order to observe how you use the web instrument, I’d like you to share your phone screen on the MS Teams.

<Assist the P to set up screensharing if needed>

<Wait until seeing participant’s smartphone screen >

Now we will start recording our session.

<Start screen recording>

<Put the link to the first video in the chat >

<Link to the first video: https://youtu.be/uLJU8BMC0w4 >

Now I’d like you watch three short videos and learn how to use the survey instrument.

Please tap the link in the chat and watch the first video.

<Wait until P completes watching the first video, then put the link to the 2^nd video in the chat>

<Link to the 2nd video: https://youtu.be/foha5yAKEBA >

Thank you. Now please tap the 2nd link in the chat and watch the 2nd video.

<Wait until P completes watching the 2nd video, then put the link to the 3rd video in the chat>

<Link to the 3rd video: https://youtu.be/z2GItngqb1A >

Thank you. Now please tap the 3rd link in the chat and watch the 3rd video.

<Wait until P completes watching the 3rd video, then put the link to the Training Quiz in the chat>

<Link to the Training Quiz: https://research.rm.census.gov/jfe/form/SV_0D0gE7opqovMoiW >

Thank you. Now I’d like you to tap the latest link in the chat and take a quiz on what you have learnt from the videos. Please keep thinking aloud while you are taking the quiz.

<Prompt the P to think aloud during the quiz if P falls into silent>

<Wait until P completes the quiz, then put the link to the survey instrument in the chat>

<Link to the survey instrument: https://research.rm.census.gov/jfe/form/SV_cZL8aAFbf5KmNfM >

Great. Now we are ready to start the food reporting tasks. If you run into any difficulties as you work with the web instrument, please don’t blame yourself. Any difficulties are the result of web instrument itself, not your skills or abilities. Please open the chat and tap the link I just put there. Let me know when you see the survey instrument on the screen.

<Wait until the P has the survey instrument on screen>

<Start FAH use case>

Thank you. Now I’d like you to perform a food reporting task. Please read aloud the task instructions.

<Hand P the Use Case 1 – FAH>

<After P completes reading aloud the use case, put the grocery bag on the table>

Here are the groceries you just got, I’d like you to report them now. Keep Thinking Aloud.

Barcode scanning is not working at this time, so please don’t use barcode scanning.

<Observe P’s performance of FAH Use Case>

<Wait until the P completes FAH Use Case>

< Tell P to click “Next” (if) when get to Names >

<Start FAFH use case>

Thank you. Now let’s do another task. Please read aloud the task instructions.

<Hand P the Use Case 2 – FAFH>

<Email P the receipt while P is reading the use case>

<After P completes reading the use case, show P the food items PPT on a laptop>

Here are the foods you remember ordered, and you have also received an electronic receipt in your email. The receipt also has some information you need to report. I’d like you to report what you ordered now. Keep Thinking Aloud.

<Observe P’s performance of FAFH Use Case >

<Wait until the P completes FAH Use Case>

<Once P completes all tasks, put the link to the satisfaction questionnaire in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

< Link Satisfaction: https://research.rm.census.gov/jfe/form/SV_b31L7f4sdQ5q2FM >

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like to ask you a few questions about your experience with completing these tasks:

<Conduct an ad-hoc debriefing on all tasks based on observation, also probe on extreme responses to the satisfaction questionnaire>

Specific probing questions:

1. Imagine that you received food from the someone who doesn't live in your household, What response option would you select in the following question? And why?

<After P replied to the above questions, ask the following>

Please tell me your interpretation of the text with three bullet points.

2. Imaging that you reported some groceries to the Food Log today, then Food Log asks the following question with 4 response options. Only the last two response options are applicable to you, which one you would select and why? Do you think there is any difference in meaning between the last two response options?

3. Imagine that you have already reported the large pizza, the Food Log prompts you to report it again, how would you handle it?

4. Imagine that you have selected “Somewhere else” to the first question below, how would you handle the 2^nd question? If you can’t find a place in the search, what would you do?

Thank you. You have completed this session.

<Stop screen recording>

<Put the link to the incentive voucher in the Chat>

< Voucher Link: https://research.rm.census.gov/jfe/form/SV_1ZVFZttN93e7swK >

Now, I’d like you to sign the incentive voucher. Please open the Chat and tap the link to the incentive voucher, fill in all the information, and sign.

<Wait for the incentive voucher to be signed, verify all information being correct and complete – Full name and full address, then give the incentive cash to PR in person>

Thank you again for your participation!

TA Notes:

Timing on Individual Cases

After 10 minutes for the FAH use case; After 7 minutes for the FAFH use case – Ask Participant:

Do you feel like you are close to being done reporting for this scenario?

If they say that they are far from being done, you can move them on, say:

Due to time we will move on to the next scenario.

If they say they are close – allow them to work for two additional minutes.

After 15 minutes for the FAH use case; After 10 minutes for the FAFH use case; After 7 minutes for the school meal use case – if participant is not close to an answer, say:

Thank you for your work on this scenario. Due to time constraints, I would like to move us onto the next scenario.

8