Pretesting 878/888 Pretesting 878/888 (Example)

Usability Protocol for Form EIA-888 Schedule B

My name is __________________ and I work for the Energy Information Administration. Thank you for agreeing to participate in our study.

What: In order to help us improve our surveys, we turn to our respondents, like you, to find out if our questions make sense and are easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with respondents and see how it works for them. We then evaluate how easy or difficult EIA products are to use. What works well, we keep. When users, such as yourself, have difficulty with something, we have an opportunity to fix it. You will be helping us test the redesigned Motor Gasoline Survey Form EIA-878 Schedule B.

Confidentiality: Our session today is completely confidential. Any names you provide will never be used in our reports. Your participation in this study is voluntary and you can decline to answer any particular question. In addition to me, there will be a person from the program office. His/Her name is _________ (Name of PO). ___________ (Name of PO) can answer any technical questions you may have.

Think aloud method: Today you will be completing an EIA survey form. I want you to answer the questions exactly the way you would if you were completing it on your computer in your office, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[IF NEEDED:]Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

Great that’s what I want you to do throughout our session.

I will remind you to think aloud if you get quiet. From time to time, I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.

If you run into any difficulties as you work on the survey, please let us know as this will help us improve the survey.

Ok, let’s begin:

You would have received a welcome letter, this form and instructions at your office. Please answer the questions as you would if you were in your office answering the form.

Do you currently respond to Form EIA-878, Motor Gasoline Survey?

Give the respondent some time to read the instructions. If needed remind the respondent to think aloud (whenever possible).

Instructions Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

1. After R finishes reading the instructions: What are your thoughts about the instructions?

   1. If necessary: What if anything did you find confusing?

2. After reading the instructions on how to report… How easy or difficult is it to file the report?

   1. What, if any, are potential issues you foresee in filing the report?

Ok let’s move on to the first section, please complete this section.

Instructions Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

1. After R finishes reading the instructions: What are your thoughts about the instructions?

   1. If necessary: What if anything did you find confusing?

2. After reading the instructions on how to file… How easy or difficult is it to file the report?

   1. What, if any, are potential issues you foresee in filing the report?

Ok let’s move on to the first section, please complete this section.

Part 1 Identification Information Section: If respondent hesitates or expresses confusion during the task: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

3. Who would typically complete and file this form?

4. For the preparer’s information, what phone number did you decide to report?

1. Why did you decide to report this number?

5. For the preparer’s information, what email did you decide to report?

1. Why did you decide to report this email?

6. Are there any other relevant information that you think we need to ask here?

Ok let’s move on to the second section, please complete this section.

Part 2 Station Status Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

7. On question 1, were there any answer categories that were not clear or confusing?

   1. If yes, which ones were unclear?

8. On question 4, how did you come up with this answer?

9. Is there anything in this section that we did not discuss that is unclear?

   1. If yes, what makes this unclear?

Ok let’s move on to the third section, please complete this section.

Part 3 ANNUAL MOTOR GASOLINE SALES DATA Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

10. On question 1, could you tell me in your own words what the instructions are telling you to do?

    1. After reading the instructions, were you able to understand what you needed to do? If necessary, tell me more.

    2. How useful or not useful was the example provided in the form? If necessary, Could you elaborate?

       1. What would you change to make the example helpful for people filling this form?

11. On question 1 under octane rating, what is this section telling you to do?

    1. How easy or difficult is it for you to select an answer category? If necessary, what made it difficult? Or What made it easy?

12. Under the ethanol content section, could you tell me how you arrived at this number?

    1. For the answer categories, were there any answer choices that were difficult to understand?

    2. If you were to change the answer categories, how would the answer choices look like?

13. Under the Reason if zero or missing, what is this section telling you to do?

    1. How clear or unclear is the “Reason if Zero or Missing” section to you? If necessary: Could you tell me more…

    2. Were there any answer categories under this section that you thought were confusing? If necessary, Which ones? Could you elaborate?

14. Tell me about your experience working with the fifth question in this section.

    1. What does full calendar year mean to you?

    2. When reading the instructions, what difficulties, if any, did you have?

Ok let’s move on to the third section, please complete this section.

Additional information Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

15. After reading the additional information, was there anything that stood out to you?

16. In this section, was there anything that was difficult to understand?

Conclusion: General Questions

17. What are your thoughts about the color and design of the survey form?

1. If you would have to change something in the form, what would it be?

18. Approximately how long do you think it will take you to complete this form?

2. How long would it take you to gather the information you need?

3. How long would it take you to fill in the information and submit to EIA?

4. Does your outlet submit price or volume data to any other organizations? If yes:

   1. How frequent do you submit data to these organizations? If needed… Is it on a frequent basis or one time/ad hoc?

   2. About how long does it take you (or the preparer) to gather and submit the information?

   3. How is the information transmitted? If needed… Is it manual process or automatic/passive process?

19. Do you have any other feedback regarding your experience completing this form?

Thank you for your participation in this study!