Various Economic Pretesting Activities

Advanced Monthly Retail Trade Survey / Monthly Retail Trade Survey (Centurion) Usability Testing Draft Protocol

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Part A – Introduction

1. Introduce observers and their background

2. Purpose of visit:

Let me start by telling you a little about what we will be doing today. The Advanced Monthly Retail Trade Survey (MARTS) and Monthly Retail Trade Survey (MRTS) are mailed out to monthly businesses in order to collect current data on retail sales and inventory.

Starting this year, we are putting the survey online and you will have the option to report through the Internet. Since this is the first time we’re doing this survey electronically, we want to get your feedback in order to make revisions before it is formally released.

3. Permission to audio-tape discussion? Have R sign consent form.

Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your tape. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.

Part B – Background Information

1. Respondent Background

• What is your role at the company?

• Have you completed any other government surveys? If so, which ones?

2. Electronic Reporting Background (if reported to other government surveys)

• Did you report electronically to any of those surveys?

• What made you choose to report electronically?

• What was your overall experience? Did anything stand out as being very helpful? Was there anything that you didn’t like?

• What benefits would there be to reporting electronically to a survey like the MARTS/MRTS? What drawbacks?

Now I’d like to show you an electronic draft of the MARTS/MRTS form. Here is the web-site address for accessing the survey (show the respondent the URL, survey key (if applicable), test username and test password).

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do.

Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.

Part C – Testing

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

3. Login Screen

• Do you think that the information given on the Login Screen was sufficient?

• What are your overall reactions to the process for logging into the system?

• Is there any other information you’d like to see on any of the screens you saw while logging in?

4. Main Menu (multiple months)

• Can you describe to me what you see on this page?

• Where do you need to go in order to start completing your form?

• If prior months were available for you to access, what would you do with them? Have you updated a prior month’s data before? How have you gone about doing that (phone, fax, other)?

5. Reporting Period Confirmation

• Is it clear what this screen is asking you about?

6. Main Menu

• In your own words, describe what this screen is showing you. What is being provided in each of the columns? Is this information helpful to you?

• (If they have gotten the prior Main Menu screen) What is the difference between this screen and the previous Main Menu screen?

• Is it clear where you need to go in order to start answering your survey? If not, what could be done to make the start of the form clearer?

• If you had questions about definitions or the survey items where would you go?

• Is there any other information that you need on this screen before you would begin answering your survey? Why?

• Did you notice the toolbar on this screen? What do you think each of the options will do?

7. Item 1: Mailing Address / Item 2: Contact Person

• Do you have any recommendations for how we could improve the layout of the items on these screens?

8. Item 3: Employer Identification Number (EIN)

• Look to see if they see the ‘No’ easily.

• What sort of information would you provide if you had made an acquisition? Is the amount of space provided sufficient for the type of information that you would typically provide?

9. Edit

• (If the R hasn’t triggered an edit at this point, have them do so on purpose. If they will get the inventory question or department stores, have them enter a miscalculation. Otherwise, you can just have them leave a screen blank to trigger an edit.)

• Is it clear to you based on the messages you are receiving on the screen that there is an issue? Can you explain to me what the problem(s) are?

• Is there a better way that we could notify you about the problem(s) on this screen?

• What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?

• What other information should we provide to you about your problems and how to correct them?

10. Item 4: Sales

• (If they looked at the Show Instructions) What indicated to you that there was more information available for this item?

  • Is this information that you would need to see every month?

• In your own words, what is the “if applicable” comment telling you?

  • Would you use this box?

  • Do you have any recommendations for revising this?

  • Is the space provided sufficient for the type of comments you would provide?

• (If they didn’t look at the Show Instructions) What does the ‘Show Instructions’ at the top of the screen mean to you?

  • Was it obvious to you that this would display additional information? If not, what could be done to make this more obvious?

  • What information would you expect for it to display?

  • Is this information that you would need to see every month?

11. Item 4: E-Commerce Sales Question (Yes/No)

• Do you have any recommendations for how we could improve the layout of this item?

12. Item 4: E-Commerce Sales Question (Value)

• What type of value is this question asking you to provide? Is this clear? If not, what could be done to make this clearer?

• Is it helpful to have the sales value repeated in the question? Why or why not?

13. Item 4: Retail – Department stores

• Did you expect the value for Item 4a(3) to populate automatically? (for R’s that entered values in 4a(1) and 4a(2)) Where did the value for that item come from?

  • Did you notice the comment in 4a(3) about the Sum of the items? Was this note helpful?

• In your own words, what is the “if applicable” comment telling you?

  • Would you use this box?

  • Do you have any recommendations for revising this?

  • Is the space provided sufficient for the type of comments you would provide?

14. Item 4b: Reporting Period

• Do you have any recommendations for how we could improve the layout of this item?

15. Item 4c: Book Figures or Estimates

• Is it helpful to have the sales value repeated in the question? Why or why not?

16. Item 5: Merchandise Inventory

• If you needed to return to a prior section, how would you do this?

• Did you have any problems navigating back to a prior section/screen? Do you have any recommendation for improving the navigation within the site?

• In your own words, what is the relationship between items 5a., 5b., and 5c.? Was it clear by the layout of wording that a+b = c?

• Is there anything that we could do to make this relationship between these three more obvious?

• Did you expect the value for Item 5c to populate automatically?

• In item 5(c) there are check boxes. In your own words, what are these check boxes asking.

• If this data were not available yet, what would you do?

  • Would you return to the form to complete it later?

  • When would you return to the form?

  • Would you return on your own or would you need a reminder?

  • How would you expect to get back to this item?

• If the inventory items are not applicable, would you enter any values in the fields?

• In your own words, please tell me what the question below 5c. is asking? Is there a better way that this item could be presented to you?

17. Item 6: Number of Establishments

• If you needed to return to a prior section, how would you do this?

• In your own words, what are you being asked to clarify in the remarks box on this screen?

18. Remarks

• Is there any other information that you need on this screen?

19. Review your responses

• What are your first thoughts when seeing this screen?

• Can you explain to me what is expected of you? What is this screen telling you?

• What does it mean when it is saying that you have 0 or X issues?

• (If there are edits) How will you go about fixing this/these issues(s)? When you have finished fixing the problems, what do you need to do next? How will you submit the form?

• What do you think of the title of this screen? Is it a logical title?

• What other types of review would you be interested in performing prior to submission?

• If you weren’t ready to submit your data at this point, what would you do?

20. Completion Certificate

• Is there any other information that you would need to have for your completion?

• Would you save a copy of this information for your records? How would you do this (print a copy or save a copy electronically)?

21. Main Menu (after submission)

• Would you be interested in accessing any prior month data? If so, where could that be provided on the site? How far back would be useful?

Part D – Wrap Up

• Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?

• Do you think that you would complete the online form in one sitting?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• What did you think about the amount of information that was displayed on each screen? Was it enough, too much, or not enough?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most about the online form?

• What are some of the things that you liked the least about the online form?

• Do you have any suggestions for how to make this online form easier for you to use?

Thank participant for their time and input!

8

MARTS/MRTS (Centurion) Usability Testing Draft Protocol

May 22, 2012