Various Economic Pretesting Activities

2012 Economic Census Direct Internet Reporting

Centurion Usability Testing

Draft Protocol

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Respondent Background

• Can you tell me a little about your company organization? What products or services does it provide?

• What is your role in the organization?

• Have you completed any other government surveys? If so, which ones?

Electronic Reporting Background (if reported to other government surveys)

• Did you report electronically to any of those surveys?

• What made you choose to report electronically?

• What was your overall experience? Did anything stand out as being very helpful? Was there anything that you didn’t like?

• What benefits would there be to reporting electronically to a survey like the Economic Census? What drawbacks?

Now I’d like to show you a draft of the web version of the 2012 Economic Census. Here is the website address for accessing the survey (show the respondent the URL and test name/password).

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do.

Don’t hesitate to tell me anything you find surprising as you are going through the survey, or if you have questions about anything you see or any of the questions. We’ll try our best to answer your questions.

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

Login Screen

• What are your initial reactions to this page? Is this what you expected to see?

• What do you think it is telling you to do?

• Do you think that the graphic used to show the location of the username and password on the paper form is helpful? Could we show anything else on this screen that would help you locate this information?

• Is there any other information you’d like to see on the Login Screen?

• What would you do to log in? (Have R enter UID and PW. Probe any difficulties the R might have)

Main Menu

• What are you initial reactions to this screen?

• Can you describe to me what you think each of the sections is for? What do you think it’s telling you? (if R is unsure about any of the sections, walk them through each of them)

  • Do you think that the labels we have for each section match with what the section is for?

  • (If not) Do you have any suggestions for more appropriate labels?

• Did you notice the links available on the tool bar? (If not, point them out to the respondent)

• Did you notice the bullets at the bottom of this page? What was you initial reaction to them? Why do you think they are there?

• If you wanted to glance at information about the survey before starting a form, where would you go?

• What type of information is being given in the Announcements section?

  • Did you notice the Announcements section when you first entered the main menu?

• What type of information would you expect to see in the Announcements section? Is the information currently provided in the announcements section helpful?

• Is there anything else that you would need at this point in order to begin completing the form? Anything else that should be included on this screen?

• If you wanted to start filling out your form(s), what would you do next?(Observe what respondents click on to access the first form. Probe any difficulty)

• Was what you needed to do clear? If not, what could be done to make the starting of the form clearer?

In this section of the testing, we will walk respondents through the first form. In the interest of time, we will not probe every question. We will ask general probes and explore any difficulties the respondents may display. For items that we focus on, we will include the general probes in addition to the specific probes listed for each item.

General probes for this portion of the testing session:

• What are your initial reactions to this screen?

• What do you think about the amount of information on the screen? Do you feel that it is too little, too much or a comfortable amount?

• What do you think it’s telling you to do?

• What do you think of the layout of the screen?

• Is there anything unclear about this screen?

• How would you answer this question?

Physical Location

• What do you think about the navigation options offered at the bottom of the screen? Do they make sense to you? Do you have any suggestions for improving them?

Operational Status

• What if you didn’t want to answer this question? What do you think would happen? (Have R leave blank and trigger the error)

• What are your reactions to what you see?

• Is the error message clear? If not, how could we make it clearer? (Have R leave this item blank and continue)

Employment and Payroll

• If you needed to return to the previous screen, how would you do this? Can you show me?

• Did you have any problems navigating back to that screen? Do you have any recommendations for improving the navigation within the site?

• How would you return to the Payroll item?

• What are your thoughts about navigating between screens? Is there anything we could do to improve this navigation?

Kind of business

• What do you think would happen if you left this item blank? (Have R leave blank and trigger the warning.)

• What are you reactions to what you are seeing?

• Is it clear what you need to do to fix the problem

• Is there a better way that we could notify you about the problem(s) on this screen?

• What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?

• What other information should we provide to you about your problems and how to correct them?

Detail of Sales, Shipments, Receipts or Revenue (Item 22)

• Did you read the instructions on this screen? If not, is there anything that we could do to make the instructions easier to read?

• How would you go about filling out this item? (Have R enter some numbers)

• Any suggestions for how we could improve this item?

Adding Departments/Concessions (Part A and B)

If R has no depts/concessions to add:

• What do you think this item is asking you?

• How would you answer Part A? Part B?

• What could we do, if anything, to make it clearer that this item does not apply to you?

If R has depts/concessions to add:

• What do you think this item is asking you?

• How would you answer Part A?

• How would you answer Part B?

• How would you add a department? Is it clear what you need to do? What could we do to make it clearer?

• (Have R click on the “Add” button) What are you reactions to this screen? What would you do next? (Walk R through the “Add” process)

• How easy or difficult was it to add a department? What could we do, if anything, to improve the process?

Form Review

• (If there are edits – Warnings, Errors) What do these edit messages mean to you?

• (If there are edits) How will you go about fixing this/these problem(s)?

• What other types of review would you be interested in performing prior to submitting this form?

• When you have finished fixing the problems, what do you need to do next?

Starting Form NC-XXX

• (Back at the main menu) What would you do next to continue with the survey?

• In the interest of time, we are going to skip through this form. However, is it clear what you need to do?

(Have respondent start the second form and quickly page through it to activate the submit button)

Submission

• What would you do if you were ready to submit your forms?

• Did you notice the “Submit” button before we started? Did you notice that it is now active? What do you think about that?

• Is it clear what you needed to do to submit your forms?

• Was it clear to you that you needed to return to the main menu in order to submit your forms? Is there anything we could do to make that clearer?

Submission Confirmation

• Is there any other information that you would need to have for your confirmation?

• Would you save a copy of this information for your records? What would you save (answers, submission confirmation)? Can you show me how you would save that?

• How would you print the submission confirmation?

• Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?

• Do you think that you would complete the online form in one sitting?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most about the online form?

• What are some of the things that you liked the least about the online form?

• Do you have any suggestions for how to make this online form easier for you to use?

Thank you for your time and assistance today!

7