Cognitive Interviews for the Survey of New Foreign Direct Investment; Usability Testing of the BRDIS Internet Reporting Application

Business Research and Development and Innovation Survey (BRDIS)

Usability Testing Draft Protocol

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Part A – Introduction

1. Introduce observers and their background

2. Purpose of visit:

Let me start by telling you a little about what we will be doing today. The Business Research and Development and Innovation Survey (BRDIS) is an annual survey co-sponsored by the U.S. Census Bureau and the National Science Foundation.

In the past, we used a program called Census Taker to gather information online, but starting later this year we are moving to a new system, Centurion, to collect your information. Since this is the first time we’re doing this survey using Centurion, we want to get your feedback now in order to make revisions before it is formally released.

3. Permission to audio-tape discussion? Have R sign consent form.

Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your tape. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.

Part B – Background Information

1. Respondent Background and Electronic Reporting

• Please tell me a little about your company’s activities, and your role in the company.

• Have you completed the R&D survey in the past? On paper or electronically?

• Please tell me a little about your company’s R&D activities.

• What do you think are advantages and disadvantages of online reporting?

• Have you reported electronically to any surveys in the past?

• What made you choose to report electronically?

• What was your overall experience? Did anything stand out as being very helpful? Was there anything you didn’t like?

Now I’d like to show you an electronic draft of the BRDIS form. Here is the web-site address for accessing the survey (show the respondent the URL, test username and test password).

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do.

Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.

Part C – Testing

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

1. Username/Password Screen

• Do you think that the information given on the Login Screen was sufficient?

• Is there any other information you’d like to see on the Login Screen?

2. Main Menu

• Can you describe to me what this screen is giving you?

• Does the information displayed on this screen make sense to you?

• Where do you need to go in order to start completing your form?

• Was this clear? If not, what could be done to make the start of the form clearer?

• In your own words, what are the options available within the toolbar at the top of the page? What would you get or where would you go if you clicked on them?

• If you wanted to glance at the help instructions before starting the form, where would you go? Was this obvious?

• Is there anything else that you would need at this point in order to start completing the form? Anything else that should be included on this screen?

3. Form-Specific section -- TBD

• Form-specific probe -- tbd

• Form-specific probe -- tbd

• Form-specific probe -- tbd

4. Form-Specific section -- TBD

• Form-specific probe -- tbd

• Form-specific probe -- tbd

• Form-specific probe -- tbd

5. Form-Specific section -- TBD

• If you needed to return to a prior question, how would you do this?

• Did you have any problems navigating back to a prior screen?

• If you needed to go to another section, how would you do this?

• Do you have any recommendation for improving the navigation within the site?

• Form-specific probe -- tbd

• Form-specific probe -- tbd

• Form-specific probe -- tbd

6. Form-Specific section -- TBD

• (If the R hasn’t triggered an edit at this point, have them do so on purpose)

• Is it clear to you based on the messages you are receiving on the screen that there is an issue? Can you explain to me what the problem(s) are?

• Is there a better way that we could notify you about the problem(s) on this screen?

• What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?

• Form-specific probe -- tbd

• Form-specific probe -- tbd

• Form-specific probe -- tbd

7. Form-Specific section -- TBD

• Form-specific probe -- tbd

• Form-specific probe -- tbd

• Form-specific probe -- tbd

8. Review

• What are your first thoughts when seeing this screen?

• Can you explain to me what is expected of you? What is this screen telling you?

• (If there are edits) How will you go about fixing this/these problem(s)? When you have finished fixing the problems, what do you need to do next? How will you submit the form?

• What do you think of the title of this screen? Is it a logical title?

• What other types of review would you be interested in performing prior to submission?

• Are there any improvements that we could make to this screen that would help you do this review?

9. Thank You

• Is there any other information that you would need to have at this point?

• Would you save a copy of this information for your records?

10. Main Menu (after submission)

• Would you save a copy of the data for your records?

• Where would you need to go on this site in order to do that? Was that clear?

Part D – Wrap Up

• Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?

• Do you think that you would complete the online form in one sitting?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most?

• What are some of the things that you liked the least?

• Are there any other options that we should provide?

• Do you have any suggestions for how to make this online form easier for you to use?

• If others are required for completing BRDIS: How would you get others to complete their questions or sections?

  • Would you distribute the website to others?

    • If not (to above): What concern would you have about doing this?

    • If yes (to above): How would you notify them about this (through email, pass along a copy of the paper form, etc.)?

• Would you use the paper form to help you gather the data? If so, how?

• If already a Census Taker user: Would the process you used to respond electronically change at all in new system? How?

Thank participant for their time and input!

4

BRDIS Usability Testing Draft Protocol

September 24, 2009