Usability Testing of the Web Version of the Microbusiness Innovation Science and Technology Survey

Microbusiness Innovation Science & Technology Survey

Usability Testing Draft Protocol

2014

We will not have time to investigate every issue with every respondent. Instead, we will keep a log of all issues encountered across all of the testing sessions, and use that log to guide which issues deserve additional attention in any given session.

Materials needed:

• website

• user ids and passwords (for in person interviews)

• Emails with user id and passwords (for WebEx interviews)

• Consent form returned via fax/email

• notebook, pen

Part A – Introduction

1. Introduce observers and their background

Welcome respondent, confirm individual’s and microbusiness’s name and thank them for participating. Ask respondent to briefly describe their role at the company. Remind the respondents that we anticipate the interview will last between 45 and 60 minutes.

2. Purpose of visit

Remind respondent of the purpose of today’s visit or call and that MIST is a survey of small or microbusinesses and that the testing today is for a web version of the MIST questionnaire. Note that the goal of the usability tests is to ensure that the online survey is straightforward and easy to understand and navigate. Tell respondent that before it goes live we want to get feedback in order to make revisions before it is formally released.

3. Consent Form

Provide consent form and ask respondent to read and sign. Remind respondent that the interviewer will be taking notes.

Part B – Background Information on working with WebEx (if this is an in-person interview, provide the respondent with a the website address and move to Section C - Respondent Completes Questionnaire)

Now I would like to go over a few features of WebEx with you before we begin looking at the electronic survey instrument.

If We Get Disconnected

If you should lose your audio connection but are still viewing the WebEx screen; hang up your phone and click the Audio button. The Audio button is located below your name. Once the Audio button opens click the “Call Me” button and answer your phone when it rings.

If you can hear the meeting but do not see the WebEx screen look for a Blue and Green ball at the bottom of your computer screen. Clicking this ball should restore your screen.

If you cannot see the meeting or hear the audio, please return to your email and use the link to join the meeting again. You will need to start over and re-enter your phone number. I will be here on the line waiting for you. If you are having trouble you can call me.

Open Browser

Which browser would you normally use to complete a survey: Internet Explorer, Firefox, or Chrome?

We will open up the browser you use and turn control of it over to you. This will allow us to see both your mouse movements and anything you see on your browser. We will not be able to see any other part of your desktop.

Background on Task

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. We do not expect that you will need to get information from records or consult with others in your organization in order to answer any of the questions. However, if you feel you must do that, please tell me that out loud as you answer a question.

User ID and PW

Please enter the User ID and Password that we provided by email. Please let me know if you have any questions or need me to repeat any of the characters.

Part C – Respondent Completes Questionnaire

I would like you now to work your way through the questionnaire. I am available if you need to ask me any questions, but we really would like to see what you would do if you were trying to do this all by yourself. Don’t feel bad if you do not know how to do something; if you have trouble, then other people will too, and by observing you we can make this easier for others. It would be helpful if you talk out loud about your experience while you are completing the questionnaire—e.g., if you have trouble finding out how to respond, you want to do something but don’t know how, or even that a particular section seems especially straightforward and easy to complete.

[Watch the respondent complete the questionnaire. If the respondent asks questions of you, write down what the issue was. Using the checklist, note which kinds of situations the respondent encountered. Observe the following things. Also complete the checklist in Part D.]

Observation guide

	Yes	No
Navigation
Did the user go backwards to look at previous responses?		
Did the user navigate the questionnaire using the mouse or keyboard?		
Make careful note of how respondent scrolls through long items. ________________________________________________________________
Make careful note of how respondent uses the navigation buttons. ________________________________________________________________
Errors and difficulties
Did the participant express any frustration or appear confused? What about? ____________________________________________________ _______________________________________________________________		
Did the user skip questions? Which ones? ____________________________________________________		
Did the user make errors? What were the errors? _____________________________________________ ________________________________________________________________		
Did the user resolve the error?		
Did any user errors result in an error message? How did s/he respond to and act on the error message(s)? ________________________________________________________________		
Did the user have difficulty working with the matrices?		
Did the user have difficulty submitting the final responses?		
Special features
Did the user ever make use of the menu?		
Did the user access the help screen while completing the survey? If so, when working on questions ________________________________		

[If the respondent seems to expect interaction, then here are some ways encouraging the respondent to talk out loud.]

• What are you thinking about here?

• What do you expect to happen next?

• What did you expect to happen when you ____?

• I noticed that you are giving a lot of thought to one question. Are you having a problem with either the question or how to respond to it? If so, what is the problem?

• That section seems to be particularly easy for you to answer. Is that right? What makes it so easy for you?

• It looks as if you just finished a major section of the questionnaire. What are your feelings about that section? Were there any places where you didn’t know how to respond? Were there any features that you particularly liked?

• [anything else?]

Part D – Review of Respondent’s Experience with Questionnaire

Review the checklist below (it is best to keep it on a separate sheet so it will always be available), noting which types of issues the respondent encountered. Depending on the time available, perform one or more of the exercises provided. After conducting the exercise, have the respondent talk aloud while reacting to the scenario (and observe/record the following for each one; you do not need to ask the respondents the questions below if you can perceive the answer).

• Was it easy or hard to identify the problem?

• Was it easy or hard to make a fix?

• Would the error messages result in frustration or would they seem helpful?

Checklist of issues

The respondent:	Yes	No	If no, possible scenario	Scenario performed
Used navigation screen			Navigate to question 4
Exited and re-entered survey			Exit and re-enter
Viewed a printable survey			View printable survey
Skipped through major section of questionnaire			Change Q1 to No
Received out of range warning			Change Q4a to 45
Entered percentages that do not sum to 100			Change Q11b or Q11a to 95.
Received error message that two responses conflicted			Change Q12 to 0, or if Q12=0, change Q12 to 100
Understood how to enter thousands			If unsure, discuss response to Q27
Generate negative statistic			Change Q28 to 5,000,000
Generated error message involving Q4			Change Q32a to 20

Following are some probes that might be used when applicable:

• I noticed that you gave a lot of thought to question # ___. What were you thinking about there?

[ONLY IF NEEDED]:

• [IF IT SEEMS TO BE A COMPREHENSION ISSUE]: In your own words, what information do you think this question is asking for? What could we change to make the question easier to understand?

• [IF IT SEEMS TO BE A RESPONSE ISSUE]: What could we change to make it easier to provide your answer?

• Section ______ seemed to be particularly easy for you to answer. Is that right? What made it so easy for you?

• What are your reactions to section _____?

• Were there any sections or questions where you weren’t sure how to respond?

• Were there any features of this survey that you particularly liked?

Part E – Wrap Up

• If you were completing this form electronically, you would have the ability to print the questionnaire. Do you believe you would you want or need a copy of the paper form? If so, what purpose would the paper form serve for you?

• Would you complete this on paper before completing online?

• Do you think that you would complete the online form in one sitting?

• If you needed to contact someone with a question about the survey, how would you do that?

• Now that you have worked through the online form, what are your overall impressions of it? Was it easy or hard to complete it?

• Did you have any issues with the navigation on the site?

  • How was the amount of scrolling? How about the amount of clicking?

• What are some of the things that you liked the most about the online form?

• What are some of the things that you liked the least about the online form?

• Are there any other features, shortcuts, or tips that we should provide in the online form to make it easier to complete?

• Do you have any other suggestions for how to make this online form easier for you to use?

Part F – Review of Sections (Time permitting)

(Selectively review the following topics as time permits.)

1. Login Screen

• Is there any other information you’d like to see on the screen while logging in?

• What are your overall reactions to the process for logging into the system?

2. General Instructions

• What are your overall reactions to this page?

• What information stands out to you?

• Would you normally pay attention to this page? Why / Why not?

• What is so obvious that it didn’t need to be said?

• What is missing that people need to know?

3. Menu Structure

• Did you notice the links to special pages?

• Where did you think those links would take you?

• Were those links helpful or not helpful? Why?

4. Navigation

• Did you find it easy or difficult to navigate through the questionnaire? Why?

• [IF DIFFICULT]: What could be changed to make it easier to navigate?

5. Layout

• Was there any question where the way it looks on the screen made it hard to figure out how to answer?

• Were there any items where you weren’t sure you needed to give an answer??

• For which question(s) was it difficult to figure out how to enter your answer? Why [was it /were those] difficult?

If respondent does not have any other suggestions or comments have them log out of the system and tell them that once they hang up the telephone they will be disconnected (If it is an in-person interview tell the respondent that you are done and he/she can close down the internet browser.)

Thank participant for their time and input!

CONSENT FORM

The National Science Foundation designs, conducts, and sponsors surveys on science, engineering, and technology. In order to produce the best information possible, the National Science Foundation routinely evaluates its surveys.

You have volunteered to take part in a study to improve one of the National Science Foundation’s surveys. In order to have a complete record of your comments, your interview session will be audio taped (and/or videotaped). We plan to use the tapes to improve the survey. Staff directly involved in this research project will have access to the tapes. In addition, a transcription service may have access to the audiotapes, to provide us with a written version of your comments. The tapes may also be used for training others to conduct this type of research, and in presentations to professional audiences.

This study is authorized by law (42 U.S.C. 1862 Section 3.a.6.).

I have volunteered to participate in this study, and I give permission for my tapes to be used for the purposes stated above.

_________________________ ____________________________

Researcher’s Signature Participant’s Signature

_________________________ ____________________________

Printed Name Printed Name

_________________________ ____________________________

Date Date

8