Read Consent message
Start Recording
Obtain verbal consent
OK, I’m now going to ask you to perform some tasks using the site. I’ll also ask some questions after each task to find out how it went. At the end I’ll ask some questions about your experience overall.
For the most part, I’d like you to perform the tasks as you naturally would if you were doing them on your own. So if you’re not sure what to do next, and you ask me a question, I may ask you to try to answer it yourself, just to see how well the site works without anyone explaining it. But if your question is about what I’m asking you to do, I’ll certainly try to clarify it.
The only difference from working as you would on your own is that I’d like you to let me know what you’re looking at and what you’re thinking as you come across things on the site. I can see what page you’re on, but I can’t always tell what you’re looking at, and it will really help me to understand your response if you let me know how you’re reacting to everything you encounter.
Any questions before we begin?
Testing System URL TBD – Display the Login Page
What do you expect “Preview the Survey” to do? What type of information do you expect to find here?
What type of information do you expect to find in “About the Survey”?
What type of information do you expect to find in “What’s New”?
How would you expect to use the documents under “Related Information”? Is this the type of information you would expect to find under “Related Information”?
What do you expect the Help link to do? [Prompt: What do you think will happen if you click the link? What type of information do you expect to find here?]
Ask participant to login using the supplied login credentials for their test record.
Display the Main Page
What do you think will happen if you click “Complete the survey now”?
What do you expect to be able to do with “Complete an Excel Version of the Survey”? Under what circumstances would you complete an Excel survey rather than the Web survey?
Display the Question List Page
What do you think will happen if you click on Question 1?
If you click the Print button, what do you expect to happen?
[Observe: Do they expect to print the page or the survey?]
If you click on the Submit button, what do you expect to happen? What do you think must happen in order for submission to occur?
[Observe: Do they notice that the button is grayed out? Is it clear to them that the rest of the screen must be completed before this option will be available to them?]
You have logged into the HERD system, and you want to take a look at your data from last year, to see if you can simply modify it. Where might you go to do that?
Participant must go to Download Previous Year’s Data and follow through the process, saving the 2009 file to their computer.
Observe:
Participant goes straight into the current year’s data
Participant fails to notice the “Download last year’s data” link
Participant clicks on “Compare 2008-2010 data” rather than “Revise Last Year’s Data”
Download begins, participant assumes an error because he or she assumes last year’s data will load in the online system
Display the Question List Page
Now we are going to refer to the data your institution reported in 2010 to solve some problems; I’ve emailed a form with your 2010 HERD data to you, and you can refer to it to complete the remaining tasks. It may help if you print this form out.
We’ve made some changes to your data in this test system (that won’t be saved after this exercise) that we would like you to find and undo in the system.
We’ve removed your responses from two questions. Please identify the two questions and re-enter your 2010 data.
Enter data for 2010 into Questions 9B and 9G-1. (Facilitator. one will be all zeros)
Observe:
Method used to identify questions with missing data. Icons in the question list should point participants to these two questions
Data entry errors
Participant response to system messages
Begin at any page
We have removed your responses from two lines in Question 12B. Please re-enter them.
You’ve discovered that you cannot report data for Institutional funds (column d) broken out like the survey asks this year. Please indicate that institutional funds by field of research is unavailable.
Enter data for 2010 into Questions 12B. Use the drop-down to select “Unavailable” for each cell on column d. Provide an explanation.
Observe:
One of the values is zero. Does participant select this from the combo box, or type it in?
Data entry errors
Participant response to system messages
Difficulty using combo box
Comprehension of options provided in the combo box.
Display Question List
Two errors have been introduced into the test data in the system. Please find and correct them.
(Facilitator: errors will be between Q1 and Q13 and between Q8 and Q13.)
Locate both errors and correct them.
Observe:
Use of question icons to locate existing errors
Ease of rejection for errors that are dependent on the an erroneous value, but aren’t erroneous themselves.
How do they detect that all problems have been resolved?
Are they distracted by Trend Variance messages?
Begin at any page
Do you know what a trend variance is? Can you explain it to me?
There are two trend variances in this system. Please find them and provide explanations in the system for the variance. If it’s not a trend variance that you are familiar with from your 2010 data, please explain that a new large grant came with a new faculty member in 2010, accounting for the large increase.
Once these have been addressed, please submit your survey.
Locate both trend variances and explain them separately.
Observe:
Use of question icons to locate existing errors
Comprehension and use of trend variance messages
Ability to find the submit button and understand the process
A small number of institutions upload their HERD survey as an Excel file. Please find where this feature is located.
We emailed you a test file to use for the upload. Please upload it.
Once uploaded, is there anything else you need to do to submit your data by file?
Locate this feature and upload the file.
Observe:
Do they understand that successfully uploading the file does not complete the process
Comprehension that errors still need to be address before submitting
Ability to find the submit button and understand the process
Display main page
We’ve seen some of the HERD data collection system today. Is there any area that you would like to look at more, or spend additional time with? If so, please take a few minutes to explore the test system. Let me know when you have questions or comments.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Windows User |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |