Census of Governments Retirement Component
(Centurion) Usability Testing Draft Protocol F-12(S)
(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)
What is your role here at this government?
Have you completed any other government surveys? If so, which ones?
Did you report electronically to any of those surveys?
What made you choose to report electronically?
What was your overall experience? Did anything stand out as being very helpful? Was there anything that you didn’t like?
What benefits would there be to reporting electronically to a survey like the Census of Governments? Would there be any drawbacks?
Did you report the Survey of Public Pensions online in the past?
Now I’d like to show you an electronic draft of the Survey of Public Pensions form (name form). Here is the web-site address for accessing the survey (show the respondent the URL, survey key (if applicable), test User ID and test password).
I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do. If you do not have the information available at this time, please feel free to insert any number into the boxes you would normally report in.
Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.
(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)
General Observations:
Scrolling: Make careful note of how respondents scroll through long items.
Instructions: Make careful note of how respondents treat the examples at the top of some of the items.
Navigation: Make careful note of how respondents use the navigation buttons
Error: Make note if R triggers any errors, if not have them trigger one purposely
3. Login Screen
What are your overall reactions to this screen?
Do you think that the information given on this screen was sufficient?
Is there any other information you’d like to see while logging in?
What are your overall reactions to the process for logging into the system?
4. General Instructions
What are your overall reactions to this page?
Where do you need to go in order to start completing your form?
Is there any other information you want on this screen to help you get started?
Would you look at this information before completing the form? Do you think you may want to return to this information at any point while completing your form? How would you do that?
5. Main Menu
Do you have any suggestions or recommendations for improving the layout of this screen?
Where do you need to go in order to start completing your form?
Was this clear? If not, what could be done to make the start of the form clearer?
If you had questions about definitions or the survey items where would you go?
Is there any other information that you need on this screen before you would begin answering your survey? Why?
Did you notice the toolbar on this screen? What do you think each of the options will do?
Did you notice the due date?
Would you scroll through this page prior to beginning the form?
6. Address Verification (Q1)
Note any difficulties with this item.
7. Retirement System Coverage and Organizational Information (Q2)
Do you have any recommendations for how we could improve the layout of the items on this screen?
Is there any information you would like to see on this screen?
8. Plan Information for Defined Benefit Plans (Q3-6)
What are your overall reactions to this page?
Would you say this is too much information, too little information, or about the right amount of information?
Could you tell me a little bit about what you see on this page?
In your own words, what is the “Specify group(s)” comment telling you?
Would you use this box?
Do you have any recommendations for revising this?
Is the space provided sufficient for the type of comments you would provide?
Do you have any recommendations for how we could improve the layout of the items on this page?
9. Ending Date of Fiscal Year (Q7-8)
Do you have any recommendations for how we could improve the layout of the items on these screens?
How would you answer this question? Is it clear what information you’re supposed to provide?
Are there any other types of information that would be helpful for you to have on this screen?
10. Membership and Benefits for Defined Benefits Plans (Q9-11)
What do you think about the layout?
What do you think about the amount of information on the screen? Do you feel that it is too little, too much or about right?
How would you answer this question? Is it clear what information you’re supposed to provide?
Is it clear where you would report your information?
Did you notice the include/exclude statements in item 9? What were these include statements telling you?
Could you look at item 11 for me? What type of information is this question asking you to provide? What are the differences between the first column and the second column? Would you be able to provide information for these items?
Are there any other types of information that would be helpful for you to have on this screen?
11. Navigation
If you needed to return to a prior section, how would you do this?
Did you have any problems navigating back to a prior section/screen? Do you have any recommendation for improving the navigation within the site?
12. Receipts for Defined Benefits Plans (Q12-13)
Do you have any recommendations for how we could improve the layout of these items?
Did you expect the value for Item 12b1(c) to populate automatically? Did you find this helpful?
Would you expect to be able to change the number that was populated in item c?
Could you test out this feature? What do you think about that?
Could you ever think of a time that you might use this feature? What would some examples be?
What do you think about the amount of information on the screen? Do you feel that it is too little, too much or about right?
How would you answer this question? Is it clear what information you’re supposed to provide?
Is it clear what type of information you are supposed to provide in these larger white boxes?
Are there any other types of information that would be helpful for you to have on this screen?
13. Edit Check
(If the R hasn’t triggered an edit at this point, have them do so on purpose)
Is it clear to you based on the messages you are receiving on the screen that there is an issue? Can you explain to me what the problem(s) are?
Is there a better way that we could notify you about the problem(s) on this screen?
What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?
For the sake of testing can you please ignore this message and continue without entering any information.
14. Payments for Defined Benefits Plans (Q14)
What is your initial reaction to this page? What do you think about the layout?
What do you think about the amount of information on the screen? Do you feel that it is too little, too much or a about right?
Are there any other types of information that would be helpful for you to have on this screen?
15. Cash and Investments for Defined Benefits Plans (Q15)
Do you have any recommendations for how we could improve the layout of these items?
Is there anything we can do to make this question clearer?
Did you notice the include/exclude statements in item 15A3? How about in 15D? How well do these categories match your records?
Are there any other types of information that would be helpful for you to have on this screen?
15. Actuarial Information for Defined Benefits Plans (Q16-26)
Do you have any recommendations for how we could improve the layout of these items?
What do you think about the amount of information on the screen? Do you feel that it is too little, too much or about right?
Did you notice the yellow box at the top of the screen? What is the information in this box telling you? Is there any way we could make this box easier to see?
Is it clear the type of information you are answering in each of these boxes?
What type of information is Q19 requesting?
Are there any other types of information that would be helpful for you to have on this screen?
16. Remarks
Do you have any recommendations for how we could improve the layout of the items on these screens?
Is it clear what type of information you should provide here?
17. Contact Information
Note any difficulties with this item.
18. Review your responses
Did you expect to come back to this screen when you completed the form? Why or why not?
Where would you have expected to go?
If no, what would you like to see?
What does “0 issues mean?” What does “1 issue” mean?
How would you go about correcting these issues? Can you walk me through the steps you would take?
How would you continue to correct or review the issues?
Is there something we could do to make this clearer?
What other types of review would you be interested in performing prior to submission?
When you have finished fixing the issues, what do you need to do next? How will you send the form to us?
How will you know the form is completed? Is this clear?
If you weren’t ready to submit your data at this point, what would you do?
Would you be able to send us the form before correcting the issues?
What does the note “must be completed prior to entering data” mean to you? Is that clear? How can we make it more clear?
What are your reactions to this (popup)? What does this note indicate to you?
19. Attach Data
Before we wrap up I was wondering if you could tell me if you would be interested in providing a copy of your Comprehensive Annual Financial Report, audit, or some other financial document in addition to completing this survey?
If you did want to provide a CAFR do you think you would be able to provide that using this internet site?
Would you mind clicking on the Attach Data tab? Did you notice this button before? If no, how could we help make this more visible?
What do you think of the overall layout of this page?
I am going to give you a minute to read through this page.
Would you mind testing out this function? Could you walk me through the steps you would take to attach a CAFR? If you do not have a CAFR available would you mind attaching a blank document?
What are your reactions to this process?
How would you know if your document was attached?
Is there anything else you would like to see on this screen?
20. Submission Certification Screen
What is your initial reaction to this page?
What do you think of the overall layout of this page?
Because we are still in testing mode this button does not work now, but what do you expect the “View statistics” button do? Would this be something you would utilize?
What would you do with this confirmation?
Would you save a copy of this information for your records?
Is there anything else you would like to see on this screen?
Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?
If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?
When the new instrument is available electronically there will be an option to print a blank form or a copy of your answers, would that be helpful to you?
Would you complete this on paper before completing online?
About how long would it take to complete this form online?
How did you feel about the amount of information that was provided on each page? Was it too much information, too little information, or about right?
How did you feel about the amount of scrolling? Too much, too little, about right?
How would you prefer to receive reminders about this form? Via letter or email?
Would you still need us to send you a paper version of this form if you chose to start completing the form online? Why or why not?
Did you have any issues with the navigation on the site?
What are some of the things that you liked the most about the online form?
What are some of the things that you liked the least about the online form?
Are there any other options that we should provide in the online form?
Do you have any suggestions for how to make this online form easier for you to use?
September 2012
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Quarterly Financial Report: Long Form |
| Author | oneil302 |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |