2018 ASM-REPORT OF ORG. Debriefings Protocol
Research Questions
Respondent Portal:
Were respondents able to…
…register for an account?
…link a survey using the authentication code?
Did respondents use the delegation system? If so, did it function as expected?
Were respondents able to find the survey information, if needed?
Did respondents attempt to contact Census through the website? If so, did this function as expected?
Multi-unit (MU) Instrument (Centurion):
Does the flow of the instrument screens support respondents’ informational and decision-making needs with regard to reporting options and instrument functionality? If not, what other information did they need, and what are the optimal locations and formats for presenting such information?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons? If not, what terms do respondents suggest?
Are respondents able to discern the functions of individual screens? If not, what are the sources of confusion, and how do they suggest making them clearer?
What features did respondents use when completing the survey, and did these features meet their needs?
What additional features do respondents want in the instrument?
If respondent reported on the spreadsheet:
Was the visual design of the spreadsheet template effective in promoting readability of questions and instructions?
What additional features or information do respondents want in the spreadsheet template?
Before beginning
Introductions (if necessary)
Audio taping – would only be done if interviews take place in a private room.
Permission to audio- record discussion?
Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. Is this all right with you?
Participant Background
How long have you been with the company?
What is your title/role?
What are your major responsibilities?
What are your government reporting responsibilities?
What other government surveys or filings, if any, do you also handle?
Now I would like ask you specific questions about your experience using the Census website to report. If possible [convenient] please log into the Census website to refresh your memory.
General Probes (Use these whenever necessary)
In your own words, what would you say is the purpose of this page?
In your opinion, is the information on this page helpful or not helpful?
Can you tell me what you are thinking about?
Can you tell me more about that?
I want to make sure I understand; can you explain that again?
Reflect back on R’s answer (“I want to make sure I have it right. I think you said, “…?”)
Survey Background
Can you tell me about your process for filling out the Annual Survey of Manufactures/Report of Organization?
Additional probes (if necessary):
Was this your first time filling out this survey?
How many locations did you report for?
What is your role? How many people are involved?
How do you typically gather the data?
How did enter the data? (Form view/Spreadsheet)
Did someone else have to review the data before you submitted it?
Respondent Portal
Historically, the Census Bureau sends out an ID and a password for you to log into our website. In recent years we’ve asked you to create an account on a new website to report your data.
How did you feel about having to create an account?
How was the account creation process for you? Did you find it easy or difficult?
You were also required to enter an authentication code to access your survey. Was that process clear or unclear to you?
This website has a function that allows you to share your survey with coworkers.
Were you aware of this feature?
(If yes): Did you use this feature?
(If yes): What did you think of this feature? Was it helpful or not helpful?
(If no): Was there a reason you decided not to use it?
(If no): Do you think you would have used this feature if you had known about it?
Reporting Screen/Location List (REPORT OF ORG.)
I want to ask you about the main Reporting Screen.
Did you find the purpose of this screen to be clear or unclear?
How did you use this screen?
For companies with REPORT OF ORG. locations:
Were you able to find the list of your Report of Organization locations from this page? (If needed: Under the ‘Num. Estabs in Survey’ column, there would have been a blue link labeled ‘View Location list’ next to the number)
(If yes): Was it clear to you where to find this list of locations?
Did you find the Location List to be useful or not useful?
(If no): Did you look for a list of your Report of Organization locations? If so, where did you expect to find such a list?
There was a list of locations you could access on the Report screen; would you have used this list if you had known about it?
Add/Delete Locations
Did you need to add any new locations this year?
(If yes): How did you add it/them? Did you use the ‘Add Location’ button on the Screen, or some other way?
(If yes): Was the ‘Add Locations’ functionality clear or unclear? Were you able to successfully add the location(s) you needed to?
(If no): Did you know about the ‘Add Locations’ button? If so, is there a reason you chose not to use it? If not, would you have used it if you had known about it?
Did you need to delete any extra locations you might have added?
If so, did you use the ‘Delete Locations’ button on the screen, and did it work like you expected it to?
Was it clear that you could only delete locations for which you added into the instrument?
If REPORT OF ORG.
Was it clear how to add locations to Report of Organization only?
Question Preview
Did you use the question preview via the Preview Survey link to help you report?
(If yes):
Did you find it to be useful, or not useful? Why or Why not?
Did you print the questions and instructions? Why (not)?
Did you share the preview with anyone when gathering your data?
Do you have any suggestions that would make the preview more useful to you?
(If no): Did you know there was a printable version of the questions and instructions?
(If yes): Is there a reason you chose not to use it?
(If no): Would you have used it if you had known about it?
If respondent reported on spreadsheet:
Please tell me about your experience downloading the spreadsheet.
If needed: Did you find the process easy or difficult?
Did you download the ‘all locations’ spreadsheet and the ‘full survey’ spreadsheet?
Was the spreadsheet itself easy to use or difficult to use?
Is there any other information you would like to see in the spreadsheet or other features we should consider?
2019 ASM REPORT OF ORG.
Formulas?
Viewing added locations
Did you encounter any issues when uploading the spreadsheet?
Notifications (Messages):
Were the notifications in the Messages (upper right portion of the screen) helpful or unhelpful?
Did you notice the pop ups? Were they helpful? Explain.
Form view
MUs: Did you use the individual location form screens (question-by-question) in any way (to report or review your data)?
SUs: In general, how did you feel about the layout of the question and answer fields?
Review Screen:
Now let’s talk about the review step.
Can you tell me about your review process?
Did you find the review step on the website to be helpful or not helpful?
Is there other information you would like to see in a review step for this survey?
Did you have to make any corrections? How easy or difficult was this to do?
Did you use any of the available tools on the screen- Download errors/warnings or Auto-Sum totals?
If no, did you notice these options?
Is it clear what their purpose is?
Review Surveys and Submit:
Let’s talk about the submission step now.
Was it clear what the purpose of this screen was? Describe.
Did you find the submission process to be clear or unclear?
MUs: Did you submit the data for all of your locations at once, or submit each location once completed?
Was it clear how one could save a copy of their responses on this screen? Describe.
Did you save a record of your response?
Submission confirmation
What were some of the most important information conveyed on the confirmation screen
Is there any additional information you would have liked to see on this screen.
Did you notice the ability to print a copy of your survey responses via the Print Responses to PDF button
[If respondent did not complete for all locations at once]
When you first submitted the survey, was clear how many remaining surveys there were to complete? Describe
Wrap-up
Overall thoughts about the navigation and functionality?
Were the reporting steps/screens clear/unclear? How so?
Did you utilize/notice any help content while completing the survey?
Videos?
Help links?
2019 revisions
Next we would like to go over some of the changes made to the most recent version of the survey to get your feedback.
Item 22 NAPCS screen
Any thoughts on the new design to be able to use a filer to add products/services?
Describe how you expect to be able to use this functionality in the new instrument? Is it clear how to input your products?
Step 3- Review Surveys and Submit: Auto Sum totals placement
Any thoughts on the placement of the Auto Sum totals option on this screen instead of the prior screen (View Errors and Warnings). Where would you prefer this feature be located?
Download spreadsheet template
MUs that downloaded the spreadsheet template:
Any thoughts on this new 2019 layout. Is it clear what you should do prior to downloading a spreadsheet template?
Go to functionality
Is it clear what the purpose of the Go to Screen feature is?
Does the label make sense?
Any thoughts on this change for the 2019 version of the instrument?
Do you find this to be helpful/ unhelpful?
Are there any other suggested changes you would recommend to help with navigating between screens?
Are there any other comments or suggestions you would like to make about anything we have gone over today?
Thank the respondent for their time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Temika Holland (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |