Purpose: Researchers in the Economy-Wide Statistics Division (EWD) will conduct usability interviews to assess functionality of the prototype interactive question preview tool. This interviewing will examine whether respondents can successfully complete tasks that are designed to mimic those they execute using the interactive question preview tool to support response to the AIES. Researchers will investigate whether the interactive question preview tool is intuitive by assessing respondents’ ability to navigate through the tool in an efficient way to a successful completion of the task.
Research Questions: The research will be guided by the following research questions:
Do users understand the purpose of the tool?
Do they identify that the content is industry driven?
Do they understand collection unit differences?
Does this tool meet the need for the ability to preview the survey prior to reporting?
Can users generate content that matches their AIES content?
What features of the tool are intuitive? What features are not?
What additional features might support response?
Informed Consent: Respondents will be asked to complete a consent form electronically before the time of the interview.
Materials Needed:
Electronically signed consent form
Link to the Interactive Question Preview Tool
Thank you for your time today. My name is XX and I work with the United States Census Bureau on a research team that evaluates how easy or difficult Census surveys and products are to use. We conduct these interviews to get a sense of what works well, and what areas need improvement. We recommend changes based on your feedback.
[Confirm they signed the Consent Form; should be sent prior to interview.]
Thank you for signing the consent form, I just want to reiterate that we would like to record the session to get an accurate record of your feedback, but neither your name or your company name will be mentioned in our final report. Only those of us connected with the project will review the recording and it will be used solely for research purposes. We plan to use your feedback to improve the design of this survey instrument and make sure it makes sense to respondents like you. Do you agree to participate? Thank you.
[start CAMTASIA/Snagit screen recording, if yes.]
Thank you.
I am going to give you a little background about what we will be working on today. Today you will be helping us to evaluate the design of the interactive question preview tool for the Annual Integrated Economic Survey (or AIES). The interactive question preview tool is in the early stages of development, so this is an opportunity to make sure it works as smoothly as possible. You are being asked to be a part of this evaluation because your company recently responded to the 2022 AIES – and I just want to take a moment and thank you so much for your response!
To do this, we will have you complete various tasks using the interactive question preview tool. These will be consistent with tasks you would normally complete if you were using this tool. We are mainly interested in your impressions both good and bad about your experience. I did not create the tool so please feel free to share both positive and negative reactions.
I may ask you additional questions about some of the screens you see today and your overall impressions.
Do you have any questions before we begin? Ok let’s get started.
First, I would like to get some information to give me some context.
Can you tell me about the business, like what types of goods or services it provides? And how it is organized?
What is your role within the company?
Are you typically the person responsible for government surveys?
Do you typically have access to all of the data needed?
If not, what areas or positions do you usually reach out to?
In the past when responding to survey requests from the Census Bureau, did you preview survey questions before beginning the survey?
If yes, can you tell me more about that? How did you use that feature – simply to preview questions, to print and write in gathered responses, to share with colleagues as a means of gathering data, and/or some other way?
If not, do you think you’d be likely to use a question preview tool prior to completing AIES for the first time?
Great, thanks for that context. Next I’ll have you open the interactive question preview tool link, AIES Survey Questionnaire (census.gov), I sent to you and walk you through how to screenshare. You make take a moment to close anything you don’t need open.
[To screenshare, if necessary] Look for a button on Teams with a box with an arrow- it might be located next to the microphone button.
To begin, I’d like to set the stage by telling you more about the Interactive Question Preview Tool that we will be testing today. Since collection for the AIES is only by web and the questions you would receive are specific to your business, there is no designated survey form in which a respondent can preview the survey questions they will receive when taking the survey. Instead, there is an interactive question preview tool that respondents can use to indicate their industry (or industries) and preview questions at the company, industry, and establishment level. Again, this tool is in its early stages of development and your feedback will be helpful in improving this tool for its users.
Expectations of accessibility: Imagine that you have entered the AIES survey on the respondent portal and would like to preview questions to the survey in preparation for coordinating the gathering of the data you will need to respond to this survey. Where would you expect to find information on the Interactive Question Preview Tool?
Show the following instructions to the respondent via Qualtrics (using the same link provided for the consent form): This is the Interactive Question Preview Tool for the Annual Integrated Economic Survey. This tool will allow users to select their type of business operation and view the question collected at the company, industry, and establishment or location levels. This tool is NOT to be uploaded with responses in the respondent portal or submitted to the Census Bureau for response to the AIES. Instead, this is to be used only as a tool for respondents to preview survey questions that are specific to their businesses.
Comprehension: In your own words, please describe what these instructions mean to you? (Interviewer: note whether respondent seems to understand that this tool is only for the purpose of previewing questions for the AIES).
Are the instructions clear and concise? If not, what requires clarification?
Now let’s turn our attention to the Interactive Question Preview Tool webpage.
Please take a few seconds to look at what’s on your screen. What are your first thoughts or impressions of what you’re seeing on this screen? What is the first thing you notice, or what pulls your eye?
Would instructions for using this tool be helpful? And, if so, where would you expect to see those instructions?
Now we will work on some tasks related to using this interactive question preview tool. While you are completing the tasks, I would like for you to think aloud. It will be helpful for us to hear your thoughts as you move through the question preview tool. Once you have completed each task just let me know by saying finished or done- then we can discuss and/or move on to the next task
[During the interview, respondents will use their device. Have participant go through each task, remind them to think out loud, and note participant’s questions or signs of difficulty. Note any content issues if they arise.]
Think about what your industry does or makes. Now, select the industry or industries that best represent your company. [Interviewer: Note respondents’ selections(s) and whether or not the respondent’s selection matches the Business Registry (BR) industry assignment.]
Did you find it easy or difficult to select your industry/industries? If difficult, please tell me more about that and please provide some ideas for improvement. (Interviewer: Note if respondent selects all applicable boxes and/or expresses any uncertainty as to whether their selections triggered anything to happen in the tool – ie, questions to be added in the industry questions tab)
Now that you’ve made selections, let’s preview questions for your business at the company, industry, and establishment or location levels.
What are you looking at on your screen?
What do you notice?
What is this survey asking you about?
Where would you go to find the question for “What were the total sales, shipments, receipts, or revenue in 2022?” by location? And where would you go to find the question for “
Did you find it easy or difficult to preview questions for your business at the company, industry, and establishment or location levels? If difficult, please tell me more about that as well as any suggestions for improvement. (Interviewer: Note if respondent expresses any uncertainty, notices and clicks on the different tabs for company, industry, and establishment)
We’ll move on to the next task of exporting the questions for your business at the company, industry, and establishment level to a PDF and/or Excel file. Please show me how you would go about doing that.
Did you find it easy or difficult to export your survey questions to a PDF and/or Excel file? If difficult, please tell me more about that as well as any suggestions you may have for improvement. (Interviewer: Please note whether respondent experienced any issues completing this task including any technical difficulties.)
Overall, what was your impression of the interactive question preview tool? Is there anything you liked / disliked about the question preview tool?
In general, would you say that it was easy or difficult to use this question preview tool? Which aspects, if any, were made the tool difficult to use?
After seeing and interacting with this question preview tool, is it something that you would use?
If yes, how would you use it?
If no, why would you not use it?
If you could change anything about the question preview tool, what would that be?
Were there any features that were not intuitive?
Are the instructions on using the interactive tool clear and easy to follow?
If not, how could the instructions be more clear and easy to follow?
Is it clear that the question preview tool is only for the purpose of previewing survey questions? More specifically, is it clear that you cannot upload responses collected using this tool or submit this/these forms and/or worksheets as response to the survey?
Do you have any thoughts about the export feature? If a feature were available allowing you to export the question preview tool, is that something you would utilize, or no?
What would you expect an export feature to look like? Do you have a preference for a fillable PDF file or an Excel CSV file? Would you print this out, would you share the file electronically with others in your organization, and/or use this file or files in some other way?
Are there any other features or information that would support your use of the tool, that you think are currently missing from the tool?
If you were having trouble with using this tool, would you be interested in watching video tutorials?
Which tasks do you think need a video tutorial?
Where would you expect to find the videos?
What is the longest video tutorial you would watch?
Is there anything else you would like to mention that we haven’t talked about, either about the interactive question preview tool or the AIES in general?
This concludes our session. Thank you for your time and valuable feedback.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Rebecca Keegan (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2025-08-12 |