NCSES Website Redesign - User Needs Analysis

Attachment M: User needs assessment interview guide
Warm-Up
Thanks for taking the time to talk with me today. My name is [NAME] and I’m from a
digital agency (Alley) working with the National Center for Science and Engineering
Statistics or NCSES to study the effectiveness of the NCSES website. [If anyone is joining,
mention, “I’m joined by some colleagues who are going to listen to our conversation.”]
Today, we are going to discuss your use of science and engineering data and related
analytic information. After that, I’ll ask you to share your screen with me as we look at
the NCSES website together. At that time, I’ll ask you to perform some tasks. Don’t
worry; there are no right or wrong answers when we get to that part of the research.
Throughout our discussion today, please be completely honest with regard to your
feelings and preferences around the NCSES website and other data sources that you rely
on.
Finally, is it OK with you that I record our session? I will not share the audio or video
from your session publicly. It will only be used for my final report to NCSES and to help
those working to improve the NCSES website.
[Assuming they say yes.] Great. And the last thing we need before we can get started is to
review a consent form. Before starting, please review this consent form. Do you have any
questions? May I have your verbal consent to conduct the interview?
[Walk through consent form, answer any questions, then turn on the recorder and get
verbal consent to record. If the respondent says no, then walk through the consent form,
noting that no recording will be made, but that participation is voluntary.]
Discussion Questions:
1. Tell me a little about yourself. What do you do and how do you use science and
engineering data and information in your work? (Or outside of work if the
participant is a casual user.)
If necessary, probe: Are they most interested in data or analytic
products/summaries related to the data? Do they use the data in grant
proposals or policy memos? Do they need facts or charts? Do they gather
data for themselves or someone else? Do they conduct their own analyses
of data? Do they share data?
2. For those who work in policy: What policy topics are you working on? What do
you plan to work on in the future? Will the topics you are interested in change in
any way? If yes, why?
27

3. I’d like to learn about some organizations that you rely on for science and
engineering data and data analyses. Other than NCSES, what are these?
If they say NCSES is the only S&E site they rely on, probe to see if there are other
data sites that are not oriented around science and engineering data, but that they
find easy to use.
a. Do you use these organizations’ websites?
b. Can you think of a site that provides you with features or functionality that
are very useful to you in your work?
c. Can you think of a data or statistical website that has frustrated you in the
past? What didn’t you like about the site and why? How could it have
been more useful to you?
d. Do you have data needs that are not fulfilled by any sites?
4. When and why do you visit the NCSES website?
Make sure that the participant is clear that we are talking NCSES and not NSF.
Probe to ensure they understand the distinction between the two entities.
a. Where does NCSES fit within the ecosystem of data resources that you
rely on?
b. What could the NCSES website do better to assist you in your work?
c. NCSES is considering adding a web-based community forum to their
future site. Here, data users could talk with one another about particular
topics and their use of the data. Could you imagine participating in such a
forum? If yes, what would it need to be successful for you personally? If
no, why wouldn’t you use it?
5. If they do not visit the NCSES website, probe why not. What could NCSES
provide to you that would be more valuable?
Usability Test of NCSES

28

OK, we are now going to switch gears and focus on the NCSES website more closely. I’d
like to give you some tasks to complete and just observe you as you use the site. This is
not a test of your online skills; it’s just a way for us to discuss how the NCSES site can be
improved.
I’d like to ask you to navigate to the NCSES website and then share your screen with me.
The website is: https://ncses.nsf.gov. I will drop the link in the chat window.
Now that you are on the site, I’d like you to share your screen with me so I can see what
you see. Before you do share your screen, please take a minute to clear off anything
sensitive that you wouldn’t want me to see on your desktop or in your browser.
As I ask you to do things, it will be really helpful if you can try to talk out loud as you
think and navigate.
OK, ready? Super.
1. First, I’d like you to identify what data topics are covered by this site.
2. Select a topic and explore what subtopics are available.
3. Now I’d like you to perform a search for data or a statistic. It can be on any topic.
a. Now that you have results, I’d like you to narrow them down.
If it is not clear how easy or hard this task was, probe and ask them how it could have
been improved.
4. Please go to the Business R&D Innovation Survey (BRDS).
See if they can find it on their own. If not, provide a link in chat or direct them via the
Surveys landing page. https://www.nsf.gov/statistics/surveys.cfm or
https://www.nsf.gov/statistics/srvybrds/
a. What do you see on this page? Is this what you would expect to see on a
survey landing page?
Probe: Have you used BRDS data or analytic products before? It used to be
BRDIS with an “I.” Have you noticed that? Did you find the change
confusing?
b. How do you feel about the presentation of the content on this page?
c. Look at the other three tabs. Without clicking, can you say what content you
29

expect to find on those tabs?
d. OK, let’s select each one and see if it is what you expected.
e. Where would you expect to find top-line findings of the survey? And what
kind of content would you expect to have in top-line findings?
If there is time, you can ask them to look at the info brief on business R&D
performance in the U.S. reaching $400 billion in 2017.
https://www.nsf.gov/statistics/2019/nsf19326/nsf19326.pdf
f. Let’s go back to the data tab and find some raw numbers. Please describe
what you see. Is this what you expected?
i. What kind of data formats do you use and need? Tables, charts, maps,
other items?
g. In your work, if you are exploring a survey, are you also interested in related
content? If so, what kind of content? What kind of related content is typically
useful to you? Would you look for content on the same topic but from a
different study? Or are you interested in the same survey but from different
years? Could you use other types of related content?
h. Now I’d like for you to identify how the data were collected and calculated.
i. How would you download, print, and share data from this survey?
5. What improvements would you like to see NCSES make to their website on the basis
of what you saw today?
Those are all the questions I had planned for you today. Is there anything that you would
like to discuss with me that I did not cover?
Thank you for your time. We really appreciate it.

30