Supporting Statement for NCSES Website Redesign

MEMORANDUM

Date:

April 12, 2020

To:

Margo Schwab, Desk Officer
Office of Management and Budget

From:

Emilda B. Rivers, Director
National Center for Science and Engineering Statistics (NCSES)
National Science Foundation (NSF)

Via:

Suzanne Plimpton, Clearance Officer
National Science Foundation (NSF)

Subject:

Notification of data collection under generic clearance

The purpose of this memorandum is to inform you of NCSES’s plan to collect data on
users’ needs under the generic clearance for survey improvement projects (OMB #31450174). This study is part of a larger set of activities assessing how well the current
NCSES website, dissemination tools, and data products meet the needs of users.
Background
The National Center for Science and Engineering Statistics (NCSES) within the National
Science Foundation (NSF) is the principal source for analytical and statistical reports,
data, and related publications that describe and provide insight into the nation’s science
and engineering resources. The Center’s publications and related content are released in
electronic format on NCSES’s various website URLs (https://ncses.nsf.gov/;
https://nsf.gov/statistics/; https://ncsesdata.nsf.gov/).
NCSES will redesign its website and data dissemination systems to support content under
a unified subdomain URL (https://ncses.nsf.gov), supported by a new content
management system (CMS) where possible. In preparation for a complete redesign of the
main NCSES website and integrated data system (i.e., interactive table tool and metadata
explorer), NCSES is looking to conduct two phases of user needs analysis, then to
compile the results into a summary report that highlights existing gaps between user
expectations and the site’s current user interface.
This project involves two studies to prepare for the redesign of the NCSES website. The
purpose of the first study (primarily user interviews with some usability testing) is to
learn how participants use the current NCSES website, NCSES data and data products,
and how these participants use and encounter comparable scientific data and products
across the web. The second study (usability testing) will test new NCSES website designs
(a “prototype”), developed per an understanding of users’ needs. The results of both
1

studies will be used to develop a final user interface for the new website. This two-study
project is a follow-on activity to an earlier project (described in a generic clearance memo
dated December 20, 2019).
Recruitment
For both studies, NCSES plans to recruit participants that represent specific groups of
data users (see Table 1 on page 3). They will not be representative of the general
population. NCSES plans to recruit participants from the U.S. through targeted email and
social media messages without regard to geography. (See Attachments A, G, P, Q, R, and
S for the initial recruiting emails and social media messages.) To maximize response
rates as well as reduce the burden of filtering out potential participants who do not have
an interest in science and engineering statistics, for the most part, participants will be
targeted on the basis of their past interactions with information on the science and
engineering enterprise.
We will overrecruit participants in order to obtain a maximum of 21 participants for the
interviews, and a maximum of 14 participants for the usability tests (35 participants in
total). (This is detailed in Table 1 on page 3). Our sources of recruitment will be:
A list of approximately 50 individuals, including those who have contacted
NCSES survey managers for assistance and from NCSES contractors who have
been gathered through networks.
2. Subscribers to an NCSES email list. The list is approximately 21,230 people,
though the number fluctuates periodically. (Note, emails must be delivered to this
list in its entirety as it cannot be segmented.)
3. Posts on and direct recruitment through social media and a banner on the current
NCSES website.
1.

NCSES expects that response rates will vary drastically between groups; for example,
academics who use NCSES data will be very likely to respond, while casual information
seekers and members of the media will be less likely to respond. Participants will be
recruited from each source in order to minimize potential burden.
NCSES and its contractor will recruit participants primarily through email invitations,
messages to professional groups on social media, and messages to individuals through
social media channels. (Recruiting messages are included in Attachments A, G P, Q, R,
and S.) Names will be chosen from the lists to obtain a diverse pool on the basis of
institutional affiliation.

2

Table 1: Description of Research Participants for User Interviews and Usability Testing
Participant
Group

Source of Participants

How We Will Recognize Each
Participant Type

Government
Analysts

·

Social media groups

·

·

Visitors to the current
website

·

Contact lists from survey
managers of government
·
analysts

Media

Academia

·

Authors of papers or
other publications that
use NCSES data or other
science and engineering
(S&E) data

·

Contact lists from survey
managers of press
requests

·

Social media groups

·

Visitors to the current
website

·

Restricted use license
holders

·

Authors of papers or
other publications that
use NCSES data or other
S&E data

·

Contact lists from survey
managers of universitybased institutional
researchers interested in
academic benchmarking

·

Social media groups

·

Visitors to the current
website

Users employed by
Statistical Agencies,
Government Agencies,
Policy Agencies, internal
NSF e.g., survey owners

Target # of
Participants
for
Interviews

Target # of
Participants
for Usability
Testing

3

2

Users use data to prepare
papers, publications, or
analyses

·

User prepares documents
and uses data to support
journalism or writing for
public consumption (e.g.,
the topic is “education as a
path to citizenship” and will
use data on science and
engineering in the U.S.)

3

2

·

User is employed by a
university or is a student
and searches for data to
support academic use

3

2

·

Authors prepare papers,
publications, or analyses
that use NCSES or other
S&E data

3

Nonprofit
Organizations

Industry

Non-NCSES
Users/Casual
Information
Seekers

·

Contact lists from survey
managers of people with
nonprofit affiliations

·

Authors of papers or
other publications that
use NCSES data or other
S&E data

·

Contact list from survey
managers of people with
industry affiliations

·

Social media groups

·

Visitors to the current
website

·

·

·

User is employed by a
nonprofit agency

·

Authors prepare papers,
publications or analyses
that use NCSES or other
S&E data

·

User is employed by a forprofit, nongovernment,
non-statistical/policy
agency, non-NSF

·

Users use data to prepare
papers, publications, or
analyses

People registered for the ·
RSS feed (random
sample of non-.edu, non.gov, and non-foreign
·
addresses)

User may not be applying
S&E data in a professional
capacity

Contact lists from
program managers of
people who appear to be
casual information
seekers (i.e., they do not
fall into the above
categories)

·

Social media groups

·

Visitors to the current
website

·

3

2

3

2

6

4

User may or may not be
doing analysis of data, but
is using summaries and
descriptions to make
decisions
User could, for example, be
a mom looking for “the
best science and
engineering school for her
son”

Study Description
This project involves two studies:
1. Exploratory interviews with site users of current NCSES website.
This first study will use an exploratory and open-ended approach to learn how people use
(or might use) the NCSES website, its data and products, and how they use and encounter
scientific information across the web. Specifically, we seek to understand:
a. What sites participants rely on for data and analytic information about the science
and engineering enterprise.
4

b. How participants use science and engineering data and related analytic
information in their work.
c. When and why participants visit NCSES.
d. How the structure, content, and functionality of the NCSES website could be
more valuable to participants.
A maximum of 21 semi-structured interviews of 45 minutes each will be conducted over
a web-based videoconferencing platform. (The script is included in this package as
Attachment M.) Should we interview multiple people from one organization, we will
consider conducting them as group interviews.
Each interview will have two parts:
a. A guided discussion about the participant’s use of science and engineering data
and data analyses.
b. A series of tasks that participants will complete on the existing NCSES website.
An NCSES contractor will conduct the interviews and NCSES staff, if available, will
observe. NCSES expects to field the instrument in the spring and early summer of 2020.
The interviews will be conducted over a 3- to 4-week period.
Participants will be asked for their permission to record the sessions (audio and video)
and whether clips and quotes from their session can be included in the final report. They
will also be told that their participation in the research is voluntary, and they will be
informed of the OMB control number (3145-0174). No sensitive information will be
collected.
Once the interviews are complete, an NCSES contractor will review them and synthesize
the qualitative data to answer the research questions noted above.
2. Usability testing of a new NCSES website prototype.
For the second study, a maximum of 14 participants will test a prototyped user interface,
developed per an understanding of users’ needs and documented requirements. The
prototype will consist of static web designs that contain clickable “hot spots.” These hot
spot areas will link participants from one static design to another, thereby giving users the
feeling that they are having a live website experience.
Through the usability test, NCSES seeks to answer questions such as:
a. Do participants find the navigation of the newly proposed prototype easy to use?
a. Can they easily find and interact with data (e.g. download tables)?
b. Can they easily find analytical information about statistical data?
c. Do they find the new look and feel professional, clean, and appealing?
5

d. Do they see the layouts as an improvement over the current website?
e. How might the prototypes be improved further?
The user sessions will be guided, meaning an NCSES contractor will verbally provide
participants with tasks (Attachment N). Participants will be asked to carry out the task,
talking aloud as they go. Again, participants will be asked whether the session can be
recorded by video and audio and whether clips or quotes from their session can be
included in the final report.
An NCSES contractor will conduct the testing sessions and select NCSES staff, if
available, will observe. NCSES expects to field the instrument in the fall of 2020. The
interviews will be conducted over a 4-week period.
Once the user testing sessions are complete, an NCSES contractor will review them and
synthesize the qualitative data to answer the research questions noted above.
It should be noted that the prototypes will also be tested among a small number of
NCSES staff. They are not listed in the table above, as OMB approval is not required for
including them in the study.
Burden Information
For first study (user interviews), NCSES expects to invite a maximum
of 21,280 people with the goal of obtaining a maximum of responses from 21 users.
There is a possibility that participants will overlap with the previously approved generic
clearance project since we are using some of the same sources of recruitment. As such,
there may be some response fatigue. With this said, NCSES calculates the burden
information as follows: It is assumed that all of those solicited will read the invitation
(21,280 * 1 minute = 355 hours) but only 1% of those solicited (213 people) will respond
to the invitation by completing the screener (213 * 5 minutes = 18 hours). NCSES
expects the recruiting process for user interviews to result in approximately 373 hours of
burden (355 hours + 18 hours = 373 hours).
The estimated time for completion of the 21 user interviews is 20 hours, including time
for scheduling each respondent (21 * 55 minutes = 20 hours). Thus, the total burden
estimate for the first study is 393 hours (373 + 20).
For the second study (usability testing), NCSES expects to recruit a maximum of 14
participants. NCSES will invite those who completed interviews in the first study, and
those who completed the screener but were not selected to participate in an interview. If
14 usability sessions are not completed as a result of those invitations, NCSES will send
another solicitation to the listserv of 21,280 people. NCSES assumes that all of those
solicited will read the invitation (21,280 people * 1 minute = 355 hours) but based on
previous experience only .5% of those solicited (106 people) will respond to the
invitation by completing the screener. (106 * 5 minutes = 9 hours). This reduced
6

completion percentage (1% for the initial solicitation, but only .5% for this solicitation)
reflects the likelihood of response fatigue. NCSES expects this process to result in up
to approximately 364 hours of recruiting burden (355 hours + 9 hours = 364 hours).
The estimated time for completion of the usability study is 13 hours, including time for
scheduling each respondent (14 * 55 minutes = 13 hours). Thus, the total burden estimate
for the second study is 377 hours (364 + 13).
The total burden for both interviews and usability testing are estimated to be 770 hours
(393 + 377).
Incentive Payments
Amazon gift cards worth $40 will be sent to each user research and usability testing
participant as an honorarium upon completion of an interview or usability test
(employees of the U.S. Federal Government will not be eligible to receive a gift card).
The gift cards represent tokens of appreciation for the participants’ time and effort and
are not considered payment for their time as professionals.
Contact Information
The contact person for questions regarding this data collection is:
May Aydin, Ph.D.
Program Director
National Center for Science and Engineering Statistics
National Science Foundation
703-292-4977
maydin@nsf.gov

7

Attachments
Attachment A: Initial recruiting email for user interviews
Attachment B: Screener for all research participants
Attachment C: User interview scheduling email for eligible participants
Attachment D: Confirmation email for user interviews
Attachment E: Day-of reminder email for user interviews
Attachment F: Thank-you email for user interviews
Attachment G: Initial recruiting email for usability tests
Attachment H: Usability testing scheduling email for eligible participants
Attachment I: Confirmation email for usability test
Attachment J: Day-of email for usability test
Attachment K: Thank-you email for usability testing participants
Attachment L: Thank-you email for non-eligible participants
Attachment M: User needs assessment interview guide
Attachment N: Protocol for usability testing of NCSES website prototype
Attachment O: Interview consent form
Attachment P: Social Media Invitation, User Interviews (Group)
Attachment Q: Social Media Invitations, Usability Testing (Group)
Attachment R: Social Media Invitation, User Interviews (Email to Individual)
Attachment S: Social Media Invitations, Usability Testing (Email to Individual)

cc: May Aydin
Rebecca L. Morrison

8