NCSES Data Tools and Product Usability Testing

Attachment A
NCSES Data Tools and Product Usability Testing
Usability Testing Draft Testing Plan/Script

We expect to conduct 10–20 tests, testing each data product during a minimum of two separate
tests. At least two products or tools will be focused on during each test. We will execute the
following cross-product test plan for all NCSES data products and tools. The consent form
(contained in Attachment C) will be emailed to the participant before the test and they will be asked
to sign it and return via email. If they are unable to do this, first the participant will be asked if they
can be recorded, then the recording will start and then the participant will confirm they have
consented to the recording and have read and agree to the provided statement of consent.

Cross-Product Test Plan/Script

Facilitating the test session:
1. Begin the session.
a. Ensure the participant understands the session will be recorded (as voice audio, and
video of the screencast), then begin recording. If verbal consent is sought, ask
participant to confirm they consent to recording and that they read and agree to the
provided consent after recording has started.
b. Take notes throughout the session labeled with the participant’s name,
organization, and job title if available.
2. Introduce the scope and goal of this project. Gather information on user.
a. “We are performing usability research on NCSES’s web-based data products and
tools, not you as a user. At times, it may feel like we are testing you, but we are
testing the website. There are no right or wrong answers to these questions. Based
on this research, we will provide recommendations for future improvements to
these products. We’d appreciate hearing any thoughts you have for improvements
as well.”
b. “Do you have any questions before we get started?”
c. “Please tell us about your job responsibilities, a bit generally and specifically how
they relate to science and engineering statistics.”
d. “Which of the NCSES products or tools have you used in the past?” List them if the
user is unfamiliar with the available tools. (Academic Institution Profiles, State
Profiles, SESTAT, WebCASPAR, SED Tabulation Engine, eTables, Public Use Files).
Explain in greater detail as applicable to users who are unfamiliar with these tools.
e. “How often do you use the product(s), if ever? What information are you typically
looking for when you get to the NCSES website? How did/do you decide which
products/tools to use?
f. “Can you tell us about your skill-level with data analysis?” (probe – more than Excel?
Which packages? Use with NCSES data? Not even Excel? Why?)

3.

4.

5.

6.

7.

g. “What other data tools do you use online?” (Census.gov, BLS, others?)
Begin the screenshare.
a. “In order to see how you use the tools, I’m going to be able to watch your computer
screen by passing you the ‘Presenter’ mode in GoToMeeting. Just click ‘Show My
Screen’ when you see the dialog.”
b. Confirm that the participant is sharing their screen.
Present task
a. “Now I’m going to ask you to perform a task. I’ll watch you perform the task on your
screen. While you work, please talk me through what you are doing. Please
elaborate on your thought process, anything you find confusing, and features you
like or dislike as you go. We’re also interested to hear if you have ideas for
improvement”
b. Read an appropriate task from the task list based on the users’ participant group
and skill level or functionality we want to test. (In addition, the task will be provided
in the GoToMeeting chat window so that the user is able to read and review the task
on their own.)
c. If the user has no idea which tool to use, or suggests a tool that does not contain the
correct data for the task, suggest the appropriate tool.
Launching the tool
a. Ask the user to access the tool using their web browser. If the user is unable to open
the product, provide the user with a link through the GoToMeeting chat window.
i.
Does the user have trouble opening the product?
ii.
Does the tool load promptly, or take a while?
b. “Would you ever use the ‘Help’ or ‘FAQ’ for this product?”
c. “Can you find ‘help’ or ‘FAQ’ on the screen?”
Watch the user perform the task.
a. “Can you please ‘walk me through’ the steps as you are performing this task?”
i.
He or she should elaborate on any likes, dislikes, or ideas for improvement
as they go.
ii.
If the user isn’t very talkative, ask them what they are doing, and have them
explain why they take certain steps, their anticipated results, and so on.
iii.
If the participant is unable to perform the task in a timely manner, the
facilitator will ask them to move on to another task.
iv.
When it’s time to move onto the next task, ask the participant, “Do you have
any comments about this task before we move on?” At this time, invite
comments on particular frustrations, ideas for improvement, likes and
dislikes, etc.
For each tool covered ask how satisfied they were with the tool using the following scale:
1, very dissatisfied
2, somewhat dissatistfied
3, neither dissatisfied
4, somewhat satisfied

5, very satisfied
(Paste scale in chat window.)
8. Repeat steps 4–7 with as many tasks as time allows varying tools and products as
appropriate.
9. Closing the session.
a. “We are approaching the end of our scheduled time. Are there any particular tasks
you’d like to walk through with me before we close, or particular frustrations you’d
like to share?”
b. “Do you have any other comments?”
c. “Once I close the meeting, the screenshare and recording will end automatically.”
d. Thank the participant and tell them how much we appreciate their time and how
helpful it has been to have their observations to inform our forthcoming
recommendations.
e. Say goodbye and end the session.

Tasks

Tool

Question(s)

Academic
Profiles

How many full-time science,
engineering, and health graduate
students are at Syracuse University? (in
the most recent year for which data is
available)

Academic
Profiles

Where did Georgia Tech rank in terms
of number of earned doctorates in
2013? Where did this data come from?

Academic
Profiles/We
bCASPAR

What were the total R&D expenditures
at Georgetown University in 2010 and
2006?

Academic
Profiles

What was the 2009 ranking of
institutions by total federal obligations?

State
Profiles

How does Oregon compare to
Washington in SBIR awards?

State
Profiles

State
Profiles

What percentage of the total 2013
expenditures is for academic R&D in
used for engineering research in
Arkansas and how does this compare to
Louisiana?
Can you download the Virginia state
profile in its entirety?

State
Profiles

Can you download all the state profiles
from 2003–4?

Associated Instructions
Navigate to AIP page
Click List of Institutions
CTRL+F search for Syracuse
Click link
Scroll down to and click link for Full-Time Grad Students in SE&H
Download table in Excel format. Open in excel.
Navigate to AIP page
Under the rankings heading, click Number of Earned Doctorates link
CTRL+F search for Georgia Institute of Technology
Scroll to bottom of page to find source information
Navigate to AIP page
Click List of Institutions
CTRL+F search for Georgetown
Scroll down to R&D expenditures by field and click link
Locate value for 2010 and 2006
Navigate to AIP page
Under the rankings heading click link for Federal Obligations
Find values for 2009
Sort by rank column for 2009
Navigate to State Profiles page
Select Washington or Oregon on map
Add the other state to the profile
CTRL+F search for SBIR
Compare values
Navigate to State Profiles page
Select Arkansas from the map
Add Louisiana to the profile
Scroll to Academic R&D expenditures and expand
Compare values for Engineering
Navigate to State Profiles page
Select Virginia on the map
Click Download the Table link
Navigate to State Profiles page
Click Previous Profiles tab
Click link for 03-04
Click download link

Functionality
Investigating
Find/view Individual
Institutions;
Download

Skill/knowledge
level
Low

Find/View individual
institutions; Link to
data sources

Low

Find/View; Link

Low

Find/view rankings;
sort by year;
download by sorted
year; Link to data
source
Find/view; Filter (add
state); link to SBIR

Medium

Find/view (map
chart); filter (add
state to chart); link
to help

Low

Find; download

Low

Find

Low-Medium

Medium

WebCASPAR

How many research doctorates were
awarded in broad academic fields in
each state by institution in 2009?

WebCASPAR

Which agencies have more than $2
million in federal obligations (in
constant dollars) during the 2000 to
2005 period? What is the top value for
the Department of Energy (DOE) during
that time?

WebCASPAR

What was the relationship between the
amount of federally-financed R&D
obligations at an institution and the
number of full-time graduate students
enrolled there in 2013?
How many graduate students were
enrolled at the top 10 institutions by
federally-financed R&D obligations that
year?
Which institutions are constructing the
most new engineering space?

WebCASPAR

WebCASPAR

How many postdocs of each gender
were there in each state in the most
recent year for which data is available?

WebCASPAR

What federal agency funded the most
R&D Plant (Facilities and Mixed
Equipment) by percentage in 1999?

Navigate to WebCASPAR home
Select Survey of Earned Doctorates as data source
Select Number of Doctorates Awarded by Doctoral Institution as
analysis variable
Click Modify Classification Variable
Select State as a Classification Variable
Select Academic Discipline (broad) as a Classification variable
Under Year, change year to 2009
Click View Table
Navigate to WebCASPAR home
Select Survey of Federal Funds as the data source
Select R&D as Analysis variable
Select Agency (standardized) as CV
Select years 2000-2005
Modify Table and move agency CV to column
View table
Click Filter AV link and select AV > 2000
Click DOE column heading and select sort descending
Navigate to WebCASPAR home
Select Survey of Federal S&E Support… and Survey of Grad Students
and Postdocs as data sources
Select Federal Obligations for R&D and Number of Graduate Students
as AVs
Select Academic Institution as CV
Select year 2013
View Table
Sort Descending on Federal Obligations for R&D column
Navigate to WebCASPAR home
Go to the Frequently Requested Tables dropdown
Select New construction of S&E space… Planned
Sort descending on the Engineering column
Navigate to WebCASPAR home
Select Survey of Grad Students and Postdocs as data source
Select Number of Postdocs as AV
Select State and Gender as CVs
Format table and move gender to column
View table
Navigate to WebCASPAR home
Select NSF Survey of Fed S&E Support… as data source
Select Federal Obligations for R&D Plant as AV and set Statistics:

Find data source,
modify classification
variables, download
table

Medium-High

Find data source,
modify format,
top/bottom row
selector, filter
variable

Medium

Finding and adding
two different survey
sources, modifying
classification
variable, selecting
row

High

Saved table; Sort by
column; access help

Medium

Find/view variable;
add classification
variables, sort

Medium

Find/view variable;
add classification
variables, sort

Medium

SESTAT/SEST
AT
Metadata
Explorer

Does the NSF publish data on the
annual salary of engineers?

SESTAT/SEST
AT
Metadata
Explorer

Does the NSF publish data on the job
status of scientists and engineers?

SESTAT/SEST
AT
Metadata
Explorer
SESTAT

What are the publically available
variables in the Scientists and
Engineers Statistical Data Systems?

SESTAT

How has the annual salary changed
from 1993 to 2003 for all engineers?

How does the job status of physical
scientists differ between age groups?

Percent of Sum
Select Federal Agency (standardized) as CV
Select year 1999
View table
Navigate to SESTAT Tool
Use National Survey of College Graduates
Row Variable for Major type
M_ED_MR_MAJOR_ED_GRP_MAJOR_NEW works
Under population conditions select variable for employed
A_JOB_STATUS_EMPLOYED_IND works for this
Select Median salary as Data type
Click Submit Table Request
Navigate to SESTAT tool
Use NSCG again
Row Variable is A_JOB_STATUS_EMPLOYED_IND
Under population conditions, select variable to limit to engineers
M_ED_MAJOR_ED_GRP_MAJOR_NEW works
Select Data type NUMBER
Submit table request
Navigate to SESTAT tool
Click the SESTAT Metadata Explorer button
Navigate to SESTAT tool
Use NSCG
Select A_JOB_STATUS_EMPLOYED_IND and
A_JOB_STATUS_LABOR_FORCE_STATUS as column variables
Select U_DEM_AGE_GROUP_5_YR_GROUPING as row variable
Select M_ED_MAJOR_ED_GRP_MAJOR_NEW = 3 for population
condition
Select Data type NUMBER
Submit table request
Navigate to SESTAT tool
Use NSCG
Select 2003 for year
Use A_JOB_STATUS_EMPLOYED_IND for column variable
Select M_ED_MAJOR_GRP_MAJOR_NEW = 5 for population condition
Select Data Type Median of H_JOB_SALARY_ANN
Submit Table Request
Repeat with 1993 selected for year

Find/view

Medium

Find/view

High

Find/view

High

Find/view variable
(needs to change
description view),
using weighted data

Medium-High

Using help/variable
search; find/view
survey year, modify
variables, filter
population, generate
table, using weighted
data

Very High

SESTAT

How many respondents to the 2013
Survey of Earned Doctorates indicated
they have a teaching faculty academic
position and how many held an adjunct
position? How did the median salary
differ between respondents?

SESTAT

According to the 2010 National Survey
for Recent College Graduate what
percentage of respondents who were
not born in the US are US citizens?
What percentage are not US citizens?

SESTAT

In 2010, what as the minimum, median,
and maximum salary of different types
of Ph.Ds.?

SED
Tabulation
Engine

How many new 2008 life science
doctorate recipients self identified their
race as Native Hawaiian or Pacific
Islander?

SED
Tabulation
Engine

How many temporary residents obtain
civil engineering doctorates from very
high research universities (according to
Carnegie classification)?

SED
Tabulation
Engine

How many women graduated from
Harvard University in 2008 with a
research doctorate in engineering?

Navigate to SESTAT tool
Use Survey of Doctorate Recipients as data source
Use E_JOB_EMPLR_ACAD_POSITION_ADJUNCT as column variable
Use E_JOB_EMPLR_ACAD_POSITION_TEACH_FACULTY as row variable
Use Data type Median of H_JOB_SALARY_ANN
Submit table request
Look at cells where No and Yes columns/rows intersect
Navigate to SESTAT tool
Use Recent College Graduates NSRCG 2010 as data source
Select U_DEM_BIRTH_PLACE_REGION_US_NONUS as column variable
Select V_DEM_CITIZENSHIP_US_CITIZEN as row variable
Select Data type NUMBER (count)
Choose % option "Include % down the column"
Submit table request
Navigate to SESTAT tool
Use SDR as data source
Select M_ED_MAJOR_GRP_MAJOR_NEW as row variable
Select A_JOB_STATUS_EMPLOYED_IND = YES for population condition
Select Data type Median, Minimum and Maximum of
H_JOB_SALARY_ANN
Submit table request
Navigate to SED tool
Outer Row = Year
Inner Row = Academic Discipline, Broad
Column = Race/Ethnicity
Filter = Discipline, Broad - Life Sciences
Execute
Navigate to SED tool
Outer Row = Carnegie 2005
Inner Row = Discipline, Detailed
Column = Citizenship
Filter = Discipline, Detailed - Civil Engineering
Execute
Navigate to SED tool
Outer Row = Year
Inner Row = Discipline, Broad
Column = Gender
Filter = Institution - Harvard University
Execute

Find survey year, find
and select variable
(position), add
salary, unweight
data

High

Find survey, year,
select variables,
modify data type,
table generator

High

Find survey, year,
select variables (2),

High

Find, modify, Help
(variable definition);
Filter; download

High-very high
(Guam = Pacific
Islander)

Modify, Download

Low

Modify, Filter, Search

Medium

eTables

eTables
eTables
eTables

Public use
files
Public use
files
Public use
files
Public use
files

What is the 2013 median salary of
employed U.S. scientists and engineers
with a masters and work in computer
applications?
What were the total R&D expenditures
for life sciences and its subfields in
2013?
How many people with doctorates in
an S&E field were unemployed?
When will data next be released from
the Survey of Graduate Students and
Postdoctorates in Science and
Engineering?
Is the GSS (Survey of Graduate
Students and Postdoctorates in Science
and Engineering) public use microdata
available for download?
Is the GSS (Survey of Graduate
Students and Postdoctorates in Science
and Engineering) public use microdata
available for SAS?
What does the variable "_forgn_v"
represent?
What variable(s) gives the number of
full-time graduate student supported
by a Department of Defense
fellowship?

*NOTE - No gender specific results were given for Engineering
doctorates at Harvard in 2008*
Navigate to Recent Data Tables listing page
SESTAT 2013 data table
Look for table 12-3

Find

Low

Navigate to Recent Data Tables listing page
HERD Data Table
Table 5
Navigate to Recent Data Tables listing page
SDR Data Table
Table 1
Navigate to Recent Data Tables listing page
Click schedule of next releases link in the left hand navigation menu

Find/download

Low

Find, download

Low-Medium

Help

Low

Navigate to Public Use Files
Click link to Survey of Graduate Students and Postdoctorates in S&E
Select a download link

Find/view

Low

See above

Find/view; Download

Medium-High

Navigate to Public Use Files
Click link to Survey of Graduate Students and Postdoctorates in S&E
Click Appendix C link
Answer is on page 1 of PDF
Navigate to Public Use Files
Click link to Survey of Graduate Students and Postdoctorates in S&E
Click Appendix C link
Answer is on Page 7 of PDF

Find documentation

High

Download
documentation

High