Protocol

Annual Business Survey Usability Protocol

4/2/24

Contents

Annual Business Survey Usability Protocol 1

Introduction 1

Participant Background 2

Question Testing 2

Wrap Up 6

Introduction

Thank you for your time today. My name is XX and I work for the United States Census Bureau. My research team evaluates how easy or difficult Census surveys are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

[Confirm they signed the Consent Form sent prior to interview.]

Thank you for signing the consent form, I just want to reiterate that neither your name nor your company name will be mentioned in our final report. We plan to use your feedback to improve the design of this survey instrument and make sure it makes sense to respondents like you.

Just to note, I did not create the instrument so please feel free to share your thoughts, both positive and negative.

I am going to give you a little background about what we will be working on today. Today, you will be helping us to evaluate one section of the Annual Business Survey. We are interested in obtaining feedback to ensure that it has all of the information and features needed and to assess how easy or difficult these features are to work with.

To do this, I’m going to ask you to screenshare with me.

Now I would like you to go through one part of the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. I will provide you data to fill in as needed. Please feel free to mention if you would need to get information from records or others within your organization in order to answer a question.

Just a reminder, you won’t need to fill out any data from your records. I will ask you additional questions about some of the screens you see today and your overall impressions.

Before we begin:

Participant Background

• Can you give me a brief description of your nonprofit?

  • What kinds of research and development work is your organization engaged in?

• What is your title/role?

  • How long have you been with the organization/in your current position?

Now let’s pull up the website: [LINK TO ABS TEST SITE]

Before you log in, can I have you share your screen with me? Feel free to take a moment to minimize anything, I only need to look at the website. It’s a little different depending on your computer but the icon should look like an arrow in a box, usually next to the mic button.

[Instruct how to screenshare]

[Provide dummy id and password]

Do you have any questions before we begin? Ok let’s get started.

Question Testing

Main Menu

• We’re going to jump you to Section E, which is why the menu reads ‘resume’, and not ‘start’, as it normally would. Let’s hit resume to get started.

Mailing Address

• Please fill out Name1. Street City State Zip

• Feel free to enter in fake data for any items you don’t know off the top of your head.

Section E: Research Activities at Nonprofit Organizations

• Do you have any thoughts about these instructions?

E.1 Nonprofit Organizations: Organization Affiliation

• Okay, let’s answer this question how you actually would.

E.2 Nonprofit Organizations: Total Expenses

• You want to indicate $300,000 here, how would you do that?

E.3 Nonprofit Organizations: Number of Employees

• Please enter 300 here.

E.4 Nonprofit Organizations: Fiscal Year End

• You may enter any month here.

Nonprofit Organizations: R&D Description

• Do you have any comments on this screen?

E.5 Nonprofit Organizations: Research Performed

• Please say yes here. During the actual survey, how likely would you be to type something into the text box?

E.6 Nonprofit Organizations: Research by Others

• Please answer this accurately.

E.7 Nonprofit Organizations: Funding for Research

• Let’s assume here you actually have this number on hand, and the answer is 160,321.

E.8 Nonprofit Organizations:

Take a moment and read over this list,

• You will enter 50k for the first two rows, and 10k for the rest. There were no funds acquired from a foreign entity.

• Foreign funds column: how would you indicate that no funds came from a foreign entity? (Entering zero versus leaving cell blank)

• Do you have any thoughts about there being two separate totals (one column and one row)?

E.9 Nonprofit Organizations: Research Performance by Field

• Let’s assume half your spending is allocated to a. Agricultural Sciences and Natural Resources and Conservation (80k), and half to f. Physical Sciences (80k)

E.10 Nonprofit Organizations: Clinical Trials

• Let’s enter 0% for this question.

E.11 Nonprofit Organizations: Type of Research

• Take a moment and read over this screen. Do you have any feedback on the design of this page?

• Let’s split the 160,000 between federal and non-federal in the Basic Research category.

E.12 Nonprofit Organizations: Type of Cost

• Let’s split the amount between the first two categories, (Salaries = 80k; Contract costs = 80k)

E.13 Nonprofit Organizations: Research Employee Type

• We’ll say the staff (150) are split evenly between these categories (Researchers, 50; Technicians, 50; Support, 50)

• If you realized this number was inaccurate and wanted to go back to adjust the number of employees, how would you do that?

E.14 Nonprofit Organizations: Full-time Equivalents

• For this question, all the employees we just listed are full time (Researchers, 50; Technicians, 50; Support, 50)

E.15 Nonprofit Organizations: Contract Employees

• Let’s say no here.

E.16 Nonprofit Organizations: Volunteers

• And no again.

E.17 Nonprofit Organizations: Research by Other Organizations

• Let’s answer none for these categories.

E.18 Nonprofit Organizations: Total Capital Expenditures

• For capital expenditures, let’s enter $10,000.

E.19 Nonprofit Organizations: Research Capital Expenditures

• For this question we can half the cap ex to be $5000.

• How would you respond if your research cap ex was less than $1000?

Edit Messages

[If the R has not yet triggered an edit, have them do so on purpose]

• Let’s enter $20,000 here.

• Do the messages on the screen clearly indicate to you that there is an issue, or are they not clear? Can you explain to me what the problem(s) is/are?

• Is there a better way that we could notify you about the problem(s) on this screen?

E.20 Nonprofit Organizations: Capital Expenditure by Type of Cost

• Take a moment and read over this last question. Let’s put the $5,000 into the first category, a. Land acquisition.

[Don’t have R click to next screen yet]

Other Items: Save

• If you wanted or needed to save your work and return to it later, how would you go about doing so?

Wrap Up

• Now that you have gotten a good feel for this portion of the online form, what is your overall impression of it? Was it easy, difficult, or somewhere in between?

• Did you have any issues with the navigation on the site?

• Did anything stand out as particularly easy or difficult? Especially as it relates to the usability of the site?

  • Do you have any suggestions for how to make this online form easier for you to use?

Thank you for your time and feedback!