Form 1 RTIPs Usability Interview Guide

Confidential and Proprietary January 14, 2021

OMB No.: 0925-0642

Expiration Date: 05/31/2020

Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0642). Do not return the completed form to this address.

RTIPs Usability Test Plan

Overview

The National Cancer Institute’s (NCI) Research-Tested Intervention Programs (RTIPs) website is a searchable database of evidence-based cancer control programs that provides program planners and public health practitioners easy and immediate access to: 1) programs tested in a research study, 2) publication(s) of the study findings, and 3) program products or materials used with a particular study population in a specific setting.

Improvements to the user experience of the RTIPs website will help the site serve its target audience groups. This document describes our test plan for conducting a usability test for https://rtips.cancer.gov.

Purpose

The purpose of the user testing is to test features and tools of the RTIPs website in order to understand how program planners and public health practitioners make decisions to adopt and implement evidence-based programs for use in their practice settings.

ICF will conduct usability testing with participants in order to:

• Gain better understanding of user goals and needs

• See where the web design is successful and unsuccessful

• Inform future enhancements

Research Questions

We are interested in gathering feedback on the following:

• General website usability and design:

• How clearly and logically the information is organized

• Usability of navigation tools

• Homepage information organization

• For target audience groups:

• Usability of program search (i.e. search filters and paths)

• Usability of program details (i.e. scoring system, implementation guides)
  
  
  

Research Method

To gather feedback about how users interact with the RTIPs website, ICF will conduct user testing using two sets of clickable wireframes. Participants will be equally and randomly divided into two groups and each group will be exposed to one set of the wireframes. The tests will consist of one-on-one interviews over the internet/conference call with test participants, while screen sharing each test participant’s computer. Each session will include one moderator, a note-taker and interview recorder, and the participant. Other members of the project teams from NCI and ICF will be able to join the sessions through the conference call links and observe the tests. Each test session will last approximately 60 minutes and will be broken into these sections:

• Background questions (for participants recruited outside of the Coffee Talk interviews conducted in the first phase):

• What is the participant’s job/role?

• What are some typical research resources and services they use?

• How often do they search for or use evidence-based research programs?

• What are the key factors users look for when choosing a research program?

• What do they expect research resource websites to provide?

• What are some typical challenges they encounter when searching for or implementing evidence-based research programs?

• Main website tasks:

• Does the title of the site match user expectations for what they’re looking for?

• What are the participant’s overall impressions?

• How does the participant go about looking for and finding information?

• How does the participant go about completing common tasks?

• Close out:

• Overall reactions

• Any other feedback

Task assignments may be slightly adjusted based on the following screening criteria gathered through participants’ self-introductions:

• Frequency of research program needs

• General demographic information (i.e. research focuses or geographical areas)

• Typical challenges known when looking for evidence-based research programs

Note: ICF would like to conduct a minimum of two practice sessions internally before the actual tests begin. At least one session will be conducted with NCI’s participation.

Test Materials and Software

In order to gather feedback on multiple design ideas, we will use the following two sets of wireframes developed by NCI contractors:

• The first set highlights enhanced search filters and content layout categorized by product type:

  • Home page (CALIBRE)

  • Search results page (ICF)

  • Program details page (ICF)

  • Program materials page (ICF)

• The second set highlights advanced search filters with a revised presentation of program details and materials, categorized by target audience:

  • Home page (CALIBRE)

  • Search results page (CALIBRE)

  • Program details page (CALIBRE)

  • Program materials page (ICF)

Equipment:

• Participants will be using their own computers

• Moderators will be using their own computers

Software:

• Screensharing and recording software: GoToMeeting (a separate web link will be generated for each session)

• Wireframing software: Axure RP / Axure Share

Participants

We will engage with approximately 9-20 test participants to gather their feedback. NCI will identify and schedule sessions with the participants.

Participants will meet the following criteria:

Demographics:

• Mix of people who work for public health practitioners, nonprofit public health organizations, and state or local health agencies/government

• Mix of genders, ages

User behavior:

• Responsible for researching cancer control interventions or other public health programs as part of their job

Internet knowledge:

• Mix of people who are familiar with NCI and RTIPs

• Mix of people who use desktop, tablet, or mobile device to access the internet

Final Report

A brief summary report will highlight the outcome of the user tests. The summary report will include an updated version of the wireframes that were tested with participants. The wireframes will be iterated on between tests based on user feedback to show new versions to the next group of users. The resulting wireframes will be included in the final design plan.

Schedule

• Testing time frame: February 2020

• No more than five test sessions per day

• Each session will be scheduled to last approximately 60 minutes

• Schedule at least 30 minutes between sessions

• Sessions will take place between 9 a.m.-6 p.m.

RTIPs User Testing Script – Moderator’s Guide for Interviews

Introduction

“Thanks for joining me today. My name is [INSERT NAME] with ICF. I am working with NCI to learn how program planners and public health practitioners search and implement evidence-based research programs. This conversation should be about an hour. Please feel free to ask questions at any time.

We will be looking at a draft website design to get your honest feedback. This is NOT the final design. Some areas are not fully working. As I ask you to review the pages, please think aloud. It will help me understand your thought process. Your answers will not be shared outside of our team.

The discussion today is not a test of you or your abilities. There are no right or wrong answers. You don’t have to worry about saying the wrong things.

There are a few of my colleagues listening in to observe and to help with notes. But I will be doing most of the talking. Also, to accurately capture our notes, we would like to record the entire session. Your responses will be kept private to the extent allowable by law and will only be used for the purpose of this research. Do we have your permission to record?

Thank you. Any questions for me before we begin?”

Section 1: Background Questions

• Can you tell me a little bit about your role?

• Have you heard of or used the Research-Tested Intervention Programs (RTIPs) website?

[IF YES] What do you typically use it for?

How often do you visit?

• Think of a time you had to find something on the website, was it easy to find?

  • If you were looking for a program, were you successful? Did it contain all the information you need? Anything missing?

• On a scale of 1 to 5, with 1 being very difficult to use and 5 being very easy to use, how would you rank the site?

[IF NO] Why not?

• What information do you think you could find on the Research-Tested Intervention Programs (RTIPs) website?

• What would motivate you to use this website?

• How do you typically look for research programs or relevant resources?

• What are some typical health care websites and resources that you use?

  • How often do you use these resources?

• What do you expect such research resource websites to provide?

• What are some typical challenges that you encounter when searching or implementing a program?

Section 2: Task Scenarios

Now we are going to switch to the tasks for you to complete today. We would like to view your screen while we have you look at a webpage. As you share, we will see everything on your screen – so make sure you have anything closed that you would like to keep private, such as email.

My colleague [Notetaker], will now make you the presenter – you should see a button that says you can share your screen.

[SHARE SCREEN NOW]

Great, we can see your screen now. Let’s bring up a browser. I’m going to send you a link now through the chat – it is to the website we are going to review today.

[LINKS TBD]

When you get the link, you can scroll up and down, but please don’t click on anything yet.

The link we will be looking at is a prototype – this means it is like a drawing that has some things you can click on, and not everything works right now. Your feedback may help shape its development. There are some links that do not work, and you may see placeholder text or images. I’m asking you to use your imagination.

• What are your first reactions to this page?

• What are some things you could accomplish using this site?

• Thinking of the name of the site – Research-Tested Intervention Programs – does that title make sense to you?

Now I will read a few hypothetical scenarios to you with questions and would like you to show me where you would go on the site to look for certain information. There are no right or wrong answers or clicks. I would just like to see how you personally use the site I am showing you.

Task Scenario 1: Program Search

Let’s look at an area where you would typically search for a new program. Consider this scenario:

[IF starting with home page] In a recent collaboration with your local cancer center, you would like to find information about nutrition programs to help recently diagnosed cancer patients handle stress and improve healthy eating. How would you conduct a search on this site?

[IF starting with program details page] In a recent collaboration with your local cancer center, you would like to find information about nutrition programs to help recently diagnosed cancer patients handle stress and improve healthy eating. You did a Google search and found this page. But it’s not what you were looking for. How would you go from here to conduct a new search on this site?

Probing questions (if needed):

• What would be a typical search term that you use?

• If it was difficult to accomplish, why?

• Were any of the links confusing? What links were confusing and why?

• What would have made it easier to complete the task?

Notetaker:

• Did the participant give up?

• Was this task easy or difficult for the participant?

  • Very Easy 1-7 Very Difficult

Task Scenario 2: Program Details

You found a program of interest on colorectal cancer prevention and would like to know more. How do you determine if it’s a good fit for your organization? Is the information here sufficient to help you make a decision?

Probing questions (if needed):

• If it was difficult to accomplish, why?

• Were any of the links confusing? What links were confusing and why?

• What would have made it easier to complete the task?

Notetaker:

• Did the participant give up?

• Was this task easy or difficult for the participant?

  • Very Easy 1-7 Very Difficult

Task Scenario 3: Program comparison

For a research grant through your local health department, you are creating a community-based nutrition program for healthy eating from scratch and have found useful information on RTIPs. How do you compare programs you found and narrow down your choices?

Probing questions (if needed):

• If it was difficult to accomplish, why?

• Were any of the links confusing? What links were confusing and why?

• What would have made it easier to complete the task?

• Now that you’ve identified a few relevant programs, what would you do afterwards? Would you print this out to compare offline? Would you share with your colleagues or supervisor?

Notetaker:

• Did the participant give up?

• Was this task easy or difficult for the participant?

  • Very Easy 1-7 Very Difficult

Task Scenario 4: Program implementation

Once a program is identified, how do you find strategies, tips, or guidelines for implementation?

Probing questions (if needed):

• If it was difficult to accomplish, why?

• Were any of the links confusing? What links were confusing and why?

• What would have made it easier to complete the task?

Notetaker:

• Did the participant give up?

• Was this task easy or difficult for the participant?

  • Very Easy 1-7 Very Difficult

Task Scenario 5: Program Materials and Product Layout

Once a program is identified, how do you find relevant materials available on the site to download?

Probing questions (if needed):

• If you are using the materials to implement the program, where would you look?

• Is it clear which materials are for delivering the intervention?

• Is it clear which materials are for the participants?

Notetaker:

• Did the participant give up?

• Was this task easy or difficult for the participant?

  • Very Easy 1-7 Very Difficult

Section 3: Final Wrap-Up

I only have a few more questions to finish.

1. What was your overall impression of the pages we reviewed today?

2. Thinking of the name of the site again – Research-Tested Intervention Programs – does that title make sense to you?

3. How would you rank these pages based on how easy it was to use, from 1 to 7 (using a scale from 1 to 7 where 1 is very easy to use and 7 is very difficult to use)?

4. Did the pages have everything you were expecting to see?

5. Were any of the links on the pages confusing to you?

6. Do you have any other feedback?

7. Does anyone else on the call have any additional questions?

This ends our session. Thank you for your time. Your feedback is very valuable to us.

10