Memo_ARP Survey_5-29-2013

Date: June 14, 2013

TO: Office of Management and Budget (OMB)

Through: Keith Tucker, Report Clearance Officer, HHS

Seleda Perryman, Project Clearance Officer, NIH

Vivian Horovitch-Kelley, PRA Liaison, NCI

From: Susan Scott, Public Health Advisor

Gordon Willis, Cognitive Psychologist

Division of Cancer Control and Population Sciences (DCCPS),

National Cancer Institute/NIH

Subject: Questionnaire Cognitive Interviewing and Pretesting (ARP/DCCPS/NCI)

OMB No. 0925-0589, Expiration Date 4/30/2014

Proposed Generic Sub-Study #7 “Survey Instrument Designed to Assess Users’ Perceptions of Website Changes”

(OMB No. 0925-0589-07)

This sub-study was originally approved on April 9, 2009 (OMB No. 0925-0589) however the sub-study was never conducted and the burden was not used. Now that this is a new approval period for the full generic, NCI is resubmitting this sub-study to OMB for approval.

The purpose of the proposed study is to conduct a usability test to assess user perceptions of specific design changes to an existing website. The instrument will allow users to make a very quick, direct assessment of the site and interaction design. Part of the reason for this is that existing survey instruments for measuring user satisfaction or usability of active websites require significant user effort to complete.

Background on Project

New website designs and website redesigns are being developed at the National Cancer Institute (NCI) on a regular basis. Typically in a user-centered process, users are interviewed, the existing design (if available) is assessed, and designers create a new design based on the available data. A usability test may be conducted on a prototype of the design, and changes are often recommended after this testing. Once the website is launched, feedback on the final design is rarely collected.

Several challenges effect the process of assessing post-launch websites. First, there is often lack of funding necessary to design a specialized instrument. Second, actual users are often not known, particularly when the site is a public-facing site as many of the NCI sites are. Third, sample size must be sufficient for analysis.

There are several existing survey instruments that have been used to measure user satisfaction or usability of active websites. For instance, The ACSI (American Customer Satisfaction) survey has been used by government agencies for this purpose. Standardized survey instruments such as the SUS (System Usability Scale), QUIS (Questionnaire for User Interaction Satisfaction), SUMI (Software Usability Measurement Inventory) and WAMMI (Website analysis and Measurement Inventory) exist for assessing designs.

Typically, however, these instruments are time-consuming for users to complete, resulting in low completion rates when participation is voluntary. In order to ensure a response rate high enough to allow conclusions to be drawn, the survey should be brief, for instance 4-5 questions in length, so that a user could complete it in two to three minutes. To our knowledge, such a survey is not in common usage.

A pop up survey would be a practical way to obtain user feedback from a large audience of users.

Pop Up Surveys

Pop up surveys appear as a new browser window when users are accessing a website. Users are asked whether they would like to participate in a brief survey, and if they agree, the survey questions appear in the window. If users decline to participate, the window closes, and users return to the website. Advantages of this methodology include direct targeting to actual users who are visiting the website and operation while the site design is fresh in the participant’s mind.

Proposed Research

The proposed study, “Survey Instrument Designed to Assess Users’ Perceptions of Website Changes,” is to conduct a usability test to assess user perceptions of specific design changes to an existing website. The survey instrument will allow current users who are familiar with both the old and new site design to provide a very quick, direct assessment of the design changes. All contact with users will be through the pop-up survey; at no time will there be any verbal probing of users. The survey questions will need to address whether the new website better meets the needs of the users in terms of finding and using information (Refer to Attachment 1).

The website for the DCCPS Applied Research Program (ARP) (see Figure 1) is currently being redesigned and developed using user-centered design principles, which provides an opportunity to test the new survey.

Figure 1: Applied Research Website, current design

For this specific use, results of the survey will allow the NCI to better understand user perceptions of usefulness and usability of the new presentation of Applied Research Program information. It is planned that visitors to the new ARP website will be invited to participate in the study until 100 persons have completed the survey. Survey participants will be anonymous visitors to the ARP website and no personally identifiable information will be collected. Incentives will not be offered.

In the larger sense, the survey will provide NCI with a new tool to use in assessing redesigned websites.

Estimates of Hour Burden and Respondent Cost
Types of Respondents	Number of Respondents	Frequency of Response	Average Time Per Response (in hours)	Annual Hour Burden (Hours)
Website users	100	1	3/60	5

Thank you for your consideration of this proposed sub-study #7.

Attachment 1: Screenshots of Pop-Up Survey