Supporting Statement B For:
Cancer Trials Support Unit (CTSU)
Public Use Forms and Customer Satisfaction Surveys (NCI)
OMB No. 0925-0624, Expiration Date: 12/31/2013
October 25, 2013
Revisions to the original submission are in yellow
throughout this document.
Michael Montello, Pharm. D.
Cancer Therapy Evaluation Program (CTEP)
Division of Cancer Treatment and Diagnosis (DCTD)
National Cancer Institute
9609 Medical Center Drive
Rockville, MD 20850
Phone: (240) 276-6080
Fax: (240) 276-7892
B. STATISTICAL METHODS 1
B.1 Respondent Universe and Sampling Methods 1
B.2 Procedures for the Collection of Information 2
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 5
B.4 Test of Procedures or Methods to be Utilized 5
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5
The Cancer Therapy Support Unit (CTSU) and Protocol and Information Office (PIO), all part of the Cancer Therapy Evaluation Program (CTEP), collects four satisfaction surveys to objectively measure customer satisfaction of select applications and services. Simple descriptive statistics are used for analysis of data.
The surveys target users of CTSU and PIO system/services within the recent past. Approximately 600 persons are randomly sampled for the web site and help desk surveys from a larger pool of potential respondents in 2012. For the most recent help desk survey (attachment 3), 1,800 (49%) e-mail addresses were randomly selected from the population of 3,679 CTSU users who had contacted the CTSU Help Desk over a 12-month period. Of the 1,800 users, there were approximately 325 respondents. This represents an 18% response rate for the survey or a sample of approximately 10% of the currently active user community. No other filtering or selection criteria were applied to selection of the names or e-mail addresses.
For the most recent annual web site survey (attachment 2), 1,000 names (15%) were randomly selected from the population of approximately 7,241 registered CTSU users who had visited the CTSU ‘members’ web site (http://members.ctsu.org) over a 12-month period. Then users who had invalid e-mail addresses or no current experience with the web site were filtered out, leaving 275 respondents. Since this is an annual survey, it is estimated that there will be 275 respondents surveyed over a one-year time frame. This represents a 28% response rate for the survey or a sample of approximately 5% of the currently active user community. No other filtering or selection criteria were applied to selection of the names or e-mail addresses.
For the Oncology Patient Enrollment Network (OPEN) survey (attachment 4), all users of the OPEN application are eligible to complete a survey. There is no consideration of other characteristics beyond use of the services/applications.
An email with the link to the Protocol and Information Office (PIO) Survey (attachment 5) will be sent out once per year to approximately 200 clinical trial site personnel. It is anticipated that up to 100 personnel will respond.
For the clinical trial surveys (attachments 6, 7, and 8), participants will be the Principle Investigator and site administrators listed in CTSU at sites that are either approved or pending for the trial of interest. Participants also will include those PIs and site administrators from sites that actively accrued patients to previous but similar trials (and who are not listed as approved or pending). Each participant will be sent an invitational email describing the request as well as a survey link to an online survey. Based on a previous pilot study conducted by NCI (OMB No. 0925-0046-21) an average of 146 participants completed a survey. Given the new structure of the National Clinical Trial Network (NCTN) (fewer but more innovative clinical trials), NCI is expecting to conduct no more than 10 clinical trial surveys a year. Assuming 150 participants a trial with ten surveys a year, the assumption is 1,500 participants completing the surveys per year.
Survey Procedures.
Survey questions fall into three general categories, frequency of use, general satisfaction, and specific questions on a new services or applications. The surveys are made up of 15 to 20 multiple choice questions with a free text field for additional comments at the end of the survey. The help desk and web site surveys have 5 to 8 core questions used quantitatively demonstrate trends in user satisfaction over time (attachments 2 and 3). Other questions are used to measure user satisfaction with new features or services, and to determine the frequency of application/service use by the respondent. A change request to update survey questions was submitted in May 2012. Questions are kept to a minimum to decrease the time necessary to complete the survey and encourage response.
The OPEN survey is an ongoing survey available to persons completing an enrollment in the OPEN application (attachment 4). A set of core questions was developed, but these may vary over time as new features are added to the system.
The survey populations for the help desk and web site surveys are randomly selected from a pool of approximately 1500 recent users of the help desk or web site over the previous 12 months. (Response rates are generally <30% and response rates are reflected in table A-12-1 in the SSA.) To avoid bias there is no attempt to stratify users. Several methods are used to obtain objective results:
To avoid sampling bias, surveys questionnaires are sent broadly to the customer base or randomly selected samples of the customers.
Identification of individual respondents is withheld from the staff involved in the service being surveyed.
To avoid over interpretation of single survey results, trends are measured over time.
Whenever possible, a core set of satisfaction measures are used over time.
Customers are provided opportunity for open-ended responses.
Survey notifications are sent by e-mail that includes instructions and a link. The survey is deployed using Survey Monkey™, an online survey system. Reminder e-mails are sent to participants that have not yet responded encouraging participation approximately a week after the initial e-mail. The survey is closed 10 business days after distribution.
All information collected is related to user experience with CTSU services and applications. No identifying information is collected in the survey and no questions of a sensitive nature are asked in the survey. E-mail information is not used within the analysis.
Following survey deployment, the project team reviews the survey data to identify recommendations for continuous improvement activities. This process will include identifying:
A summary of strengths identified by the customer;
A summary of weaknesses and customer concerns;
Recommendations for leveraging strengths identified by the survey data; and
Opportunities for improvement and priority recommendations.
Survey results for all multiple choice questions are categorized as “satisfied”, “neutral” and “dissatisfied.” These categories are selected based upon a Likert scale with 1 and 2 response categories as “dissatisfied”, a response of 3 is “neutral,” and responses of 4 or 5 are categorized as “satisfied.” Results are compared across surveys to show general customer satisfaction trends. In addition, free text comments are reviewed and categorized by area. Often comments are related to other CTSU processes, and these are shared with the team leaders from the appropriate task area.
The project team drafts the survey findings report using a standard survey template. Survey results are reported to CTEP using the Survey Findings Template that describes the following areas: survey objectives, methodology, results, important findings, conclusions and recommendations.
The decision to use clinical trial surveys was based on regular reviews of the CTEP NCTN trial portfolio (at least twice a month in CTEP branch meetings and/or the PRC meetings). If a trial in its development process is considered to be challenging, or if an active trial is deemed slow accruing, CTEP will work with the study team to identify if a feedback survey can help with their decision making. If so, the templates are tailored based on information in the trial’s protocol. Once the survey is tailored, it is reviewed by CTEP and the study team. An invitation email list is developed based on those sites that have opened the trial, are listed as pending to open the trial, or have opened/accrued to a previous trial related to the one listed in the survey. The list includes the site PI and the site administrator. When ready, an invitational email is sent via CTSU to these individuals. The email invitation has a description of the request and a link to the online survey. A reminder email is sent out after one week, and the survey closed after two weeks. Data are compiled from an excel sheet print out and put into a standard report and shared with CTEP and the study team to review.
Rationale for Sample Size.
For the help desk and web site surveys, a sample of approximately 1,500 participants was selected to avoid bias and achieve a significant number of respondents. Response rates are generally less than 30% and are reflected in table A.12-1 in the SSA.
Quality Control.
The contractor will follow a standard operating procedure developed for the CTSU surveys. The CTSU Standard Operating Procedures are reviewed every 6 months, and staff is trained on procedures in their work area.
Expected response rates are approximately 15 to 25%. The number of questions is kept to a minimum to limit the time needed to complete the survey and encourage response. In addition, responses are multiple choice using a Likert scale of 1 to 5 that also minimizes the time needed to complete the survey. Followup e-mails are sent to participants that have not responded, and a gift card giveaway is used to encourage participation. No follow up is completed for non-respondents.
The previously approved surveys were distributed to a small number of staff for review prior to distribution to the target population in 2010 (attachments 2, 3, and 4). The PIO survey (attachment 5) was first piloted in 2011 (OMB No. 0925-0046, Expiration Date 2/28/2013) and a decision was made to collect this information on an annual basis. This survey was subsequently approved under OMB No. 0925-0642 (NCI Fast Track) in August, 2013 so that this customer satisfaction survey could be fielded by October. Pilot tests of the clinical trial concepts, prospective and low accrual surveys were done in 2012-2013 (under OMB No. 0925-0046-21) and it was determined that this was both valuable and helpful in assisting CTEP and trial study teams to identify which elements of a trial posed the greatest barriers, clinicians’ interest in a trial (i.e., was the trial worth opening at their sites?), and ways a trial could be better supported to increase patient accrual.
No consultations with persons outside of the CTEP contracting team or the contractor have been made for the purposes of the surveys. The survey team is lead by the CTSU Project Director, Steve Riordan, and the CTSU Quality Assurance Associate, Mary Williams. The implementation team consists of the CTSU Help Desk Supervisor, the CTSU Promotion, Education and Training Manager, the CTSU web site lead, and other CTSU staff members as assigned.
Steve Riordan CTSU Project Director Westat Telephone: 301-517-4045 Email: steveriordan@westat.com |
Mary Williams CTSU Quality Assurance Associate Westat Telephone: 240-285-0824 Email: marywilliams@westat.com |
Monique Terry CTSU Help Desk Supervisor Westat Telephone: 240-246-6183 Email: moniqueterry@westat.com |
Megan Rossmann CTSU Promotion, Education and Training Manager Westat Telephone: 301-279-4503 Email: meganrossmann@westat.com |
Amandeep Singh CTSU Web Site Lead Westat Telephone: 301-212-2112 Email: amandeepsingh@westat.com |
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | TABLE OF CONTENTS |
| Author | Vivian Horovitch-Kelley |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |