SUPPORTING STATEMENT B For:
A Generic Submission for Formative Research, Pre-testing,
Stakeholder Measures and Advocate Forms at NCI
March 9, 2011
Shannon K. Bell
Office of Advocacy Relations
Office of the Director
National Cancer Institute
31 Center Drive, Bldg. 31
Room 10A30, MSC 2580
Bethesda, Maryland 20892
Telephone: 301-451-3393
Fax: 301-480-7558
E-mail: bells@mail.nih.gov
Table of Contents
B. statistical methods 1
B.1 Respondent Universe and Sampling Methods 1
B.2. Procedures for the Collection of Information 2
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4
B.4 Test of Procedures or Methods to be Undertaken 6
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 7
List of Attachments
NCI Office of Advocacy Relations, Post-Activity Survey for Advocates
NCI Office of Advocacy Relations, Post-Activity Survey for Staff
Pre-Test for Advocates, CARRA Peer Review Workshop
Post-Test for Advocates, CARRA Peer Review Workshop
Reflections and Directions: Interviews and Focus Groups with CARRA Members and NCI Staff
Benefits of Advocacy Involvement in the Research Setting
Advocates in Research Working Group Membership
Memo from NIH Privacy Act Officer
Privacy Impact Assessment (PIA)
B. Statistical Methods
B.1 Respondent Universe and Sampling Methods
The respondent universe consists of research advocates and organizations that interact with the National Cancer Institute (NCI), and staff (both at NCI and funded through NCI grants) that work with the advocacy community. It is difficult to define and anticipate all the types of potential respondents, but below are descriptions of the people who could represent the respondent universe in this generic submission:
People with cancer, their family members, and their caregivers who wish to serve as research advocates on behalf of NCI;
Researchers, physicians (general practitioners, oncologists, radiologists), nurse practitioners, physician assistants, and nurses at NCI-funded Cancer Centers who interact with research advocates;
Advocacy organizations that support NCI’s mission and involve research advocates in their work.
The Office of Advocacy Relations (OAR) will conduct information collection for the purpose of program development, monitoring and improvement. As such, most sampling methods will not require large numbers of survey respondents, and are not intended to ensure generalizability of results. The methods, however, will ensure that NCI collects enough data to inform decisions about messages, materials, programs, and services.
The OAR will occasionally send questionnaires to all advocates or organizations in its database – negating the need for sampling as the entire population will be contacted. At other times, OAR may use stratified sampling to group different stakeholders and then systematically survey each group so that respondents have an equal chance of being selected.
The OAR will ensure that each study population receives numerous requests to enable the highest response rate possible. Again, this will help guarantee that NCI is collecting enough data to make informed decisions about program development, monitoring and improvement.
B.2 Procedures for the Collection of Information
Studies involving formative research, pre-testing, customer satisfaction, and stakeholder measures may involve a wide range of information collection methods. Each information collection method is described below.
Focus Groups: Focus groups serve as a particularly useful medium to collect information when rich, in-depth information regarding attitudes and reactions to products is desired. Focus groups traditionally take place in an in-person format, in which a moderator facilitates a discussion regarding a product, issue, or program. Depending on factors such as the geographic distribution and schedules of NCI staff, grantees, and research advocates, and on the nature of the products and services under investigation, focus groups may use a variety of technology-based formats, including videoconferencing, Internet, and teleconferencing (see Supporting Statement, Section A.3. for a brief description of each).
Individual In-depth Interviews: Individual in-depth interviews will be used when the information in question requires an in-depth, open-ended approach. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. Specific written instructions to guide the one-to-one discussion will be utilized by skilled interviewers.
Written surveys (email surveys or questionnaires) will be distributed to stakeholders electronically using the internet. Respondents will complete the questionnaires and return them for tabulation and analysis.
Telephone surveys are particularly useful for collecting information in relatively short time frames while maintaining high response rates, with resulting estimates generalizable to the universe of study. A sample of potential respondents is drawn and trained interviewers contact them to administer a questionnaire over the phone.
Computer-assisted telephone interviewing (CATI) surveys conducted by telephone are well-suited to this technology, which employs automated dialing; schedules unanswered calls or interrupted interviews for efficient callbacks; randomly selects respondents; uses automated skip patterns and instantaneous out-of-range checks; insertion of information from one question to guide a subsequent question; and the automated generation of databases for subsequent analysis.
In-person surveys, implemented in a central location to elicit information regarding certain topics or issues. This method may be particularly useful for the examination of stakeholder satisfaction with OAR products and services during a large scientific meeting. In-person data collection methods that incorporate Computer-assisted personal interviewing (CAPI) and fill in words (ACASI) technology are described below:
Computer-assisted personal interviewing (CAPI) technology allows interviewers to ask questions of a respondent using a computer to enter data. Some primary advantages of CAPI include: the elimination of routing and looping problems within a paper-and-pencil questionnaire; respondents and interviewers cannot accidentally skip questions; interview questions are customized to account for personal information provided by the respondent (e.g., age, information from previous questions); software can automatically perform mathematical calculations and tabulations; software checks for inadmissible or inconsistent responses; allows interviewers to administer surveys to geographically isolated groups, respondents without telephones or Internet access, or other difficult-to-reach populations; CAPI eliminates errors that arise from separate data entry. ACASI software technology offers many advantages of CAPI technology, but removes the need to have a person administer an interview. Instead, survey questions are pre-recorded and played back through the sound system of a computer, which the respondent can listen to privately by using headphones. Respondents select an answer by pressing a key that corresponds to one choice shown on the screen, after which answers are fed directly into a computer database.
When necessary, OAR will ensure that industry standards are followed to ensure quality control. Data analysis will be conducted under the advice of a statistician, as appropriate.
B.3 Methods to Maximize Response Rates and Deal with Nonresponse
In the case of data collection activities that involve interviews or telephone, email, and in-person surveys, several procedures that have proven effective in previous studies will be used when possible to maximize response rates:
Potential respondents will be informed about the importance of these studies and encouraged to participate through a variety of methods, including newsletters from professional associations or community organizations, and letters of support from key individuals.
Experienced, highly-trained staff will moderate all focus groups and conduct all interviews and surveys.
Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.
Well-defined conversion procedures will be established. If a respondent for a survey declines to be interviewed, a member of the contractor's conversion staff will contact the respondent to explain the importance of their participation. (Conversion staff is highly experienced telephone interviewers whose style and persuasive abilities have demonstrated success in eliciting cooperation. They receive a pay differential to acknowledge these skills, which also serves as an incentive to the interviewer pool, whose completion rates are carefully monitored to assess their qualifications to serve as conversion staff.)
For telephone interviews, outgoing calls that result in a disposition of no answer, a busy signal, or an answering machine will be automatically rescheduled for subsequent attempts. Up to 20 outgoing calls to a given number with dispositions of the sort listed will be made before declaring it a non-response.
Should a respondent interrupt an interview for any reason, such as needing to attend to a personal matter, the interviewer will reschedule or, in the case of telephone surveys, a predictive dialer will automatically reschedule the interview for completion at a later time.
Fielding for telephone and email surveys will occur over a period of six or more weeks. Based on past experience, this time frame will allow the contractor to reach individuals who are on vacation, out of the home during irregular periods, have a temporarily disconnected telephone, or who are not answering the phone for some other reason.
Interview staff will be able to provide respondents with the name and telephone number of an official at NCI. This official will confirm with respondents the importance of their participation.
For email surveys, a number of techniques will augment response rates:
Creative and attractive graphics will be used to attract the attention of respondents (e.g., different colored paper for successive survey iterations).
Follow-up mail or phone contacts will be made to encourage participation.
Respondents will be allowed the option of completing surveys online or faxing back completed surveys.
B.4 Tests of Procedures or Methods to be Undertaken
Before each information collection is implemented, contractors will pilot-test the instrument(s) and method of data collection. Lessons from the pilot test will be identified, and changes will be incorporated into the instrument and method, as necessary. All pretests will involve no more than nine individuals unless OMB clearance is sought for more than nine.
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data
The following individuals, among others, including contractors, will be consulted for the design of statistical and sampling procedures undertaken as part of these data collection activities:
Larry Solomon, Ph.D.
Senior Health Science Analyst
Program Assessment Branch
Office of Science Planning and Assessment
National Cancer Institute
6116 Executive Boulevard
Suite 205, MSC 8320
Bethesda, MD 20892-8320
301-451-9985 (voice)
Gordon Willis, PhD
Cognitive Psychologist
Applied Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
6130 Executive Blvd, Suite 4006, MSC 7344
Bethesda, MD 20892-7344
301-594-6652 (voice)
Nina Goodman, MHS
Senior Advisor
Office of Market Research and Evaluation
Office of Communications and Education
National Cancer Institute
6116 Executive Blvd., Suite 400
Rockville, MD 20892-8351
301-435-7789 (voice)
| File Type | application/msword |
| Author | McCarthyK |
| Last Modified By | Vivian Horovitch-Kelley |
| File Modified | 2011-03-09 |
| File Created | 2010-06-09 |