Supporting Statement B for:
Testing successful health communications surrounding aging-related issues
From the National Institute on Aging (NIA)
March 23, 2011
Megan J. Homer, M.A.
Office of Communications and Public Liaison
National Institute on Aging
9000 Rockville Pike
Bldg 31C, Suite 5C27
Bethesda, MD 20892
Telephone: 301-496-1752
Fax: 301-496-1072
Table of contents
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 3
B.1 Respondent Universe and Sampling Methods 3
B.2 Procedures for the Collection of Information 3
B.3 Methods to Maximize Response Rates and Deal with Non-response 6
B.4 Test of Procedures or Methods to be Undertaken 7
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data…………………………………………………………………..7
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
Communications testing employs a variety of methods and approaches. Methods are selected based on the nature of the message or materials tested, as well as their intended audience. Recommended methodologies and sample sizes are based on a review of the relevant literature, consultation with experts in the field, and a baseline of data gathered over many years of testing materials among professional, patient, and public audiences.
Communications testing generally relies on qualitative methods and is not intended to yield results that are statistically projectable. In qualitative studies, quota sampling is often used to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not representative of the population. In qualitative studies, all respondents are initially contacted by telephone, through the mail, or in-person at a central, public location or place of residence, such as a shopping mall, senior center, or senior living facility; over-recruiting by approximately 30% is done to compensate for non-respondents.
Where quantitative methods are used, information collection activities will target these particular audiences with statistical sampling procedures employed to identify potential survey respondents. Mail, telephone, and Internet surveys will seek a convenience sample that nonetheless has a reasonable degree of diversity in key demographic characteristics such as age (young-old to old-old), gender, education, and race/ethnicity. NIA does not intend to generate nationally representative results or precise estimates of population parameters using these surveys. Telephone samples may be selected with random digit dialing (RDD) techniques, or with stratified sampling of telephone exchanges. For these samples, each sampling unit (e.g., telephone household or respondent within a household) has a known non-zero probability of selection.
B.2. Procedures for the Collection of Information
Questions in all communications testing methodologies include the following:
Standard measures of communications that are designed to assess to what degree the message or communications approach was successful in communicating information. These questions include measures of personal relevance, trustworthiness, readability, usefulness, and likes and dislikes.
Questions tailored for the communication message to address any special concerns the producer of the message may have.
The methodologies planned for use in this submission will follow standard state-of-the-art approaches adapted from marketing and communications research. In this context, the term communications testing refers to testing messages, strategies, and communication materials. The following methodologies will be used:
Individual In-depth Interviews. Individual in-depth interviews are used to elicit attitudes and perceptions that offer insight into better understanding critical influences on people’s mental models (i.e., belief structures), or for testing message concepts, draft materials, and communication strategies. Individual in-depth interviews are ideal when the information in question requires in-depth probing or when individual rather than group responses are considered more appropriate. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. In-depth interviews are also cost-effective in eliciting comments on print materials. Individual in-depth interviews can either be conducted on-line at a designated Internet location, conducted in-person, or conducted over the telephone. In some cases, respondents can be sent material in advance, asked to read it, and told that someone will call to get their opinion.
Respondents for in-depth interviews are recruited from members of the target audience for the particular message or print material being tested. They are also recruited based on meeting other appropriate screening criteria, including their willingness to be interviewed. Specific written instructions in the form of a screening questionnaire are used during the recruitment process. The interviews themselves are conducted by skilled interviewers who follow a prescribed discussion outline. Over 5 years, NIA estimates that it will conduct a maximum of 100 in-depth interviews with older people; 100 in-depth interviews with non-physician health professionals, caregivers, or people who work with seniors; and 50 in-depth interviews with physicians. Typically, 15-20 respondents are interviewed per communications project; more or fewer respondents may be interviewed until it is decided that interviewing additional respondents will not yield any new information (the saturation point). In-depth interviews are generally 45 to 60 minutes in length.
Focus Groups. Focus groups, or group interviews, are used to obtain insights into target audience perceptions, beliefs, and attitudes, often at an early stage of the communication process (i.e., for materials or communications approaches in development.) Focus groups are usually composed of 8 - 10 people who have characteristics similar to the target audience, or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. Focus groups are valuable in exploring consumer reactions to message concepts before additional resources are put into their development. Over 5 years, NIA estimates that it will conduct focus groups with a maximum of 200 older people; 150 non-physician health professionals, caregivers, or people who work with seniors; 50 physicians.
Self-Administered Questionnaires. Self-administered questionnaires can be used to test drafts of NIA concepts and materials. Surveys can either be mailed to respondents along with the draft being tested, accessed on-line at a designated Internet location, or distributed to respondents gathered at a central location. NIA does not intend to generate nationally representative results or precise estimates of population parameters using these surveys.
Internet-based/website surveys can be administered to online panels and may use experimental designs to test hypotheses of about the relative efficacy of communication messages. After being exposed to the communication (including a website redesign), participants are asked to provide information such as their reactions to the messages, whether they find it usable, their general beliefs about the trustworthiness and applicability to their lives, and behavioral intentions based on the communications material. When this method of communications testing is used, having at least 100 respondents per condition is desirable.
When central location surveys are employed, people frequenting an expected location of the target audience (for example, a senior center) are randomly stopped and screened to determine whether they meet the pre-determined selection criteria. When this method is used, at least 50 respondents are included; using approximately 100 respondents is desirable.
Over 5 years, NIA plans to collect information through self-administered questionnaires from a maximum of 500 older people; 800 non-physician health professionals, caregivers, or people who work with seniors; and 200 physicians that should take approximately 15 minutes to complete. Over 5 years, NIA plans to collect information through a maximum of 1000 website surveys (presumably half with older people and half with caregivers or others who work with older people). These surveys should take approximately 10 minutes to complete.
For all methodologies, professionally recognized procedures will be followed in each information collection activity to ensure high quality data. Examples of these procedures include the following:
A minimum of 10 percent of in-depth interviews will be monitored by supervisory staff;
Observers will monitor focus groups, and focus group proceedings will be recorded;
Data from mail or paper-and-pencil surveys will be checked through double-key entry; and
Data submitted through on-line surveys will be subjected to statistical validation techniques (such as disallowing out-of-range values).
All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.
B.3. Methods to Maximize Response Rates and Deal with Non-response
In the case of data collection activities that involve interviews or telephone, mail, and in-person surveys, several procedures proven effective in communications literature will be used to maximize response rates:
Potential respondents will be informed about the importance of these studies and encouraged to participate through a variety of methods, including letters of support from key individuals.
Experienced, highly-trained staff will moderate all focus groups and conduct all interviews and surveys.
Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.
Well-defined conversion procedures will be established. If a respondent for a survey declines to be interviewed, a member of the contractor's conversion staff will contact the respondent to explain the importance of their participation. Conversion staff are highly experienced telephone interviewers whose style and persuasive abilities have demonstrated success in eliciting cooperation. If the respondent still declines to be interviewed, the respondent will be thanked for his/her time and asked if it would be okay for NIA to make contact at a later time to see if he/she has changed his/her mind and would be interested in participating.
For telephone interviews, outgoing calls that result in a disposition of no answer, a busy signal, or an answering machine will be automatically rescheduled for subsequent attempts. Up to 10 outgoing calls to a given number with dispositions of the sort listed will be made before declaring it a non-response.
Should a respondent interrupt an interview for any reason, such as needing to attend to a personal matter, the interviewer will reschedule or, in the case of telephone surveys, a predictive dialer will automatically reschedule the interview for completion at a later time.
Fielding for telephone and mail surveys will occur over at least a six-week period. This time frame will allow the contractor to reach individuals who are on vacation, out of the home during irregular periods, have a temporarily disconnected telephone, or who are not answering the phone for some other reason.
Interview staff will be able to provide respondents with the name and telephone number of an official at NIA. This official will confirm with respondents the importance of their participation.
A toll-free number for NIA or a contractor’s office will allow potential respondents to speak with someone to confirm a study’s legitimacy at their convenience.
For mail surveys, a number of techniques will augment response rates:
A self-addressed, stamped return envelope will be enclosed with each survey.
Creative and attractive graphics will be used to attract the attention of respondents (e.g., different colored paper for successive survey iterations).
Hand-signed cover letters will be sent with each survey.
Follow-up mail (up to 3 mailings) or phone contacts (up to 10 call-backs) will be made to encourage participation; participant objectivity will be encouraged by reminding participants about the importance of providing both negative and positive feedback.
Respondents will be allowed the option of faxing back completed surveys (and possibly offered the option of completing the survey on-line).
B.4. Test of Procedures or Methods
Before each information collection is implemented, a contractor will pilot test the instrument(s) and method of data collection. Lessons from the pilot test will be identified, and changes as necessary will be incorporated into the instrument and method. All pilot tests will involve no more than nine individuals unless OMB clearance is sought for more than nine.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Contractors chosen to collect communications testing information will be responsible for the design of statistical and sampling procedures undertaken as part of these data collection activities. NIA staff will confirm these procedures.
| File Type | application/msword |
| Author | gittlesond |
| Last Modified By | curriem |
| File Modified | 2011-03-23 |
| File Created | 2011-03-23 |