Generic Statement B Communication Campaign REV 6 22X

REQUEST FOR CLEARANCE FOR Communications Testing for Comprehensive communication campaign for hitech act

ICR # 201105-0990-005

Submitted by:

Peter Garrett

Office of the National Coordinator for Health Information Technology
U.S. Department of Health and Human Services

Email: Peter.Garrett@hhs.gov

Phone: (202) 205-5820

August 27, 2010

Revised February 22, 2011

Statement B: Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

Communications testing includes various methods and approaches. The methods chosen for use depend on the nature of the message or materials tested, as well as their intended audience. Recommended methodologies and sample sizes are based on a review of the relevant literature, consultation with experts in the field, and a baseline of data gathered over many years of testing materials among professional, patient, and public audiences.

In general, communications testing relies on qualitative methods and is not intended to yield results that are statistically projectable. In qualitative studies, quota sampling is often used to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not representative of the population. In qualitative studies, all respondents are initially contacted by telephone or through the mail; over-recruiting is done to compensate for not following up with non-respondents.

Where quantitative methods are used, information collection activities will target these particular audiences with statistical sampling procedures employed to identify potential survey respondents. Telephone and Internet surveys will seek a convenience sample that nonetheless has a reasonable degree of diversity in key demographic characteristics such as age, gender, education, and race/ethnicity. ONC does not intend to generate nationally representative results or precise estimates of population parameters using these surveys. Telephone samples may be selected with random digit dialing (RDD) techniques, or with stratified sampling of telephone exchanges. For these samples, each sampling unit (e.g., telephone household or respondent within a household) has a known non-zero probability of selection.

B.2. Procedures for the Collection of Information

Questions in all communications testing methodologies include the following:

• Standard measures of communications that are designed to assess to what degree the message was successful in communicating information. These questions include measures of main idea recall, comprehension, believability, personal relevance, and likes and dislikes.

• Questions tailored for the communication message to address any special concerns the producer of the message may have (e.g., the effect and /or appropriateness of graphic depictions of negative health outcomes).

The methodologies planned for use in this submission will follow standard state-of-the-art approaches adapted from marketing and communications research. In this context, the term communications testing refers to testing messages, strategies, and communication materials, and should not be confused with "pretesting of questionnaires" prior to their full-scale use. The following methodologies will be used:

Individual In-depth Interviews. Individual in-depth interviews are used to elicit attitudes and perceptions that offer insight into better understanding critical influences on people’s mental models (i.e., belief structures), or for testing message concepts, draft materials, and communication strategies. Individual in-depth interviews are ideal when the information in question requires in-depth probing or when individual rather than group responses are considered more appropriate. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings. In-depth interviews are also cost-effective in eliciting comments on print materials. Individual in-depth interviews can either be conducted on-line at a designated Internet location, conducted in-person, or conducted over the telephone. In some cases, respondents can be sent material in advance, asked to read it, and told that someone will call to get their opinion.

Respondents for in-depth interviews are recruited from members of the target audience for the particular message or print material being tested. They are also recruited based on meeting other appropriate screening criteria, including their willingness to be interviewed. Specific written instructions in the form of a screening questionnaire are used during the recruitment process. The interviews themselves are conducted by skilled interviewers who follow a prescribed discussion outline. A minimum of 20-25 respondents are queried using this method of data collection; normally, 50 interviews are conducted. In-depth interviews are generally 30 to 45 minutes in length.

Focus Groups. Focus groups, or group interviews, are used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials development.) Focus groups are usually composed of 8 - 10 people who have characteristics similar to the target audience, or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. Focus groups are valuable in exploring participant reactions to message concepts before additional resources are put into their development. We may also include card sorting exercises in focus groups to gain an understanding of how participants group, sort and label various topics, phrases and terms related to health information technology. Card sorting is a frequently used qualitative method for determining the “information architecture” of a web site, i.e., how information should be presented on the web site to provide maximum benefit to users.

Web Usability Testing. Formative web usability testing obtains user feedback for early concepts or designs of web site features and content. The primary source of data in formative web usability testing is verbal data from the user. Early testing may use paper prototypes or initial screen designs. Later testing can be done on partial prototypes or prototypes of only one portion of a user interface. When possible, logging software is also used to capture user interaction with the software. Additionally, usability engineers often take notes of critical incidents that occur during the testing. The debriefing interview is an excellent source of information in formative testing. Usability engineers can probe in depth to understand sources of confusion in the interface. Formative testing needs to be conducted in a fairly rapid pace in order to provide design input when it is needed. As a consequence, the testing usually focuses on a small portion of the user interface, involves relatively few user-participants, and has less formal reporting mechanisms than summative testing. Ideally, software designers and developers can observe the evaluations and discuss results and potential solutions with the usability engineers after the testing. Usability testing has always tried to make the “context-of-use” as realistic as possible. However, a usability laboratory cannot duplicate actual conditions of use within an office or home. Interruptions and other demands for attention do not, and cannot, occur during usability testing conditions. As such these studies represent the best case condition. If the software is not usable in the laboratory, it will certainly not be usable in real-world use. However, usability in the laboratory does not guarantee usability in more realistic conditions.

Self-Administered Surveys. Self-administered surveys can be used to validate belief structures or to test drafts of ONC concepts and materials. Surveys can either be accessed online at a designated Internet location, or distributed to respondents gathered at a central location. ONC does not intend to generate nationally representative results or precise estimates of population parameters using these surveys.

Internet surveys can be administered to online panels and may use experimental designs to test hypotheses of about the relative efficacy of communication messages. Willing participants recruited from the target audience are randomly assigned to treatment or control conditions. After being exposed to the communication, participants are asked to provide information about their recall, emotional and cognitive reactions, beliefs, and behavioral intentions. When this method of communications testing is used, having at least 100 respondents per condition is desirable. In some cases, a follow-up survey may be conducted within a reasonable timeframe (e.g., one week) with those who complete the baseline survey.

When central location surveys are employed, people frequenting an expected location of the target audience are randomly stopped and screened to determine whether they meet the pre-determined selection criteria. When this method is used, at least 50 respondents are included; using approximately 100 respondents is desirable.

Cognitive Testing of Survey Instruments: We propose to engage in cognitive testing of survey instruments to ensure that respondents are able to understand the questions being asked, that questions are understood in the same way by all respondents, and that respondents are willing and able to answer such questions. Participants from different demographic segments will be recruited for cognitive testing of the various language versions of the survey instruments. Survey methodologists will administer the survey instruments to participants using the appropriate interview administration mode, followed by an individual in-person debriefing to assess the questionnaire items. To properly assess participants’ understanding of the survey questions and uncover potential issues, cognitive interviews will be conducted with approximately 25 participants. Cognitive testing sessions are generally 2 hours total in length.

Omnibus Survey. An omnibus survey is a telephone interview survey in which different organizations add questions to a single questionnaire, thereby sharing the cost. This technique uses random-digit-dialing (RDD) to speak to approximately 1,000 respondents with the intent of having a reasonable degree of diversity in key demographic characteristics such as age, gender, education, and race/ethnicity. ONC does not intend to generate nationally representative results or precise estimates of population parameters using these surveys. Because these surveys are conducted on a weekly or bi-weekly basis, they are an efficient way to obtain test communications messages from a larger number of consumers and/or professionals in a short period of time. To get such a quick and cost-effective turnaround, however, the vendor can make up to only four callbacks, resulting in a lower response rate than for custom surveys (where additional callbacks are made). Generally, for the most cost-effective approach, computer-assisted telephone interviewing (CATI) is used to complete the interviews.

Because of the increase in the prevalence of adults with only wireless telephones, cell-phone survey methodology may be used to limit coverage biases that could potentially be introduced by landline-only telephone surveys. Results from the 2006 National Health Interview Survey suggest that one out of every eight American adults live in households with only wireless telephones. The percentage of wireless-only households is highest among younger adults (25% of 18 to 24 year olds, and 29% of 25 to 29 year olds). Furthermore, results indicate that wireless-only adults are more likely to be current smokers than adults living in households with a landline (30% vs. 19%).¹ Given the known differences in demographics and lifestyle preferences between wireless-only and landline adults, cell-phone surveys offer a way to reduce this potential problem of coverage bias.

ONC recognizes that, for the purposes of this generic clearance, omnibus surveys should only be used for internal purposes (e.g., to better understand consumer perceptions before going public with communication campaigns). Omnibus survey results will not be used for evaluation purposes, or to make policy or regulatory decisions.

For all methodologies, professionally recognized procedures will be followed in each information collection activity to ensure high quality data. Examples of these procedures include the following:

• A minimum of 10 percent of telephone interviews will be monitored by supervisory staff;

• Data from mail or paper-and-pencil surveys will be computerized through scannable forms or checked through double-key entry;

• Observers will monitor focus groups, and focus group proceedings will be recorded; and

• Data submitted through on-line surveys will be subjected to statistical validation techniques (such as disallowing out-of-range values).

All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.

B.3. Methods to Maximize Response Rates and Deal with Non-response

In the case of data collection activities that involve interviews or telephone and in-person surveys, several procedures proven effective in previous studies will be used to maximize response rates:

• Potential respondents will be informed about the importance of these studies and encouraged to participate through a variety of methods, including letters of support from key individuals.

• Experienced, highly-trained staff will moderate all focus groups and conduct all interviews and surveys.

• Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.

• Well-defined conversion procedures will be established. If a respondent for a survey declines to be interviewed, a member of the contractor's conversion staff will contact the respondent to explain the importance of their participation. Conversion staff are highly experienced telephone interviewers whose style and persuasive abilities have demonstrated success in eliciting cooperation. They receive a pay differential to acknowledge these skills, which also serves as an incentive to the interviewer pool, whose completion rates are carefully monitored to assess their qualifications to serve as conversion staff.

• For telephone interviews, outgoing calls that result in a disposition of no answer, a busy signal, or an answering machine will be automatically rescheduled for subsequent attempts. Up to 20 outgoing calls to a given number with dispositions of the sort listed will be made before declaring it a non-response.

• Should a respondent interrupt an interview for any reason, such as needing to attend to a personal matter, the interviewer will reschedule or, in the case of telephone surveys, a predictive dialer will automatically reschedule the interview for completion at a later time.

• Fielding for telephone surveys will occur over at least a six-week period. Based on past experience, this time frame will allow the contractor to reach individuals who are on vacation, out of the home during irregular periods, have a temporarily disconnected telephone, or who are not answering the phone for some other reason.

• Interview staff will be able to provide respondents with the name and telephone number of an official at ONC. This official will confirm with respondents the importance of their participation.

B.4. Test of Procedures or Methods

Before each information collection is implemented, a contractor will pilot test the instrument(s) and method of data collection. Lessons from the pilot test will be identified, and changes as necessary will be incorporated into the instrument and method. All pilot tests will involve no more than nine individuals unless OMB clearance is sought for more than nine.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Melinda B. Buntin, Ph.D., Director, ONC Office of Economic Analysis and Modeling, among others, including contractors who may be chosen to collect communications testing information, will be responsible for the design of statistical and sampling procedures undertaken as part of these data collection activities.

1“Wireless Substitution: Early Release of Estimates Based on the National Health Interview Survey, July – December 2006. (PDF)” Blumberg, Stephen J., and Julian V. Luke. May 14, 2007a. Report by the U.S. Centers for Disease Control and Prevention.

5