0930-0196 Ssb 12-7-2021

Pretesting of Substance Abuse Prevention and Treatment

and Mental Health Services Communications Messages

Supporting Statement

B. Statistical Methods

1‑2 Respondent Universe and Sampling Methods/Information Collection Procedures

Pretesting includes a variety of methods and approaches. The methods chosen for use depend on the nature of the message or materials tested, as well as their intended target audience. Recommended methodologies and sample sizes will be based on a review of the relevant literature, consultation with experts in the field.

In general, pretesting relies on qualitative methods and is not intended to yield results that are statistically projectable. In qualitative studies, quota sampling is used to select a sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a non‑probability sample which is not representative of the population. In qualitative studies, all respondents will be initially contacted by telephone or through the mail; over‑recruiting is done to compensate for not following up with non‑respondents.

Questions in all pretesting methodologies will include the following:

• Standard measures of communications that are designed to assess to what degree the message was successful in communicating information. These questions include measures of main idea recall, comprehension, believability, personal relevance, and likes and dislikes.

• Questions tailored for the specific print piece/public service message to assess audience comprehension of specific information or to address any special concerns the producer of the message may have — for example, are the illustrations appropriate.

The methodologies planned for use in this pretesting submission represent the standard state‑of‑the art approaches adapted from marketing and communications research. In this context, the term pretesting refers to testing messages, strategies and communication materials, and should not be confused with "pre‑testing of questionnaires" prior to their full‑scale use. The following methodologies will be used:

Focus Groups. Focus groups, or group interviews, will be used to obtain insights into target audience perceptions beliefs and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials development). Focus groups will be usually composed of 8‑10 people who have characteristics similar to the intended target audience. The groups will be conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. Focus groups are valuable in exploring consumer reactions to message concepts before additional resources are put into their development.

Individual In‑depth Interviews. Individual in‑depth interviews will be used for pretesting PSA concepts, draft materials, and communication strategies when the information in question requires in‑depth probing or when individual rather than group responses are considered more appropriate. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. In‑depth interviews are also cost‑effective in eliciting comments on print materials. Respondents can be sent a copy of the material in advance, asked to read it, and told that someone will call to get their opinion. At the designated time, the recipient of the material is called and interviewed.

Self‑Administered Questionnaires. Self‑administered questionnaires will be used to pretest drafts of concepts and materials. They can either be mailed to respondents along with the draft being pretested or distributed to respondents gathered at a central location. When the latter method is employed, people frequenting a central location representative of the target audience may be randomly stopped and screened to determine whether they meet the pre‑determined selection criteria.

Gatekeeper Review. Because public and patient education materials are often distributed to their intended target audiences through health professionals or other intermediary organizations, the input of these groups to the concepts and materials is critical to the program’s success. As a result, these intermediaries, or gatekeepers, are often queried as part of the pretesting process. The information included in the questionnaire is complementary to that requested of target audience members.

3. Methods to Maximize Response Rates

Typically, quota sampling is used in the developmental activities, which comprise the bulk of effort under this approval. With such non‑probability sampling, response rates per se are not as important as assuring that focus groups and one‑on‑one interviews reflect the input of representatives of key consumer or provider populations (e.g., age, race).

4. Tests of Procedures

The activities to be conducted under this approval are all pretests. Interview guides and questionnaires to be used in the pretests will all be carefully developed and given careful scrutiny and limited, informal testing to assure completeness and smooth flow.

5. Statistical Consultants

Activities conducted under this approval will be under the overall supervision of the Director of the Center for Behavioral Health Statistical and Quality. Both SAMHSA employees and contractors will carry out most pretesting activities.

2