National Survey on Drug Use and Health:
SUD Module Cognitive Interview Study
SUPPORTING STATEMENT
A. JUSTIFICATION
The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting Office of Management and Budget (OMB) approval to conduct up to 44 cognitive interviews to test a series of new and revised questions designed to produce prevalence estimates of substance use disorders (SUD) for the National Survey on Drug Use and Health (NSDUH) based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) (American Psychiatric Association [APA], 2013). This cognitive interview package is submitted under the NSDUH Methodological Field Test generic OMB clearance (OMB Control No. 0930-0290).
NSDUH is sponsored by SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) and approved under OMB Control No. 0930-0110. The data collection is a national survey of the U.S. civilian, noninstitutionalized population aged 12 or older. This survey is paramount in meeting a critical objective of SAMHSA’s mission—to maintain current data on the incidence and prevalence of substance use and mental health problems in the United States. NSDUH has been conducted on a periodic basis from 1971 to 1988 and annually since 1990.
NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa4 – Data Collection). Section 505 specifically authorizes annual data collection for monitoring the incidence and prevalence of illicit substance use and mental health problems, as well as the abuse of licit substances in the U.S. population.
The purpose of this methodological pretest study is to conduct cognitive testing on new and revised measures of SUD questions based on the DSM-5 should the NSDUH be updated to reflect DSM 5. The timing is critical as any changes to the SUD items have to be cognitively tested and validated in order to ensure accurate measure. The SUD questions were designed using questionnaire best practices, including using clear reference periods and avoiding double-barreled questions. Review by a panel of subject matter experts has also been completed. As the next step in the question development process, cognitive testing is needed to empirically evaluate whether survey respondents understand the questions as intended and can provide accurate answers.
The DSM is the manual used by clinicians and researchers to diagnose and classify mental disorders, including SUD. The APA published the DSM-5 in 2013, culminating in a 14‑year revision process. Since 2000, NSDUH has included an assessment of DSM-IV-based SUD as part of its data collection. To continue producing current, high-quality mental health data, CBHSQ has been working to update the NSDUH SUD modules to reflect the changes in DSM-5 and address items shown to have higher than ideal rates of false positives and false negatives.
In 2014, CBHSQ began by revising the existing NSDUH SUD module to better align with the updated substance use dependence criteria in the DSM-5, including the addition of questions on marijuana withdrawal symptoms, questions measuring the craving of any substance, and modification to existing questions on the withdrawal from any substance. That revised module was reviewed by mental health experts and survey methodologists to ensure the questions addressed the DSM-5 criteria and would be easily and accurately answered by respondents. After that expert review, the revised SUD items were cognitively tested in 2015 (under NSDUH’s generic clearance, OMB Control No. 0930-0290). As a result of the cognitive interview findings, CBHSQ determined that additional research and testing were needed before updating the current NSDUH questions to reflect DSM-5 criteria.
In 2017, the NSDUH SUD module—the existing items and new items evaluated in 2015—was reevaluated and updated to improve validity and ensure that the questions capture the DSM-5 criteria. A panel of experts in substance use assessment and survey design was convened, after which further updates were made to the SUD module to incorporate the experts’ feedback.
The next phase of this effort is now planned to conduct up to 44 cognitive interviews to assess how well the revised SUD questions perform. These cognitive interviews will be conducted with past 12-month users of substances such as alcohol, marijuana, cocaine, and/or heroin. Because NSDUH is designed to produce SUD estimates for the civilian population aged 12 or older, cognitive interviews will be conducted with both adolescents (aged 12 to 17) and adults (aged 18 or older). In addition, these revised SUD questions will be administered in English and Spanish as is done in the main NSDUH instrument.
The purpose of this methodological pretest study is to conduct cognitive testing on new and revised measures of SUD questions based on the DSM-5 should the NSDUH be updated to reflect DSM 5. The timing is critical as any changes to the SUD items have to be cognitively tested and validated in order to ensure accurate measure.
Specifically, the study aims to collect and analyze qualitative data from past year users of alcohol, marijuana, cocaine, heroin, hallucinogens, inhalants, prescription sedatives, prescription tranquilizers, prescription stimulants, and opioid pain relievers to evaluate if they understand the concepts and language of the new and revised SUD questions, whether they find them difficult to answer, and whether they are able to provide accurate reports of their substance use behaviors.
Approximately 150 individuals will be screened, and up to 44 of the screening respondents will be enrolled across three rounds of cognitive interviews. The targeted final sample size for completed cognitive interviews is 42 (15 English-speaking adolescents, 15 English-speaking adults, and 12 Spanish-speaking adults), with 1 or 2 additional screening respondents needed for interviews if respondents who indicated substance use during the initial screener report no substance use during the interview. Such a scenario, while unlikely to occur, would require conducting a total of up to 44 interviews to reach the final target of 42.
Approximately one quarter of the sample (16 respondents) will be past year alcohol users, one quarter will be past year marijuana users, and half will be past year users of other drugs to reflect the fact that use of alcohol or marijuana is more common than other substances. Although only one half of the sample will be other drug users, the questions are the same across the SUD modules, except for the withdrawal questions. Therefore, the specified sample selection should provide an adequate number of responses to all questions.
As explained in Section A.1, the findings from the cognitive interviews will be used to revise the survey questions that will be included in a NSDUH clinical validation.
Information technology will be used in two primary situations: (1) participant screening, and (2) participant interviewing.
Screening. English-speaking adult participants will be screened for eligibility using a short questionnaire administered via the web. Potential adolescent participants will be screened over the telephone (once parental consent has been received). Spanish-speaking participants will be screened over the telephone or in-person.
Adult participants who are recruited using online advertisements, such as those placed on www.craigslist.com, will complete a web screener to determine if they are eligible to participate in the study. The web screener will be programmed using a web survey platform that uses secure (i.e., https) share links, is password protected, and is hosted inside a network that is FIPS-Moderate compliant. Response data will be collected and stored in SQL Server databases that reside inside the FIPS-Moderate security perimeter. Electronic screening data are accessible only within the FIPS-Moderate Virtual Desktop Infrastructure (VDI) environment.
Responses to the phone screening for adolescents and Spanish-language speakers will be entered into the same survey platform used for the web screener by authorized project staff. All hard-copy screeners completed for in-person screening of Spanish-speaking participants will not include a name and will include only an ID number. They will then be transcribed electronically and secured within the Contractor’s FIPS-Moderate network. Hard-copies will be destroyed once a digital record has been made.
During the screening, information will be collected on each respondent’s age, race/ethnicity, education, phone number, city of residence, any significant physical limitation that could preclude participation, and past substance use. The telephone numbers collected will be used to remind the participants about upcoming appointments and to recruit additional participants placed on a wait list if some of the originally recruited participants are no longer available or miss an appointment.
Each individual screened will be assigned a unique case number, so participants’ names will not be stored with their interview responses.
Screening responses stored electronically will be destroyed within 4 weeks after the final memorandum with the cognitive interview findings is completed and approved by CBHSQ.
Interviewing. Information technology will be used for both adolescents and adults in administering the cognitive interviews via computer-assisted personal interviewing (CAPI) and audio computer-assisted self-interviewing (ACASI), the same technology used for main study NSDUH interviews. Information technology will also be used for audio recording and screen capture, as well as enabling interview streaming to allow secured, remote observation of interviews by CBHSQ staff.
In all rounds of cognitive interviews, lightweight, ultra-book laptops will be used, identical to those used for the 2018 NSDUH main study data collection. The interview will use a combination of CAPI and ACASI. The interview will focus on questions in the revised SUD module. All interviews will be conducted by experienced survey methodologists with many years of experience with cognitive interviewing.
The interview will begin after the participant has given his or her consent. The consent procedure includes consent for the interview, as well as consent for audio/screen recording and observation using online streaming. Participants can refuse consent for audio/screen recording, and observation, and the interview will proceed without those elements. In such instances, the cognitive interviewer will take handwritten notes.
If the participant consents to recording, audio/screen recordings of the participant’s ACASI interview experience will be created on the encrypted project laptop using audio/screen capture software such as Camtasia® or a comparable product. It will include only a recording of what the respondent saw on the laptop display, including the respondent’s interaction with the ACASI instrument (mouse movements and clicks). The audio component of the recordings will include audio generated by the ACASI instrument, as well as any audible interaction between the respondent and the cognitive interviewer. The audio and screen capture recordings will not contain any information that identifies a respondent. The recorded files will be identified by a unique participant ID number. Data structures linking the recording IDs to participants will be stored in a FIPS-Moderate network. Recordings will be destroyed within 4 weeks after the final memorandum with the cognitive interview findings is completed and approved by CBHSQ.
If consent is provided for observation, and there are staff observing remotely, then remote observation (via a program such as Cisco WebEx™) will be viewable only through a password-protected link. Broadband Internet cards will be used to create wireless Internet hotspots in all interview locations outside the secured network at the Contractor’s facilities. The use of these broadband cards will enhance security of the interviews by limiting access to the wireless network to only authorized project staff. Observing the interview includes listening to the cognitive interviewer/respondent interaction and watching a screen sharing of only what is displayed on the participant’s screen. It will not include any visual of the participant’s face or body or images of the room where the interview is taking place.
The interview will begin with the interviewer asking the participant for demographic information such as age (to determine how the interview program should route the participant through questions in the interview). After the CAPI portion of the interview, the interviewer will show each respondent how to navigate through the interview program. Participants will then complete a tutorial to learn how to use the ACASI portion of the survey. Only one new ACASI tutorial question was added to this module, which explains how participants should answer grid-style survey questions.
Following the tutorial, participants will complete an abridged version of the 2018 NSDUH main study core drug screening modules for alcohol, marijuana, cocaine, heroin, and methamphetamines, as well as medical use and misuse of prescription drugs. These modules will include the same questions used in the 2018 NSDUH, except that questions not necessary for cognitive testing (such as age at first use) will be removed. No new questions will be added to these modules for cognitive testing. Answers that participants give to questions about use of substances will determine whether they are asked more detailed questions about substance dependence and withdrawal.
For all three rounds of interviews, participants will complete the core drug screening modules via ACASI. For the SUD module, participants will be given the option of having questions played over the computer’s speakers or turning off the sound and reading the questions aloud. These procedures are required so the interviewer can ask the participant cognitive interview probes after certain questions. Providing participants the option of hearing the questions or reading them allows the cognitive interview process to more closely mimic an actual NSDUH interview where participants can turn down the volume and read the questions if desired.
Cognitive interviewers may take notes electronically on their laptops, which are also secured with Check Point Endpoint disk encryption software. Thus, the data on the laptops will be encrypted. Any hard-copy notes that are then transcribed electronically will be stored only on interviewers’ laptops. Interviewers will use a unique case number rather than the participant’s name on all interview notes. Other electronic files containing personal information such as telephone numbers will be password protected, with the password set to expire within 4 weeks after the final memorandum with the cognitive interview findings is completed and approved by CBHSQ. The electronic files and hard-copies will be destroyed at that time.
CBHSQ is in contact with all major federal health survey managers and is aware of no other efforts to assess how potential respondents may react to changes made to the NSDUH questionnaire revisions to the SUD modules. To date, no duplication of effort has been identified.
This survey does not involve small businesses or other such entities.
This data collection is designed to revise questions prior to their inclusion in the NSDUH. This project is a onetime collection and will not be repeated.
This information collection fully complies with 5 CFR 1320.5(d)(2).
External expert reviews were conducted in two phases. The first phase involved conducting an in-person expert panel meeting on May 8 and 9, 2017, to review the existing NSDUH SUD module. Experts were chosen to provide expertise in survey methodology, adolescent and adult SUD assessment, as well as SUD among Spanish-speaking populations. In preparation for the in-person expert panel meeting, experts were provided with an overview of NSDUH, copies of the DSM-5 criteria for SUD, and the existing NSDUH SUD module questions for alcohol use disorder and cocaine use disorder. Experts were also provided an informal report prepared by Contractor staff that examined diagnostic criteria changes from the DSM-IV to DSM-5 and evaluated their impact on NSDUH.
The expert panel included the following persons:
Paul C. Beatty, Ph.D., Chief, Center for Survey Measurement, U.S. Census Bureau, (301) 763-5001;
Raul Caetano, M.D., Ph.D., Senior Research Scientist, Pacific Institute for Research and Evaluation, (510) 883-5728;
Michael First, M.D., Professor, Columbia University, (646) 774-7935;
Prudence Fisher, Ph.D., Associate Professor, Columbia University, (212) 543-5357;
Deborah Hasin, Ph.D., Professor, Columbia University, (212) 543-5035;
Aaron Hogue, Ph.D., Director of Adolescent and Family Research, The National Center on Addiction and Substance Abuse, (212) 841-5200;
Brent Moore, Ph.D., Research Scientist, Yale School of Medicine, (203) 932-5711; and
Glorisa Canino, Ph.D. (via telephone), Professor, University of Puerto Rico, (787) 758-2525.
Additional written feedback was sought from an expert at the National Institute on Drug Abuse, who was unable to attend the expert panel:
Wilson M. Compton, M.D., Deputy Director, National Institute on Drug Abuse, (301) 443-6480.
Once feedback was received from the expert reviewers, the revised NSDUH SUD modules were drafted. Then a second, written review of the revised module was requested from the experts. For this second review, the revised questionnaire was sent to the experts for their review and feedback. They were asked to review the questions for face and criterion validity concerns, as well as potential sources of participant confusion or language concerns. All prior expert reviewers provided written feedback on the items, except for Dr. Beatty, who was unavailable at the time. Based on the feedback provided by the experts, the items were further revised into their current form.
Both adult (aged 18 or older) and adolescent (aged 12 to 17) cognitive interview participants will be given $40 cash upon completion of the interview. The interviews will last, on average, 60 to 75 minutes. This incentive amount will be sufficient to compensate for the participants’ time and any travel expenses incurred. This amount is consistent with the amount requested for other studies of this length.
This incentive amount is recommended for adolescents as well as adults based on experience recruiting participants from a specific population—in this case, adolescent drug users. Given the relationship of the schedule for cognitive interviewing to the schedule for fielding the revised NSDUH instrument by the 2020 clinical validation study, significant delays in recruiting participants with a lower incentive amount could adversely affect the timely implementation of these future activities.
The incentive for the cognitive interview is mentioned in the following materials: Recruitment Flyers (Attachment A), Web Recruitment Screener (Attachment B), Telephone Recruitment Screener (Attachment C), Parental Permission Form (Attachment D), Participant Informed Consent/Assent Forms (Attachment E), Protocol for Cognitive Interviews (Attachment F), and Cognitive Interview Receipt for Participation (Attachment G).
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of NSDUH and will continue to be given the utmost emphasis.
The Contractor’s Institutional Review Board (IRB) was granted a Federalwide Assurance (Attachment H) by the Office for Human Research Protections (OHRP) and the U.S. Department of Health and Human Services (HHS) in compliance with the requirements for the protection of human subjects (45 CFR 46). The Contractor’s IRB will approve the protocols and consent forms for these cognitive interviews prior to any respondent contact. The IRB's primary concern is protecting respondents’ rights, one of which is maintaining the confidentiality of respondent information. By obtaining IRB approval for NSDUH procedures and materials, CBHSQ is assured that respondent confidentiality will be maintained.
The cognitive interviews for this study will incorporate several procedures to ensure respondents’ rights will be protected. The Recruitment Flyers (Attachment A) will advertise to the participants that “all responses will be kept confidential under federal law.” Also, the Web Recruitment and Telephone Recruitment Screeners (Attachments B and C), Parental Permission Form (Attachment D), and the Participant Informed Consent/Assent Forms (Attachment E) all indicate to the participants that the interview will be conducted in private to ensure the following:
no one aside from authorized project staff at CBHSQ or the Contractor will overhear their answers;
all answers will be kept private and confidential;
information given by the participants will not be shared with any persons outside the authorized project staff;
their name will never be connected with the answers they provide; and
federal law (Confidential Information Protection and Statistical Efficiency Act of 2002 [CIPSEA]) requires that their answers be kept confidential and used only for statistical purposes.
In these same study materials, participants are informed that their responses are voluntary and are assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular question.
During the recruitment process, information will be collected on the age, race/ethnicity, education, phone number, city of residence, any significant physical limitation that would preclude participation, and past substance use. The telephone numbers collected will be used to remind the participants about their upcoming appointments and to recruit additional participants placed on a wait list if any originally recruited participant is no longer available or misses an appointment.
All internal communication among project staff regarding participants will include only the first name of the participant and the time of the interview (e.g., when the recruiter notifies the interviewer of a scheduled appointment with a participant).
All recruitment materials connecting the first name of the participant with his or her last name and other personal information will be locked in a cabinet (if in hard-copy form) or password protected (if in electronic form). All electronic files and hard-copies containing identifying information will be destroyed within 4 weeks after CBHSQ approves the final memorandum with the cognitive interview findings.
For those selected for the cognitive interview, recruiters will schedule a time to conduct one-on-one interview appointments at one of the following locations:
the Contractor’s cognitive laboratory facilities,
a substance use treatment facility or
at other private location, such as a private room in a public library or community center.
For interviews conducted with adolescents, parents will accompany adolescents to the interview. Upon arrival at the assigned facility, the interviewer will review the consent form and assent form (Attachment D, Parental Permission Form; Attachment E, Participant Informed Consent/Assent) with both the adolescent and the parent or guardian and will receive verbal consent.
For interviews conducted with adults, interviewers will review the consent form (Attachment E, Participant Informed Consent/Assent Form) with the participant and collect verbal consent. To protect respondent anonymity, the informed consent/assent form will be signed only by the interviewer after receiving verbal consent/assent from the participant. Participants will receive a copy of the consent, assent, and parental consent forms.
Only those respondents who give verbal consent/assent to participate will be interviewed. Participants will also be asked to provide consent to have the interview audio/screen recorded. In the event that authorized project staff are present to observe (remotely or in-person), participants will provide consent for that observation to take place. If participants decline to have the interview recorded or observed, the interview will still be conducted without any recording or observations.
During the cognitive interviews, the age and gender will be collected from all participants but only to inform the interview program which questions to display and to tailor wording. No links to individual participants will be preserved in the cognitive interview memorandum, and personal identifying information will not be included in the data or final memorandum delivered to CBHSQ.
Many of the questions to be tested concern topics likely to be of a sensitive nature, including alcohol use by persons under age 21, use of illegal drugs by participants of all ages, and questions about substance use dependence and withdrawal. Consequently, some questions could be distressing to respondents. The questions included in this SUD cognitive interview are provided in Attachment I, Cognitive Interview Study Questions. The cognitive interview protocol is provided in Attachment F.
Measures will be taken to reduce risks to the respondents. For participants interviewed onsite at treatment facilities, arrangements will be made with the programs to have a counselor on call at the facility during the times when interviews will be conducted, in case any treatment clients participating in the study become upset by the interview questions and want to speak with a counselor. For participants being interviewed at the Contractor’s facilities who become upset with the questions and would like to speak with someone, referral options include a health care provider (if applicable), the Boys Town National Hotline (1-800-448-3000), and the Lifeline Network (1-800-273-TALK [8255]). Participants recruited from outside of substance abuse treatment who request information on substance abuse treatment options will be referred to SAMHSA’s 24-hour toll-free Treatment Referral Helpline (1-800-662-HELP [4357]). These procedures are outlined in the Distressed Respondent Protocol in Attachment J.
In addition, all participants regardless of age will be reminded periodically not to report anything that could identify another person, such as referencing individuals who sold or gave them illegal drugs. After the tutorial, prior to the first interview question (Attachment F, page 1), the interviewer will give examples of ways a participant may talk about another person and examples of ways not to talk about someone. As further protection, the interviewer will interrupt participants who appear ready to report identifying information about themselves or someone else in response to a question/probe.
As noted in Section 10, participants will be assured at all stages of recruiting and interviewing processes that the information they provide is voluntary and will be handled in a confidential manner. These efforts will help participants feel more comfortable with the interview situation and more at ease with the interviewer.
Data from the screening questionnaires, cognitive interviewing protocols, and screen/audio recordings that include sensitive information will be stored in locked cabinets (if in hard-copy form) or password protected (if in electronic form) during the recruiting and interview process. None of this information will be retained once the data have been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.
Approximately 150 individuals will be screened and up to 44 of the screening respondents will be enrolled in cognitive interviews across the three rounds. The targeted final sample size for cognitive interviews is 42 (15 English-speaking adolescents, 15 English-speaking adults, and 12 Spanish-speaking adults), but 1 or 2 additional screening respondents might need to be interviewed if respondents who indicated substance use during the initial screener report no substance use during the interview. Such a scenario, while unlikely to occur, would require conducting a total of up to 44 interviews to reach the final target of 42.
Administration of the screening questionnaire during the recruitment process will take an average of 5 minutes per participant. It is estimated the average amount of time required to conduct each cognitive interview will be approximately 60 to 75 minutes.
The recruitment, cognitive interviewing, and analysis phases for all three rounds of interviews for this study will span approximately 12 months, from May 2018 through May 2019.
The respondent burden for this study is shown in Table 1. The hourly wage of $17.60 was calculated based on weighted data from 2016 NSDUH respondents’ personal annual income reports.
Table 1. Estimated Burden for SUD Cognitive Interviews Study
Activity |
Number of Respondents |
Responses per Respondent |
Average Burden per Response (Hours) |
Total Burden (Hours) |
Hourly Wage Rate |
Total Hour Cost |
Screening |
150 |
1 |
0.083 |
12.45 |
$17.60 |
$219.12 |
Full Cognitive Interviews |
44 |
1 |
1.000 |
44.00 |
$17.60 |
$774.40 |
TOTAL |
150 |
– |
– |
56.45 |
– |
$993.52 |
There are no capital, start-up, operational, or maintenance costs to respondents.
Total costs associated with the cognitive interviews are estimated to be $129,140 over a 12-month period. Of the total costs, approximately $110,619 are for contract costs (e.g., recruiting for, conducting, analyzing, and reporting on cognitive interviews), and approximately $18,521 represents CBHSQ costs to manage the task.
Currently there are 2,741 total burden hours in the OMB inventory. For the DSM-5 cognitive testing, SAMHSA is requesting 56.45 burden hours.
These SUD cognitive interviews will be used to test the wording changes and question additions being considered for the NSDUH questionnaire. The sample size and design do not allow for statistical inference to be conducted; therefore, the analyses will be qualitative. Debriefings with the cognitive interviewers will be conducted to learn from their experiences regarding participants’ reactions and responses to the survey questions and interviewer probes. The results will be summarized in a memorandum and used to make recommendations for questions to be revised and/or included for the NSDUH.
The schedule for the cognitive interviews for the NSDUH SUD module cognitive interview study is included in Table 2.
Table 2. Schedule for SUD Cognitive Interviews Study
Subtask |
Date |
Recruiting for cognitive interviews begins |
5/11/2018 |
Round 1 cognitive interviews begin |
5/22/2018 |
Recruiting for Round 2 cognitive interviews begins |
9/06/2018 |
Round 2 cognitive interviews begin |
9/13/2018 |
Recruiting for Round 3 cognitive interviews begins |
11/20/2018 |
Round 3 cognitive interviews begin |
11/20/2018 |
All cognitive interviews completed |
12/07/2018 |
Final cognitive interview memorandum delivered to SAMHSA |
12/21/2018 |
The OMB expiration date will be displayed on the Parental Permission Form (Attachment D) and Participant Informed Consent/Assent Forms (Attachment E).
The certifications are included in this submission and fully comply with 5 CFR 1320.9.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | National Survey on Drug Use and Health: SUD Module Redesign Study |
| Author | RTI International |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |