DFC Core Measures Alignment to NOMS

Attachment 6:

Drug Free Communities Core Measures Alignment to NOMS

Drug Free Communities National Evaluation

Alignment of Core Measures to NOMS

July 19, 2011

Summary

The Drug Free Communities National Evaluation will focus on four core measures for four substances. These core measures have been modified in part to conform to NOMS standards and in part to maintain comparability to legacy data. Proposed core substances will include alcohol, tobacco, and marijuana (as in the past), and prescription drugs will be added as a core substance. Proposed core measures for the DFC National Evaluation include:

• 30-Day Use: This core measure will remain a dichotomous metric. Although the ideal NOMS question focuses on the number of days used in the past 30 days, SAMHSA’s reporting of this metric at the community level focuses on prevalence (which treats the NOMS question as a yes/no proposition). By keeping 30-day use as a dichotomous measure, the DFC National Evaluation will not need to re-baseline its evaluation findings.

• Perception of Risk: This will be aligned with the NOMS, which involves a movement to a binge drinking question for Perception of Risk of Alcohol. Binge drinking is a more appropriate metric since it approximates a more common pattern of use among teenagers than drinking every day. Because the perception of regular alcohol use (i.e., nearly every day) is a key legacy metric for STOP Act grantees, we intend to keep this metric in COMET as a requirement for STOP Act and an optional question for non-STOP Act grantees.

• Perception of Disapproval: Perception of parental disapproval will remain as a core measure, and a set of questions on peer disapproval will also be added. By measuring perceptions from the two largest influences in a youth’s life (family and peers), we expect to develop a more well-rounded view of how perceptions influence substance use.

• Age of Onset: These questions will be dropped from the core measures. There are numerous well-documented problems with measuring age of onset that bring into question the reliability of this metric.

To summarize, the core measures for DFC are (1) 30-day use, (2) Perception of Risk, (3) Perception of Parental Disapproval, and (4) Perception of Peer Disapproval.

Background

On June 9, 2010 ONDCP convened a meeting to discuss proposed changes to the COMET system, which is the point of entry for the four core measures of the DFC National Evaluation. During this meeting, SAMHSA's DACCC contractor (HSRI) notified the national evaluation team that some of the current core measures for the DFC evaluation -- including 30-day use -- were not in compliance with SAMHSA's National Outcome Measures (NOMs). ICF resolved to make this change.

On June 10, 2011, ONDCP convened a conference call with SAMHSA and ICF International to discuss proposed changes to the core measures. During this call, SAMHSA indicated that the DFC National Evaluation’s measure of 30-day use was in compliance with the NOMs. SAMHSA indicated that although the "ideal" NOMs question focuses on the number of days used in the past 30 days (a continuous measure), it was acceptable for DFC grantees to report the prevalence of 30-day use (a dichotomous measure). In short, there appears to be a distinction between compliance in reporting NOMs and compliance in using ideal NOMs questions in data collection efforts. From our conversation with SAMHSA, we believe that the key litmus test in NOMs compliance is maintaining fidelity to NOMs reporting conventions. There is some flexibility in the application of NOMs questions as long as the data can be wrapped up to be compliant with NOMs reporting.

On July 8, a meeting was held between ONDCP, SAMHSA, and ICF to finalize the core measures. During this call, all parties came to agreement that the four core measures should be (are (1) 30-day use, (2) Perception of Risk, (3) Perception of Parental Disapproval, and (4) Perception of Peer Disapproval. Age of First Use will be dropped from COMET due to numerous problems that call into question the reliability of the data. Moreover, it is not a current GPRA measure for STOP Act so there is no consequence for dropping it.

This memo serves to clarify understanding of the DFC National Evaluation’s current state of NOMs compliance, and how our proposed changes to the COMET system will affect NOMs compliance in the future.

Recommendations for Changes to the Core Measures

Table 1 presents an alignment of (a) current core measures used for the DFC National Evaluation, (b) current NOMs questions, (c) proposed changes to the DFC core measures, (d) other considerations in making changes to the core measures, and (e) an assessment of whether the current DFC measures are NOMs compliant. A summary of proposed changes appears below.

• Addition of Prescription Drugs as a Core Substance: The primary change to core measures reporting is the addition of prescription drugs as a core substance. Although this is not a change in the core measures, per se, it does necessitate the addition of core measures questions to gather the required data. The need for questions about prescription drugs is clear. The broad availability of prescription drugs and misperceptions about their dangers is an alarming combination, especially given that:

• New users of prescription drugs are relatively equal in number to new users of marijuana;

• Teens are abusing prescription drugs because they believe the myth that these drugs provide a medically safe high;

• Painkillers, such as OxyContin and Vicodin, are the most commonly abuse prescription drugs by teens.¹

While the incorporation of painkillers will add burden to grantees, for the great majority that additional burden will be minimal. Many existing surveys (e.g., Pride, CTC, Monitoring the Future, ADAS) already have questions on 30-day use of prescription drugs.² Where an item can be added to a survey without substantial resistance, the reporting of an additional single item is little burden. The major addition of burden will be experienced if adding an item requires substantial effort.

With respect to item wording, we originally intended to separate painkillers from other prescription drugs. However, it was clear from our discussions with DFC grantees that youth were oftentimes taking pills they couldn’t identify (e.g., at “pill parties”). Therefore, we recommend keeping this question anchored on the use of prescription drugs not prescribed to the youth. Moreover, we considered using the phrase “to get high” in the stem of the questions on prescription drugs, but the widespread use of Ritalin and Adderall to improve concentration obviated the need to include that qualifier. In short, we would like to capture a wide-ranging measure of prescription drug use, not limited by the purpose of taking the drug or the type of high it produces.

• Keep 30-Day Use as a Dichotomous Measure: Evidence indicates that the amount of use is a key correlate of negative outcomes of substance use.³ The proposed change in the ideal NOMs use measure to report the number of days used in the past 30 days will improve measurement of amount. However, using that item would necessitate that we re-baseline our key outcome variable in the evaluation. Moreover, moving to a “number of days used” measure would create some difficulties for the standardization of data across grantees:

• There are four primary “flavors” of 30-day use questions in different surveys: (1) the number of days used in the past 30 [YRBS Wisconsin, Michigan, Alaska, Ohio, Maryland], (2) the number of occasions used in the past 30 days [CTC, Iowa, Illinois], (3) the number of times used in the past 30 days [Indiana, ADAS], and (4) the last time a given substance was used [County surveys]. By moving to a continuous measure of the number of days used, a large number of DFC grantees that base responses on the number of occasions and times used will no longer be technically compliant with reporting requirements.⁴

• Moving a large number of DFC grantees to a new measure of 30-day use will interrupt our ability to track this key outcome over time. Re-baselining data would also damage our ability to draw upon the past six years’ of data already in the COMET system.

• There is a problem with the denominator of the “number of days used” measure. Some questionnaires use a skip pattern to identify the number of days used (e.g., Did you use in the past 30 days? If yes, how many days did you use?), while others do not (e.g., How many days in the past 30 did you use?). In the former example, the number of days used would be based only on users, while in the latter example, the number of days used is based on the entire sample. While it is a relatively simple matter to put these two types of questions on the same scale, it would require some readjustment on grantees’ part – many of whom do not have local evaluators.

Given the complications listed above, we believe that it is preferable to focus on prevalence of use instead of the amount of use. This will allow the evaluation to maintain consistency on its key outcome of interest and will ultimately provide the most consistent data reporting on this critical variable.

• Change perception of risk for alcohol to a measure of binge drinking; define perception of risk for marijuana use more clearly: The current measure for perception of risk for alcohol is based on having one or two drinks nearly every day. In light of research on the benefits of reservatrol, youth may perceive moderate alcohol use to be beneficial. While this may be the case, it doesn’t capture the realities of youth drinking which is the focus of our evaluation. By refocusing on binge drinking – which would put the DFC evaluation in NOMs compliance on this measure – we could focus on youths’ risk perception of drinking that is unequivocally harmful. However, because the perception of regular alcohol use (i.e., nearly every day) is a key legacy metric for STOP Act grantees, we intend to keep this metric in COMET as a requirement for STOP Act and an optional question for non-STOP Act grantees.

The key problem with moving to a measure of binge drinking for alcohol is that the majority of grantee surveys (e.g., YRBS, Pride, CTC, Illinois, Wyoming, New Hampshire, Vermont) measure perception of risk with our original measure of 1-2 drinks nearly every day. The implication of this change to the NOMs is that we may force many local, state, and national surveys to re-baseline as well. On the other hand, such standardization is the purpose of the NOMs in the first place. The upshot is that there will be a significant transition period for this measure, and it would be helpful to know whether SAMHSA plans to anchor perception of risk on binge drinking in the future.

We also investigated whether it was common for youth surveys to pose binge drinking differently for boys (5 or more drinks 1-2 times a week) and girls (4 or more drinks 1-2 times a week). Very few surveys make this distinction in binge drinking (e.g., Dover Youth Survey), and we therefore feel that it would minimize burden to keep the 5 drink standard for both boys and girls.

We believe that the NOMs measure of marijuana use is also preferable to our current measure because it defines regular use of marijuana as 1-2 times per week. The current guidance on the core measures indicates that regular use of marijuana is not defined. This creates too much room for interpretation on the grantees’ part, and we feel it is not in the evaluation’s best interest to leave an important concept undefined. Unlike 30-day use, which may be NOMs compliant if reported in specific ways, the fundamental differences in our original questions and the NOMs questions indicates that we are not in NOMs compliance by any measure. By moving to the NOMs, we hope to improve the comparability of DFC data with other measurement efforts – and we hope to abide by the gold standard in prevention survey research.

• Keep perception of parental disapproval questions and add questions about peer disapproval. Although the current measure of parental disapproval suffers from problems (e.g., parental disapproval suffers from ceiling effects), we have observed that 30-day use does move up when parental disapproval drops. By including peer disapproval, we can ensure more variance in the data while providing a meaningful measure that supplements parental disapproval. Together, both peer and parental disapproval encompass the influences in a child’s decision to use ATOD. Peer disapproval will be framed in the following way: “How do you feel about someone your age…”.

• Drop age of first use questions. The current age of onset questions suffer from weaknesses in reporting (e.g., youth may be asked to recall an event that happened years earlier). Age of first use is no longer a GPRA measure for STOP Act, and we do not feel that data of questionable validity is worth the additional burden in asking this question.

Implications for GPRA Reporting

The following performance measures have been used to meet GPRA reporting requirements:

• Percent of coalitions that report a decrease in at least one targeted risk factor.

• Percent of coalitions that report an increase in at least one targeted protective factor.

• Percent of coalitions reporting at least 5% improvement in past 30-day alcohol, tobacco, or marijuana use in at least one grade.

• Percent of coalitions that report positive change in youth perception of risk from alcohol, tobacco, or marijuana in at least two grades.

• Percent of coalitions that report positive change in youth perception of parental disapproval of the use of alcohol, tobacco, or marijuana in at least two grades.

• Percent of coalitions reporting positive change in age of initiation of alcohol, tobacco, or marijuana in at least one grade.

By moving to the proposed core measures, two of our current GPRA measures will have to be re-baselined: (1) Perception of risk from alcohol and marijuana and (2) Perception of peer disapproval for all substances. Moreover, the age of initiation measure will be dropped. Because these GPRA measures are based on positive movements in data, there will be only a one-year lag before the new GPRA measures are reported.

Table 1: Alignment of the Core Measures to the NOMs
Indicator	Current Measure	NOMs Measure	Proposed Changes	Notes	Current Measure NOMs Compliant?	Proposed Measure NOMs Compliant?
30-Day Use	During the past 30 Days did you... Drink one or more drinks of an alcoholic beverage Smoke part or all of a cigarette Use marijuana or hashish Response Categories Yes No	During the past 30 days, on how many days did you... Drink one or more drinks of an alcoholic beverage Smoke part or all of a cigarette Use marijuana or hashish Response Categories: ___ Days	None, but add prescription drugs as a core substance. We would prefer to require a single measure of 30-day use. The "amount of use" question may present challenges in data collection due to skip patterns in surveys. For example, if all youth are asked about the amount of use, we can get a grand mean, but if the question is delivered in two parts (did you use? if so, on how many days?), this may present an issue in reporting.	Although the current measure is not on the same scale as the NOMS question, it is reported in a manner congruent with NOMS reporting. We need to figure out whether the litmus test for NOMS compliance is adherence to the question or to the manner in which the question's results are reported.	No; possibly NOMs compliant for reporting purposes	No; possibly NOMs compliant for reporting purposes
Perception of Risk	How much do you think people risk harming themselves (physically or in other ways) if they: Take one or two drinks of an alcoholic beverage (beer, wine, liquor) nearly every day? Smoke one or more packs of cigarettes per day? Smoke marijuana regularly? Response Categories: No risk Slight risk Moderate risk Great risk	How much do you think people risk harming themselves physically or in other ways when they... Have five or more drinks of an alcoholic beverage once or twice a week? Smoke one or more packs of cigarettes per day? Smoke marijuana once or twice a week? Response Categories: No risk Slight risk Moderate risk Great risk	Add prescription drugs as a core substance. Change perception of risk for alcohol to binge drinking, as specified in the NOMs, instead of regular use. However, keep legacy measure as a requirement for STOP Act grantees only. Change marijuana frequency to “once or twice a week”, as specified in the NOMs. Currently, regular use of marijuana is not defined, which leaves too much room for individual interpretation.	The regular use measure for alcohol has become problematic in light of research about the health benefits of regular alcohol use (in moderation). A better measure – which is more congruent to the realities of youth alcohol use – would focus on binge drinking. A very small percentage of youth are regular users of tobacco (the 2009 YRBS indicates that 7.3% of youth smoked cigarettes on at least 20 of the past 30 days).	No for alcohol; yes for tobacco and marijuana	Yes
Perception of Disapproval	How wrong do your parents feel it would be for you to: Drink beer, wine, or hard liquor (for example, vodka, whiskey, or gin) regularly? Smoke cigarettes? Smoke marijuana? Response Categories: Not at all wrong A little bit wrong Wrong Very wrong	How do you feel about someone your age: Having one or two drinks of an alcoholic beverage nearly every day? Smoking one or more packs of cigarettes a day? Trying marijuana or hashish once or twice? Using marijuana once a month or more? Response Categories: Not at all wrong A little bit wrong Wrong Very wrong	Add prescription drugs as a core substance. Keep parental disapproval questions and add peer disapproval as a core measure.	Parental disapproval covaries with 30-day use and therefore provides a solid link between attitudes and behaviors. Peer disapproval questions supplement parental disapproval and provide information on what are most often the most powerful influences in a child’s life: family and peers.	No	Yes
Age of Onset	How old were you when you first: Had more than one sip or two of beer, wine, or hard liquor (for example, vodka, whiskey, or gin)? Smoked a cigarette, even just a puff? Smoked marijuana? Response Categories: 10 or younger 11 12 13 14 15 16 17 or older	How old were you the first time you: Had a drink of an alcoholic beverage? Smoked part or all of a cigarette? Used marijuana or hashish? Response Categories: 10 or younger 11 12 13 14 15 16 17 or older	Drop question: Data on age of first use are not reliable and this question poses the largest burden to DFC grantees, since average age of first use needs to be calculated from a set of varying response categories.	Average age of onset is the least reliable and most difficult to interpret of the core measures. It is confounded by both the age of respondent at time of measurement (i.e., younger samples have ceiling effects on the available age reporting first use) and by the lag between first use and implementation of the coalition’s activities (i.e., coalitions cannot affect use that occurred prior to the program being implemented). This measure also relies on the youth being surveyed to recall an event that may have happened years before.	Yes	N/A

Drug Free Communities (DFC) Support Program Evaluation of Core Measures Survey

I. General Information

II. 30-Day Use

1. Sex:  Male  Female

2. Grade:  6  7  8  9  10  11  12

3. Age  10 years old or less  11 years old  12 years old  13 years old  14 years old  15 years old  16 years old  17 years old  18 years old  19 years old or more

Yes

No

Have you had alcoholic beverages (beer, wine, cocktails, hard liquor, etc.) to drink – more than just a few sips – during the past 30 days?





Have you smoked cigarettes during the past 30 days?





Have you used marijuana during the past 30 days?





Have you used prescription drugs not prescribed to you during the past 30 days?





III. Perception of Risk
	No Risk	Slight Risk	Moderate Risk	Great Risk
How much do you think people risk harming themselves (physically or in other ways) if they take five or more drinks of an alcoholic beverage (beer, wine, liquor) once or twice a week?				
Required for STOP Act Grantees: How much do you think people risk harming themselves (physically or in other ways) if they take one or two drinks of an alcoholic beverage (beer, wine, liquor) nearly every day?				
How much do you think people risk harming themselves (physically or in other ways) if they smoke one or more packs of cigarettes per day?				
How much do you think people risk harming themselves (physically or in other ways) if they smoke marijuana once or twice a week?				
How much do you think people risk harming themselves (physically or in other ways) if they use prescription drugs that are not prescribed to them?				

IV. Perception of Parental Disapproval
	Not at all wrong	A little bit wrong	Wrong	Very wrong
How wrong do your parents feel it would be for you to have one or two drinks of an alcoholic beverage nearly every day?				
How wrong do your parents feel it would be for you to smoke one or more packs of cigarettes a day?				
How wrong do your parents feel it would be for you to try marijuana once or twice a week?				
How wrong do your parents feel it would be for you to use prescription drugs not prescribed to you?				

V. Perception of Peer Disapproval
	Neither Approve Nor Disapprove	Somewhat Disapprove	Strongly Disapprove	Don’t Know/ Can’t Say
How do you feel about someone your age having one or two drinks of an alcoholic beverage nearly every day?				
How do you feel about someone your age smoking one or more packs of cigarettes a day?				
How do you feel about someone your age using marijuana once a month or more?				
How do you feel about someone your age using prescription drugs not prescribed to them?				

1 Office of National Drug Control Policy (2007). Teens and prescription drugs: An analysis of recent trends on the emerging threat. Washington, DC: Author.

2 YRBS, however, does not include a question on painkillers.

3 In ICF’s work on substance abuse treatment programs, we found that clients who relapsed were using less drugs than at baseline. While this is hardly a positive finding since many clients relapsed, it does indicate that abstinence rates and amount of use are two distinct measures that should ideally be applied in tandem.

4 Arguably, we could treat times, occasions, and days as equivalent, but this would be a second-best option.