Form Evaluation Plan Evaluation Plan Evaluation Plan

This instrument is not OMB approved and is provided for informational purposes only while the instrument is under OMB review.
The final items are subject to change as part of the OMB approval process.

SAMHSA Strategic Prevention Framework for Prescription
Drugs (SPF Rx) Evaluation Plan Checklist
This checklist serves multiple purposes: (1) It provides guidelines for developing a well-rounded evaluation plan;
(2) it specifies evaluation design elements that are required for the Strategic Prevention Framework for
Prescription Drugs (SPF Rx) grant program; and (3) it provides SAMHSA and the national cross-site evaluation
team with information about each grantee’s evaluation design. Design elements that are not required are noted.
However, if an optional item is included in your evaluation, it should be addressed in the plan. Please note that
the evaluation plan is intended as a stand-alone document—that is, information included in other documents
submitted to SAMHSA, such as grant proposals and impact statements, should be discussed in the plan as
appropriate.
Note that tribal and jurisdiction grantees who decide to define themselves as single-community grantees need
this decision approved by their CSAP Project Officer and should make this clear in their evaluation plans. In the
items below, instructions specific to single-community grantees are provided where relevant.

I.

SPF-Rx Evaluation Goals
The goals should address the CSAP grant program requirements as well as any other goals specific to your
State, jurisdiction, or tribal organization. Do the evaluation goals address…
Questions
1.

Yes

Preventing the onset and reducing the progression of prescription drug
misuse, or illicit opioid misuse?
Please specify the targeted substance (check all that apply):

No

Page
Reference

_____

Prescription drugs
Illicit opioids
2.

Reducing prescription drug-related consequences among adolescents
and adults?
Please specify the targeted age range (check all that apply):

_____

12 to 17
18 to 20
21 to 25
26 to 34
35 to 44
45 to 54
55 to 64
65 and over
Other (optional) _________________________________
3.

Assessing the current use of PDMP and its component as it relates to
accessibility of data, usage by communities, and outcomes related to
previous PDMP use.

_____

4.

Conducting continuous strategic planning, including assessment,
monitoring, and analysis of the program to meet the desire outcomes.

_____

5.

Coordinating funding streams by leveraging funds from other sources
(including, for states, the prevention set-aside of the Substance Abuse
Prevention and Treatment Block Grant (SABG).

_____

6.

Implementing organizational/structural changes that create focus of
responsibility for specific PDMP issues/populations, address behavioral
health disparities, or increase access to, or efficiency of, services

_____

Questions

Yes

Page
Reference

No

7.

Developing interagency coordination mechanisms that improve the
effectiveness of the partnerships between PDMPs; federal, state, tribal,
regional, and local agencies; private and non-profit organizations;
hospitals; pharmacies; and health professional and advocacy groups

_____

8.

Adopting and implementing policy (i.e. programs and guidelines) that
facilitate establishment of standards of care, adherence to the National
Standards for Culturally and Linguistically Appropriate Services (CLAS) in
Health and Health Care, and development/revision of credentialing,
licensure, or accreditation requirements

_____

9.

Demonstrating measurable quality improvement on all aspects of
access to, and usage of the PDMP data, as well as coordination with
high need communities to achieve the goals and objectives of the
program.

_____

10. Strengthening workforce development around activities to ensure use
of PDMP data (i.e., training, support for licensure, credentialing, and
accreditation).

_____

11. Other goals? (optional) ___________________________________

_____

Comments related to Goals:

II.

Evaluation Questions
Do the evaluation questions address…
Questions

Yes

No

Page
Reference

1.

Process by describing how the grantee progressed through the SPF steps;
which promising or evidence-based programs, policies, and practices
(EBPPP) were implemented; how fidelity of implementation was ensured;
and how changes from the original plan affected performance?

_____

2.

Process by describing how the grantee developed a community-based
social marketing/public education plan to increase awareness of
prescription drug misuse issues, the need for a coordinated approach, and
promote increased use of PDMP data?

_____

3.

Outcomes by reporting on the effects of the SPF Rx initiative on the
consumption, intervening variables, and consequences as listed in the
Required Outcome Measures Section (IV)?

_____

Comments related to the Evaluation Questions:

2

III.

Required Performance Measures
Grantees are required to report certain data so that SAMHSA can monitor performance and evaluate its
programs. Note that single-community grantees without subrecipient communities will skip question 2
below. Single-community grantees without subrecipients are grantees for whom the SPF process occurs at
the grantee level, not at the level of smaller areas within the grantee's jurisdiction. While such grantees
may contract with community agencies to implement the selected interventions, the planning and
interventions affect the community as a whole. See the Single-Community Grantee Supplement to decide
whether you are considered a single-community grantee with or without subrecipients.
Questions
1.

2.

3.

Does the grantee represent a single community without subrecipients
(and will therefore report data at only one level)? If yes, skip to Q3 in this
section.1

Yes

No

Page
Reference

_____
Skip to
Q3

Does the evaluation plan include the following required grantee-level
performance measures?
a.

Incorporation of PDMP data into your needs assessment in
developing your strategic plan

_____

b.

Measure of reductions in opioid overdoses

_____

Does the evaluation plan include the following required performance
measures at the community level and program level where appropriate?
a.

Number of active collaborators/partners supporting the grantee’s
comprehensive prevention approach

_____

b.

Number of people served and/or reached by Institute of Medicine
(IOM) category (universal, selected, indicated), six strategies, and
demographic group. (This should include target and numbers reached
for public education programs.)

_____

c.

Number and percent of promising or evidence-based programs,
policies, and/or practices implemented by subrecipient communities

_____

d.

Number of prevention activities at the subrecipient level that are
supported by collaboration

_____

e.

Number, type, and duration of promising or evidence-based
interventions by prevention strategy implemented at the community
level

_____

1

Single-community grantees with subrecipients should complete questions 2 and 3 in this Required Performance
Measures section.

Comments related to the Required Performance Measures:

3

IV. Required Outcome Measures
Which of the following outcome measures are included in the evaluation plan at both the grantee and
subrecipient community levels (if different from the grantee)? Grantees and subrecipient communities
must report all of the outcome measures below, with a couple of exceptions. Please note that NSDUH
State-level consumption measures and CDC WONDER state and county overdose death data will be
provided by the cross-site evaluation team.
Single-community grantees (with or without subrecipients) should respond only to the Subrecipient
Communities column.
Outcomes
a.

♦
♦
♦

Page
Reference

Past-30-day nonmedical use of prescription
drugs
Past-12-month nonmedical use of
prescription drugs
Past-30-day nonmedical use of prescription
pain relievers
Past-12-month nonmedical use of
prescription pain relievers

_____
_____
_____
_____

Proximal Outcomes: Prescribing Patterns2
♦
♦
♦
♦
♦
♦

♦
♦

c.

Subrecipient
Not
Communities1 Applicable

Consumption (if survey data are available)
♦

b.

Grantee

Total number of unique residents
prescribed opioid analgesics
Number of opioid analgesic prescriptions
Number of high-dose opioid analgesic
prescriptions (>90 MME/day)
Number of opioid pills dispensed
Average MME/day for all opioid
prescriptions dispensed in this period
Percentage of patient prescription days with
overlapping opioid and benzodiazepine
prescriptions
Number of multiple provider episodes
(indicator of possible doctor shopping)
Other optional indicator targeted by
grantee or community (specify)
___________________________________

_____
_____
_____
_____
_____
_____
_____
_____

Proximal Outcomes: Prescriber Use of PDMP
♦
♦
♦

Number of prescribers registered with the
PDMP
Number of prescribers (or their delegates)
who queried the PDMP
Number of queries by prescribers (or their
delegates) to PDMP

_____
_____
_____
(continued)

4

Outcomes
d.

Subrecipient
Communities1

Not
Applicable

Page
Reference

Consequences
♦
♦
♦

1

Grantee

_____

Opioid overdose emergency department
(ED) visits
Opioid overdose-related hospital
admissions (if ED visits not available)
Opioid overdose-related deaths (states and
counties do not report deaths)

_____
_____

Or single-community grantees.

2

Grantees report on all 7 indicators for prescribing patterns, whereas subrecipient communities report on the first
5 and can choose one additional indicator, if feasible.

Comments related to the Required Outcome Measures:

V.

Measurement
How will substance use, intervening variables, and consequences be measured?
Question

Yes

No

Page
Reference

1.

For each measure, does the evaluation plan include the source of the
data, the frequency, and the method of data collection? (See Exhibits 1
and 2 for examples. Single-community grantees see Exhibit 3 for
examples.)

_____

2.

Is information provided about the sampling plan that was used to collect
data? For example, were survey data collected from a random sample of
individuals or were data collected from a census of all students in school
on the day of survey administration?

_____

3.

Is information provided about the survey response rate? For example, if
100 students completed the survey from a school with 200 eligible
students, the response rate would be 50%.

_____

4.

Is information provided about the reliability and validity of the measures?
Please note whether the measure is based on a standard instrument (e.g.,
NSDUH, MTF, YRBS, CTCYS). If yes, please briefly note that fact. If the
measure is being newly developed, please note how validity and reliability
will be established.

_____

Comments related to Measurement:

5

Exhibit 1.

Example Grantee-Level Measures for SPF-Rx
Measure

Source

Frequency
Collected

Method of Collection

Level of Data

Consumption: Past-30-day
nonmedical use of prescription drug

School survey

Annual

In-person collection in public
schools

State

Proximal Outcomes: Opioid
Prescribing Pattern

PDMP

Annual

Administrative Data

State

Consequence: Opioid Overdose

Emergency
Room Data

Annual

Administrative data

State

Exhibit 2.

Example Subrecipient Community-Level Measures for SPF-Rx
Measure

Source

Frequency
Collected

Method of Collection

Level of Data

Consumption nonmedical use of
prescription drug

School survey

Annual

In-person collection in public
schools

All public schools
in county

Proximal Outcomes: prescriber use
of PDMP in their subrecipient
communities

PDMP

Annual

Administrative Data

Community,
County, or District

Consequence: Opioid Overdose

Emergency
Room Data

Annual

Administrative data

County

Exhibit 3.

Example Single-Community Grantee Measures for SPF-Rx
Measure

Source

Frequency
Collected

Method of Collection

Level of Data

Consumption nonmedical use of
prescription drug

School survey

Annual

In-person collection in public
schools

All public schools
in the grantee’s
jurisdiction or
service area

Proximal Outcomes: prescriber use
of PDMP in their subrecipient
communities

PDMP

Annual

Administrative Data

Community,
County, or District

Consequence: Opioid Overdose

Emergency
Room Data

Annual

Administrative data

6

VI. Behavioral Health Disparities
SAMHSA requests that grantees use their data to (1) identify subpopulations (e.g., racial, ethnic,
sexual/gender minority groups) vulnerable to disparities; and (2) implement interventions to decrease the
differences in availability of, access to, use of services, and health outcomes among these vulnerable
subpopulations.
Question
1.

Yes

No

Page
Reference

Does the plan address behavioral health disparities affecting any of the
following? You must identify at least one disparate population. If your disparate
population is not listed, please add as appropriate.
a.

Racial/ethnic minority group

b.

Sexual/gender minority group

c.

A single tribe

d.

Specific populations within a tribal or jurisdiction grantee

e.

A specific community within a tribal or jurisdiction grantee

f.

Tribal members in urban areas or reservations

g.

Other vulnerable population (specify below)

2. Does the evaluation plan discuss a clear measure of access to culturally and
linguistically appropriate services (such as the level of adherence to the National
CLAS standards
[https://www.thinkculturalhealth.hhs.gov/pdfs/enhancednationalclasstandards.pdf]
by service providers, percent of the population reached for each vulnerable group,
or number of interventions implemented by type for each vulnerable group)?
3. Does the evaluation plan include an assessment of outcomes for the identified
vulnerable groups?

_____
_____
_____
_____
_____
_____
_____

_____

_____

Comments related to Behavioral Health Disparities:

7

VII. Analysis Plan
Does the evaluation plan include…
Questions
1.

2.

Yes

No

Page
Reference

Description of the analyses to be used for process evaluation questions.
a.

Do analyses include description of implementation?

b.

Do analyses include measures of fidelity?

c.

Do analyses include measures of dosage?

_____
_____
_____

The types of analyses to be used for outcome evaluation questions.
a.

Do the analyses include pre- and post-test comparisons? (optional)

b.

Do analyses include a control or comparison group? (optional)

c.

Do analyses explore differences in trends over time?

d.

Do analyses include statistical procedures to measure change?

e.

Do analyses provide the expected sample sizes for grantee and
community-level measures?

_____
_____
_____
_____
_____

Comments related to the Analysis Plan:

VIII. Participation in SPF Rx National Cross-Site Evaluation
Questions
1.

Yes

No

Page
Reference

_____

Does the plan mention collection and submission of data via the cross-site
evaluation instruments at the grantee and community levels?

Comments related to the National Cross-Site Evaluation:

IX. Reporting Plan
(Note that a local evaluation report is required for SPF-Rx.) Does the evaluation plan include…
Question
1.

A plan for how the data will be disseminated to inform key stakeholders?

2.

The frequency of communication?

3.

A plan for collection and submission of data for the local evaluation at the
grantee and community levels?

4.

How you will use information from this evaluation plan to inform your
local evaluation efforts?

Yes

No

Page
Reference

_____
_____
_____
_____

Comments related to the Reporting Plan:

8