Program Developer Protocol

l
OMB No: XXXX-XXXX
Expiration Date: X/XX/XXXX

[Apply label here]

Study of the Implementation of the
Safe and Drug-Free Schools and
Communities Act Program State Grants

Sponsored by:
The U.S. Department of Education
Conducted by:
Westat

Program Developer Protocol

This protocol is designed to be used in telephone interviews with program developers.
Name of program: _________________________________
Name of person interviewed: ________________________ Telephone: _________________
Title/position: ____________________________________ E-Mail: ____________________

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it
displays a valid OMB control number. The valid OMB control number of this information collection is [XXXX - XXXX]. The
time required to complete this information collection is estimated to average 45 minutes per response, including the time to
review instructions, search existing data resources, gather the data needed, and complete and review the information collected. If
you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to:
U.S. Department of Education, Washington, DC 20202-4651. If you have comments or concerns regarding the status of your
individual submission of this form, write directly to: Debbie Alexander, Westat, 1650 Research Boulevard, Rockville, MD
20850-3129.

Study of the Implementation of the Safe and Drug-Free Schools and
Communities Act Program State Grants

Introduction
NOTE TO INTERVIEWER: THROUGHOUT THIS PROTOCOL, YOU WILL SEE NOTES TO INTERVIEWER IN UPPER CASE AND
ITALICS.

PLEASE LEARN AS MUCH ABOUT THE PROGRAM AS POSSIBLE BEFORE THE INTERVIEW.

PLEASE USE THE FOLLOWING AS A GENERAL INTRODUCTION. YOU DO NOT HAVE TO READ THE SCRIPT VERBATIM

Hi. My name is ____________________ and I’m calling on behalf of the Department of
Education. I am part of the Westat/ISA Associates team conducting the study of the
implementation of research-based programs as part of the Safe and Drug-Free Schools and
Communities Act. As mentioned in the e-mail message we sent to you on the study, we are
contacting program developers like yourself to learn as much as we can about the prevention
curriculum they have developed and how that curriculum should be implemented in schools.
This information will not only provide valuable information about the prevention curriculum,
but also help us to assess the extent to which schools are implementing the program with
fidelity. Would you be the best person to speak with about _______________________
curriculum?

IF NOT, GET CONTACT INFORMATION FOR APPROPRIATE PERSON.

Name of person interviewed: ________________________ Telephone: _________________
Title/position: ____________________________________ E-Mail: ____________________

Is now still a good time to talk?

IF NECESSARY, RESCHEDULE THE INTERVIEW.

Do you have any questions before we begin?

2

If we could start with a little bit of history, that would be great.
TO THE EXTENT THAT YOU ARE FAMILIAR WITH THE PROGRAM, FEEL FREE TO PROVIDE WHAT INFORMATION YOU KNOW.
FOR EXAMPLE, “I UNDERSTAND THAT ______________ WAS FIRST DEVELOPED IN 1999, IS THAT CORRECT?

CAN YOU

TELL ME A BIT MORE ABOUT THAT?”

OR YOU COULD USE THE FOLLOWING:

“CAN YOU TELL ME A BIT ABOUT WHEN AND HOW THE PROGRAM WAS INITIALLY

DEVELOPED AND FOR WHOM IT WAS INITIALLY DEVELOPED?”

NOTE: THESE AREAS SHOULD BE COVERED IN THIS SECTION OF THE INTERVIEW. USE PROBES AS NECESSARY:

ƒ HAS THE PROGRAM CHANGED SINCE THE ORIGINAL DEVELOPMENT?
ƒ HAVE DIFFERENT VERSIONS OF THE PROGRAM BEEN DEVELOPED?
PROGRAM WERE DEVELOPED AND ARE STILL IN USE?

IF SO, HOW MANY DIFFERENT VERSIONS OF THE

ƒ IF MULTIPLE VERSIONS, WHAT IS THE NATURE OF THESE VERSIONS?
¾

DIFFERENT TARGET POPULATIONS (E.G., AGE OR GRADE). IF YES, HOW MANY DIFFERENT VERSIONS
WERE DEVELOPED AND FOR WHICH POPULATIONS? WHEN WERE THEY DEVELOPED?

¾

DIFFERENT “GENERATIONS” OF THE SAME PROGRAM (I.E., UPGRADES OR ENHANCEMENTS TO THE
CURRICULUM). IF YES, HOW MANY GENERATIONS WERE DEVELOPED? WHEN WERE THEY
DEVELOPED?

¾

LONGER OR SHORTER VERSIONS OF THE PROGRAM. IF YES, HOW MANY DIFFERENT VERSIONS, AND IN
WHAT WAY ARE THEY DIFFERENT? WHEN WERE THEY DEVELOPED?

¾

ANY OTHER DIFFERENCES NOT NOTED ABOVE.

ƒ IF MULTIPLE VERSIONS, DO THEY HAVE DIFFERENT NAMES?

IF YES, WHAT ARE THOSE NAMES

ƒ IF MULTIPLE VERSIONS, ARE ALL VERSIONS STILL IN USE BY SCHOOLS? HAVE SOME VERSIONS BEEN DISCONTINUED OR
REPLACED BY UPDATED VERSIONS? DO YOU KNOW HOW MANY SCHOOLS ARE USING EACH VERSION?
ƒ IF MULTIPLE VERSIONS, IS BLENDING ELEMENTS OF DIFFERENT VERSIONS EVER OK? FOR EXAMPLE, USE
TH
COMPONENTS FROM AN OLDER VERSION AND PUT THEM IN A NEWER VERSION? CAN YOU PROVIDE 7 GRADERS WITH
TH
THE 5 GRADE VERSION OF THE PROGRAM?

3

Now, I’d like to get into some of the nuts and bolts of the curriculum. The next set of questions will focus on the
target population, length and duration of lessons, content topics, allowable modifications, implementation strategies
and staff, and training and technical assistance. Our goal is to learn as much as we can about the program so that
when we get information about how schools are implementing the program we can figure out if the schools are
implementing the program the way you, the developer, intended it to be implemented. At any time, if you think I
need to have any additional information that can help me make that judgment, please just let me know.
IF ONLY ONE VERSION, PROCEED WITH THE INTERVIEW QUESTIONS.

IF MULTIPLE VERSIONS, USE THE FOLLOWING ADDITIONAL INTRODUCTION:

You mentioned that multiple versions of the program were developed. The next set of questions will probe specific
areas of the curriculum. To the extent that the curricula are different, the responses to these questions could be very
different for each version. So, we can proceed in two ways with this section of the interview. I could simply ask
each question and then you can point out those places where the response varies for each version. This approach
would probably work best if the differences among the versions are relatively minor. If, however, the differences
among the versions are relatively major, or a number of different versions were developed, we might want to go
through these questions separately for each version of the program. What do you think would be the best approach?
DIFFERENT PEOPLE MAY NEED TO BE INTERVIEWED FOR THE DIFFERENT VERSIONS. PROBE IF NECESSARY.

USE SEPARATE CODE SHEETS FOR ITEMS 1 TO 21 FOR EACH VERSION, AS NEEDED. NOTE ON THE TOP OF THE CODE SHEET
THE VERSION TO WHICH THE ANSWERS APPLY.

4

CODE SHEET
IF YOU ARE ASKING EACH QUESTION ONLY ONCE BUT THE CURRICULUM HAS MULTIPLE VERSIONS, PLEASE PROBE AFTER
EACH QUESTION FOR DIFFERENCES AMONG VERSIONS.

VERSION: _________________________________

Let’s start with a couple of questions about the target population.
1.

For what grade levels is the program intended? IF YOU ALREADY KNOW THIS FROM THE RESPONSES
THE PROGRAM HISTORY, SIMPLY FILL IN THIS QUESTION AND DOUBLE CHECK WITH INTERVIEWEE.

a.

Pre-kindergarten .......................................................................

b.

Kindergarten .............................................................................

c.

Grade 1 ......................................................................................

d.

Grade 2 ......................................................................................

e.

Grade 3 ......................................................................................

f.

Grade 4 ......................................................................................

g.

Grade 5 ......................................................................................

h.

Grade 6 ......................................................................................

i.

Grade 7 ......................................................................................

j.

Grade 8 ......................................................................................

k.

Grade 9 ......................................................................................

l.

Grade 10 ....................................................................................

m. Grade 11 ....................................................................................
n.

2.

Grade 12 ....................................................................................

Yes

No

†
†1
†1
†1
†1
†1
†1
†1
†1
†1
†1
†1
†1
†1

†2
†2
†2
†2
†2
†2
†2
†2
†2
†2
†2
†2
†2
†2

1

Which of the following best describes the students for whom the program is intended?
AND FILL IN ONE RESPONSE.
All students in the targeted grade or grades (universal) ....................

†1

Only students who are members of groups at high risk for ATOD
use or school crime (e.g., those living in a high crime
neighborhood) (selective) .................................................................

†2

Only students who already show evidence of having problems with
ATOD use or school crime (e.g., experimented with drugs,
suspended or arrested for violent behavior) (indicated)....................

†3

5

TO

READ THE LIST

Now, I’d like to learn a bit more about the number and duration of program sessions or lessons.
3.

How many sessions would comprise full implementation of the program?
_____ _____

4.

Does this program require booster sessions?
Yes †1
No †2
IF YES, PROBE TO DETERMINE IF THERE IS A RECOMMENDED PERIOD(S) WHEN SUCH SESSIONS SHOULD BE
OFFERED

5.

What is the recommended number (or range of numbers) of student participants per session?
_______ NUMBER OF STUDENTS
PROBE TO DETERMINE IF THESE NUMBERS VARY DEPENDING UPON THE ACTIVITY.

† THE PROGRAM/DEVELOPER MAKES NO RECOMMENDATION REGARDING THE NUMBER OF STUDENT
PARTICIPANTS PER SESSION

6

6.

What is the recommended length of the program?

READ THE LIST. SHOULD ONLY SELECT ONE RESPONSE.

An entire school year ........................................................................
7 to 8 months ....................................................................................
4 to 6 months ....................................................................................
1 to 3 months ....................................................................................
1 to 3 weeks ......................................................................................
1 to 6 days.........................................................................................
Less than 1 day .................................................................................

7.

†1
†2
†3
†4
†5
†6
†7

How often should sessions take place? (That is, if the program has more than one lesson, should the
lessons be delivered in a set period, for example, once a week?)
More than once a day........................................................................
Once a day ........................................................................................
2 to 6 times a week ...........................................................................
Once a week......................................................................................
2 or 3 times a month .........................................................................
Once a month....................................................................................
Less than once a month.....................................................................
Once or twice during a school year...................................................
No set period is required as long as all lessons delivered ................

7

†1
†2
†3
†4
†5
†6
†7
†8
†9

OK, we’re going to talk now a bit more specifically about the program topics.
8.

I’m going to read a list of topics and I’d like you to tell me which topics are components of the program
and which are not.
Yes,
Is a program
component

a.

General health or safety promotion...........................................................................

b.
c.

Cultural or historical topics (e.g., race, class, society)..............................................
Alcohol, tobacco, or other drug information (e.g., facts vs. myths, effects and
consequences) ..........................................................................................................

d.
e.
f.

Sex education............................................................................................................
Violence prevention (including victimization avoidance, coping with victimization
or loss experiences)...................................................................................................
Ethics, moral, or character education (e.g., personal responsibility, male
responsibility) ...........................................................................................................

g.

Etiquette or manners education.................................................................................

h.

Job skills, career education, work experience, career exploration or development ..

i.

Academic study skills or test-taking .........................................................................

j.
k.

Self-confidence, self-esteem, or self-image..............................................................
Emotional inventory (e.g., recognizing and understanding feelings, recognizing
physical cues such as sweaty hands).........................................................................

l. Emotional control (e.g., anger management, stress control).....................................
m. Emotional perspective taking (e.g., anticipating the perspectives or reactions of
others, empathy training) ..........................................................................................
n. Social influence (e.g., recognizing and resisting social influences to engage in
problem behavior, recognizing and resisting risky situations, refusal or resistance
skills training, assertiveness training) .......................................................................
o. Social problem-solving skills (e.g., identifying problem situations, generating
alternative solutions, evaluating consequences, decision-making, applying good
judgment) .................................................................................................................
p. Self-management (e.g., personal goal-setting, self-improvement, self-monitoring,
self-reinforcement, self-punishment)........................................................................
q. Attribution (e.g., attributing the cause of events or circumstances to one’s own
behavior⎯as in teaching students that poor grades are due to insufficient effort on
the part of the student rather than the task being too difficult) .................................
r. Communication skills (e.g., interpreting and processing social cues, understanding
nonverbal communication, negotiating) ...................................................................

9.

Did I omit any topics that are included as components of the program?

________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________

8

Not a
program
component

†1
†1

†2
†2

†1
†1

†2
†2

†1

†2

†1
†1
†1
†1
†1

†2
†2
†2
†2
†2

†1
†1

†2
†2

†1

†2

†1

†2

†1

†2

†1

†2

†1

†2

†1

†2

10.

Now, going back to that previous list, I’m going to ask you to tell me which of the components (that is,
those topics included in your program), you would consider mandatory, preferred, or optional.
Mandatory program components are those that are required to be delivered for the program to be
considered to be implemented with fidelity
ONLY FOCUS ON THOSE THAT WERE CHECKED PREVIOUSLY. FOR ALL OTHER TOPICS, MARK N/A.
Mandatory

a.

General health or safety promotion....................................................

b.
c.

Cultural or historical topics (e.g., race, class, society).......................
Alcohol, tobacco, or other drug information (e.g., facts vs.
myths, effects and consequences) .....................................................

d.
e.
f.
g.
h.

Sex education.....................................................................................
Violence prevention (including victimization avoidance,
coping with victimization or loss experiences)..................................
Ethics, moral, or character education (e.g., personal
responsibility, male responsibility)....................................................
Etiquette or manners education..........................................................
Job skills, career education, work experience, career
exploration or development ...............................................................

i.

Academic study skills or test-taking ..................................................

j.
k.

Self-confidence, self-esteem, or self-image.......................................
Emotional inventory (e.g., recognizing and understanding
feelings, recognizing physical cues such as sweaty hands) ...............

l. Emotional control (e.g., anger management, stress control)..............
m. Emotional perspective taking (e.g., anticipating the
perspectives or reactions of others, empathy training).......................
n. Social influence (e.g., recognizing and resisting social
influences to
engage in problem behavior, recognizing and resisting risky
situations, refusal or resistance skills training, assertiveness
training) .............................................................................................
o. Social problem-solving skills (e.g., identifying problem
situations, generating alternative solutions, evaluating
consequences, decision-making, applying good judgment)...............
p. Self-management (e.g., personal goal-setting, selfimprovement, self-monitoring, self-reinforcement, selfpunishment) .......................................................................................
q. Attribution (e.g., attributing the cause of events or
circumstances to one’s own behavior⎯as in teaching
students that poor grades are due to insufficient effort
on the part of the student rather than the task being too
difficult).............................................................................................
r. Communication skills (e.g., interpreting and processing
social cues, understanding nonverbal communication,
negotiating) .......................................................................................
s.

Other (specify) _________________________ ................................

9

Preferred

Optional

N/A

†
†1

†
†2

†
†3

†4
†4

†1
†1

†2
†2

†3
†3

†4
†4

†1

†2

†3

†4

†1
†1

†2
†2

†3
†3

†4
†4

†1
†1
†1

†2
†2
†2

†3
†3
†3

†4
†4
†4

†1
†1

†2
†2

†3
†3

†4
†4

†1

†2

†3

†4

†1

†2

†3

†4

†1

†2

†3

†4

†1

†2

†3

†4

†1

†2

†3

†4

†1
†1

†2
†2

†3
†3

†4
†4

1

2

3

11.

As you know, sometimes schools take it upon themselves to make changes or modifications to the
program for a variety of reason when implementing the program. We are interested in finding out
what changes to the program, if any, would be acceptable (that is, what changes would be allowed and
still have the program considered to be implemented with fidelity)? I’m going to read a list of possible
modifications that schools sometimes make. For each item, please state whether such a change would
be acceptable and, if so, whether consultation between your organization and the school is required
before such a change is made.
FOR ALL “YES” RESPONSES, PLEASE PROBE AS TO WHETHER THESE CHANGES REQUIRE CONSULTATION BETWEEN THE
SCHOOL AND THE DEVELOPER, IF NOT PROVIDED.
Developer consultation

a.

Drop topics from the curriculum...........................................................

b.

Add topics to the curriculum ................................................................

c.

Modify the curriculum topics (e.g., to make them more culturally
relevant, age-appropriate) ....................................................................

d.

Offer fewer sessions than the curriculum called for .............................

e.

Offer more sessions than the curriculum called for ..............................

f.

Revise the order in which lessons were presented ................................

g.

Shorten the duration of the program .....................................................

h.

Lengthen the duration of the program ..................................................

i.

Vary how often sessions are held (e.g., teach the program once
rather than twice a week) .....................................................................

j.

Modify the instructional strategy called for in the curriculum..............

l.

Increase or decrease the number of students included in a
program, session, or activity.................................................................

l.

Allow lower grade version to be implemented in higher grades...........

Yes

No

†
†1

†
†2

†1
†1

†1
†1
†1
†1
†1
†1

†2
†2
†2
†2
†2
†2

†1
†1
†1
†1
†1
†1

†1
†1

†2
†2

†1
†1

†1
†1

†2
†2

†1
†1

†1

†2

†1

1

required
2

N/A (different grade versions not available) †3
m. Allow higher grade version to be implemented in lower grades...........
N/A (different grade versions not available) †3

10

This next section focuses on implementation strategies and student incentives
12.

I’m going to read a list of instructional strategies. For each strategy, please state whether school should
used this strategy when implementing the program?
Yes

a.

Cooperative learning (e.g., small group discussion and
activities) ....................................................................................................

No

b.

Class discussions .........................................................................................

c.

Story telling .................................................................................................

d.

Videos, films, or other audiovisual aids.......................................................

e.

Use of props (e.g., puppets, dolls) ...............................................................

†1
†1
†1
†1
†1

f.

Establishment and enforcement of ground rules (e.g.,
confidentiality, respect for different opinions) ............................................

†1

†2
†2
†2
†2
†2

g.

Behavioral modeling (e.g., use of peer models to
demonstrate a new skill) ..............................................................................

†2
†2
†2
†2
†2

h.

Role-playing ...............................................................................................

i.

Skits, games, or other interactive exercises .................................................

j.

Rehearsal and practice of new skill .............................................................

†1
†1
†1
†1

k.

Applying program strategies to other settings or
situations......................................................................................................

†1

†2

Use of cues to remind individual to display a behavior
(e.g., posters) ...............................................................................................

†1

†2

†1
†1
†1

†2
†2
†2
†2
†2
†2
†2
†2
†2

l.

m. Coaching (e.g., directing or guiding students on the use
of skills, providing support as skills are practiced)......................................
n.

Active or reflective listening........................................................................

o.

Goal setting..................................................................................................

p.

Behavioral management/behavioral modification
techniques (e.g., gold stars, points, tokens, strikes,
verbal praise, daily report cards, validation)................................................

q.

Peer teachers/leaders....................................................................................

r.

Co-teaching/co-leadership ...........................................................................

s.

Parent involvement ......................................................................................

t.

Homework ...................................................................................................

†1
†1
†1
†1
†1

u.

Other (Please
specify):________________________________........................................

†1

11

13.

Are rewards for student learning an integral part of the program?
Yes

†1

No †2
IF YES, ASK WHICH OF THE FOLLOWING STUDENT REWARDS ARE APPROPRIATE.
IF NO, GO TO QUESTION 14.

a.

14.

15.

16.

Should groups be rewarded for the achievement of individual
group members ..........................................................................

b.

Should groups be rewarded for group accomplishments ...........

c.

Should individuals be rewarded for their own achievements ....

Yes

No

†1
†1
†1

†2
†2
†2

Using the following scale (very often, sometimes, seldom, never), can you tell me the extent to which
student should be formally assessed regarding mastery of the program content?
Very often .........................................................................................

†1

Sometimes ........................................................................................

†2

Seldom ..............................................................................................

†3

Never ........................................................................................... …

†4

Using the same scale (very often, sometimes, seldom, never), can you tell me how frequently instructors
should re-teach program content not initially mastered by students?
Very often .........................................................................................

†1

Sometimes ........................................................................................

†2

Seldom ..............................................................................................

†3

Never ........................................................................................... …

†4

I’d like to know the type of school or other personnel recommended to implement the program? I’m
going to read a list of possible program personnel. Please rank order the list in terms of the staff
preferred for the program.
General education teacher.................................................................

___

Health teacher ...................................................................................

___

Guidance counselor, school psychologist, social worker..................

___

Outside trained program personnel ...................................................

___

Other, please specify _________________ ......................................

___

12

Now, I’d like to turn our attention to training and technical assistance.
17.

Is initial training required for program instructors prior to implementing the program?
Yes †1

No †2
IF NO, SKIP TO QUESTION 21.
IF YES:
18.

Please describe the type of training required. Is it…

In person training..............................................................................
Web-based training ..........................................................................
Teleconference or web-conference training.....................................
Training based only on a manual (no additional training
required)… ................................................................................
Other (please specify)__________________________ ...................

19.

No

†1
†1
†1

†2
†2
†2

†1
†1

†2
†2

How much initial training should each instructor receive prior to implementing the program?
4 days or more ..................................................................................
2 – 3 days..........................................................................................
1 full day...........................................................................................
A half day .........................................................................................
Short demonstration or orientation only ...........................................

20.

Yes

†1
†2
†3
†4
†5

Who can provide the initial training for this program?

School staff knowledgeable about the program ................................
School district staff knowledgeable about the program ....................
Trainer(s) from the program developer/publisher .............................
Other (Please specify) ___________________________________

13

Yes

No

†
†1
†1
†1

†2
†2
†2
†2

1

21.

Is additional technical assistance and support provided by the program developer?
Yes

†1

No †2
IF YES, PLEASE DESCRIBE THE TYPES OF SUPPORT PROVIDED (E.G. FOLLOW UP TRAINING,
TELEPHONE/WEB T/A, ON-SITE T/A, ETC.)

________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________

14

IMPLEMENTATION MATERIALS AND CLOSE-OUT
ASK THESE QUESTIONS ONLY ONCE, AFTER YOU’VE COMPLETED ALL OF THE CODE SHEETS FOR THE PROGRAM.

1.

We are nearly finished with the interview. We are interested in any implementation manuals that you might
have. Can you provide us with a list of implementation manuals? If yes, what is the best way to obtain that list
(e.g., e-mail, developer website)?

2.

How can we obtain copies of the implementation manuals? If so, what if any is the cost and how can we order
those manuals?

3.

Just one more question. Do you have any additional information you think I need to know to be able to assess
if a school is implementing your program with fidelity?

Thank you so much for speaking with us today. We really appreciate you taking the time to help us with the study.
Your responses will help us tremendously as we learn more about how schools are implementing different researchbased programs.

15