Attachment F_Pretest Report

ATTACHMENT F
PRETEST REPORT (PRETEST MEMO AND Q BY Q PRETEST APPENDIX)

MEMORANDUM

600 Maryland Ave. S.W., Suite 550
Washington, DC 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com

TO:

Seth Chamberlain

FROM:

Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan
Hershey

SUBJECT:

Pregnancy Prevention Approaches First Follow-Up Pretest
Findings

10/25/2010
6549-100
DATE:

This memo describes the Pregnancy Prevention Approaches pretest of the first follow-up
survey conducted June-August 2010 by Mathematica Policy Research and Child Trends. The
purpose of the pretest was to improve both the survey instrument and the data collection process.
The memo describes (1) the recruiting process and youths selected for the pretest, (2) the
processes of administering the pretest survey and debriefing participants, and (3) the overall
findings about survey administration and content. At the end of this memo, we have included an
appendix listing individual survey questions and issues discovered during the pretest. We have
also included a copy of the debriefing guide used. When issues are raised we have not included
all possible solutions, as we recognize that further discussion is needed before we make
modifications to the instrument. Indeed, because many of our questions come from wellestablished national surveys or are on our baseline survey, it is important to weigh the findings of
our pretest—and possible alterations of wording designed to make questions as clear as
possible—against the aim of comparability with other research, national data and our baseline
data.
RECRUITING PROCESS AND FINAL PRETEST SAMPLE
Mathematica recruited participants in Princeton, NJ and Washington, DC. In Princeton, the
team worked with HiTops, a community-based organization (CBO) that serves teens. In
Washington, DC, efforts to work through several CBOs were not fruitful. Therefore respondents
were recruited through Mathematica staff networks. The recruitment strategy used a combination
of email, print and electronic fliers, as well as word of mouth. The flyers asked parents of teens
to call Mathematica if they and their children were interested in participating. Mathematica staff
then talked directly with interested parents to explain the pretest and the need to obtain parental
consent prior to participation.
Due to the young age of the recruits (the youngest being 12 years old) and parental
sensitivity, teens could not be asked prior to the pretest about their sexual activity status.
Therefore, the pretest was rolled-out in phases to ensure enough sexually active youth were
included without going over the OMB limit of nine pretest interviews with non-sexually active
youth. Despite recruiting efforts to locate sexually active youth, we ultimately found more nonsexually active youth than expected from our pretest recruiting. In the end, we only interviewed
six sexually active youth. We also ignored two interviews that were conducted with non-sexually
An Affirmative Action/Equal Opportunity Employer

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
2
active youth who we recruited specifically because we thought they were sexually active. When
the surveys showed they were not, and thus increased our number of non-sexually active
interviews above nine, we disregarded the data collected.
In Washington, D.C., six follow-up pretest surveys and debriefings were conducted at the
offices of Mathematica between June 9th and June 11th. In New Jersey, 11 follow-up pretest
surveys and debriefings were conducted at the offices of HiTops between June 14 and June 17th.
The teens were scheduled to participate during one of the pretest administration days. The
pretests were administered in small groups (ranging from two to seven teens). The teens
completed the self-administered questionnaire in a setting designed to emulate a schooladministration setting, and then participated in a one-hour long, one-on-one debriefing session
with a Mathematica or Child Trends researcher. Upon completion of the debriefing, the teens
were given $50 in cash for their participation. Teens recruited through Mathematica staff
contacts in D.C. were brought to Mathematica by parents for their appointments. Teens in
Princeton, N.J. met us at the HiTops center.
In total, 15 teens participated in the pretest—six sexually active youths and nine nonsexually active youths; nine boys and six girls, who ranged in age from 12 to 16. The following
table details the distribution.

Age
12
13
14
15
16
Total

Male–
Sexually
Active
0
2
1
1
0
4

Male–
Non-sexually
active
0
2
2
0
1
5

Female–
Sexually
Active
0
0
1
0
1
2

Female–
Non-sexually
Active
1
1
1
1
0
4

Total
1
5
5
2
2
15

Younger teens were well represented in the pretest, providing crucial data about how to
make the instrument easy to understand and easy to answer for youth their age. Most of the teens
in the sample seemed to have high or middle socio-economic status. Through well-educated
parents and affluent school districts, they seemed to have been exposed to topics such as sexual
health, pregnancy, contraception, HIV/AIDS, alcohol and illegal drug use. There were two to
three teens that seemed to belong to lower SES families with lower levels of income and
knowledge exposure.

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
3
PRETEST AND DEBRIEFING PROCESS
The administration of the pretest mirrored (as closely as possible) what will happen during
the actual study in a classroom environment. That is, we gathered the teens together in one room,
had a researcher give an introduction and verbal instructions about how to complete the
questionnaire, and had the teens read and sign an assent form before opening the survey packet.
Each teen was then told to open his or her survey packet, which contained one Part A
questionnaire and two Part B questionnaires (one Part B is designed for sexually active youth
and the other Part B for non-sexually active youth). The only differences between the
instructions we gave to teens during the pretest and the instructions that will be given to teens
during implementation of the actual study were that in the pretest we asked the teens to circle
any questions or words that gave them trouble so that we could discuss them during the
debriefing, and we explained how the one-on-one debriefing and incentive payment would work.
Researchers timed how long each teen took to complete the questionnaire. When a teen
finished the questionnaire, one of the researchers accompanied the youth to a private office for
the one-on-one debriefing. The assigned researcher was matched by gender to the pretest
participant debriefed. Each debriefing lasted approximately one hour.
Before the first pretest, all debriefing researchers attended a three-hour training to review
best practices for talking with teens about sensitive subjects, discuss what to prioritize during the
debriefing, and talk about how to address issues or problems should they arise during the
debriefing. Although the debriefing guide prepared by Mathematica and Child Trends included
specific probes for many items in the survey, each researcher was given latitude to rephrase the
questions if needed and to choose which items to ask about should time be a limiting factor
during the hour-long debriefing.
The debriefing guide focused on how the respondents understood the questions in the
survey, how they came up with their answers (that is, the thought process they used), and how
well the answer categories on the survey reflected their experiences. We also asked whether they
read instructions, observed how well they followed them, and asked questions about inconsistent
or incorrect responses. We also asked if there were any questions or words that were confusing
or out of date. In a few cases, alternative question wording or answer categories were given to
respondents during the debriefing interviews, and respondents were asked about how they
understood each version and how they would answer alternative phrasings.
It should be noted that each researcher had the respondent’s completed questionnaire to refer
to during the debriefing, which was especially helpful when answers appeared contradictory to
what was said during the debriefing. These discrepancies are good indications of problems with
question wording and were investigated with the respondents.

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
4
Not all debriefing questions were asked of each participant due to lack of time. In the
question by question results in the appendix, it should be noted that one of the 15 participants
was not debriefed about Part A, thus only a maximum of 14 participants have responses to Part A
debriefing questions. In addition, one of the non-sexually active participants was not asked
debriefing questions for Part B2, thus responses to those questions only have a maximum of
eight participants’ comments. Part B1 has responses from all six sexually active participants.
SURVEY ADMINISTRATION OVERALL
We were interested in learning about a number of different survey administration issues,
including the length of time needed to complete the questionnaire, whether instructions were
clear and were followed by the teens, and in particular, whether teens understood how to select
the correct version of Part B. The debriefings provided the following feedback on survey
administration:
• Assent Process. The assent process went smoothly during the pretest process, and no
respondent raised questions. However, during the debriefing interviews, three out of
the 12 respondents asked to comment on the assent form said that they thought it did
not give enough information to make a good decision about participating in the
survey. One respondent thought there could be more information on how personal the
questions would be. Another thought that the form could have mentioned that they
were going to be asked about their sexual behavior. Nonetheless, the majority thought
it did give enough information, and one respondent mentioned that she was pleased to
see that we acknowledged that their parents had given permission for them to
participate in the survey.
The need for confidentiality of the information asked for in the survey came up as a
theme during the debriefings at a few points in the survey, and this might be a good
place to reinforce it.
• Length of the survey instrument. The overall length of the questionnaire is a little
long. The time needed for teens to complete the survey (Parts A and B) ranged from
18 to 50 minutes. We expect that the study introduction, distribution, and collection
of the survey packets will take about 10 minutes when we get into a classroom
setting. (The pretest is not a good test for these administrative functions, given the
need to explain the debriefing process and the fact that the pretest groups were much
smaller than we will typically encounter in a classroom environment.) If the typical
class period is 40 minutes long, there would be about 30 minutes for students to
complete the survey. About 75% of pretest respondents finished in 30 minutes or less.
If the class period is 35 minutes or the administration functions take longer than
expected, leaving only 25 minutes for survey completion, less than 40% of pretest
respondents would have finished the survey in the class time.

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
5
Therefore, we recommend shortening the survey by about five to 10 minutes, to
ensure a higher percentage of students can finish during the class time. Given the
higher than average SES of most pretest respondents, we could expect the actual
sample of respondents to take the survey will require even more time than the pretest
sample. Furthermore, sexually active respondents were more likely than non-sexually
active youth to take longer than 25, 30 or even 35 minutes to complete the survey. As
these respondents are of the utmost importance to the study, this further underscores
the need to shorten the survey. We suspect that the reason the sexually active
respondents took longer than the non-sexually active respondents had more to do with
respondent characteristics than any difference in the length of the two Part B sections.
• Instructions on how to complete the questionnaire. During the debriefing we asked
if the instructions were clear or confusing in any way. Many teens reviewed the
instructions and reported that they were clear and straightforward. However, a
number of teens said that they had not actually read the instructions. Most went back
and read them when they got confused, but this is of some concern.
We recommend spending a good amount of time explaining how to complete the
instrument during the introduction phase in the actual data collection.
•

Issues with skip logic. There were a few respondents who did not skip correctly in the
survey, however, this usually meant answering questions they were not meant to
answer, rather than skipping questions they were meant to answer. The teens who
made these errors were usually able to answer the questions they should have skipped
and continue with the survey, as most of these questions are made to be answerable
by anyone, so that respondents will not get lost and will be able to continue. We will
be sure to review all questions to make sure as many as possible can be answered
even if they should be skipped.

•

Missed questions. Some respondents skipped questions they should not have skipped.
Sometimes the questions were missed. More often, they were read, but there was
some reason the respondents could not answer them. These will be addressed in the
question by question review in the appendix, but overall, comprehension problems
seem to be the most common reason for skipped questions. These problems were
more common among the sexually active youth in Part A, and more pronounced in
Part B1 (for sexually active youth). This appears to be caused by the characteristics of
the sexually active respondents rather than an issue with the difficulty of the
questions in Part B1, and this correlation should be considered.

• Selection of Part B. For the most part, the teens read the instructions and correctly
identified which version of Part B they should complete. Only one teen had a difficult
time choosing the correct Part B. With guidance, she chose the correct packet, but we

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
6
suspect that would not have been the case without aid. In the debriefing she explained
that she was not paying attention and did not read the directions.
Two out of eight respondents said they would be concerned about picking the part for
sexually active youth when in the classroom. One was non-sexually active and the
other was sexually active. The sexually active youth said she would have picked the
packet for non-sexually active youth in the classroom. Two other youth said they
would not be concerned about which packet to pick if there was enough room
between people or they could cover their packet or be clever about grabbing the
correct booklet. The other four said they would not be concerned (three were sexually
active and one was not sexually active). One respondent specifically pointed out that
the packets look similar, which made him feel more comfortable selecting the correct
one to fill out.
We recommend that the instructions at the beginning of the class period emphasize
the similar look of the two Part B packets. This could be done in the context of
emphasizing being careful to select the correct one, but be used to suggest that it will
be hard for neighbors to know what part a respondent is filling out. We also suggest
an experiment using desktop carrels / desk dividers / privacy shields in Chicago to see
if their use increases the reporting of sensitive behaviors.
GENERAL AND CONTENT OBSERVATIONS
In addition to determining problems or issues with the completion of the instrument, the
pretest was designed to review the content and language used in the questionnaire. The following
are some general observations and themes that emerged from our discussions with teens. We
found that the follow-up benefited greatly from the design work and pretesting of the baseline
survey and thus fewer issues were found.
Terminology and language level. Teens were asked to highlight words that they did
not understand. There were several words that caused concern. The main terms to be
mentioned were Chlamydia, gonorrhea, abstinence, halfway house, and “spur of the
moment.” General understanding was not a problem for most respondents, but at least
one sexually active 13-year-old boy had a very hard time with quite a bit of the
language. Respondents were also asked if there were specific words and terms that
were not up to date. Issues with terms are summarized in detail in the appendix, some
with suggestions for additions or alternative wording.
• Definitions. When asked directly, most teens told us that they thought the definitions
provided were clear and understandable. When asked if they would have liked more
definitions, many said this was not necessary, but some said they would have liked
definitions for oral and anal sex, or thought other respondents would. Given the

MEMO TO: Seth Chamberlain
FROM:
Kristen Velyvis, Mustafa Menai, Melissa Thomas, and Alan Hershey
DATE:
8/25/2010
PAGE:
7
number who did not know what oral and / or anal sex is, we recommend adding these
definitions at first reference. Fewer respondents thought it was necessary to have a
definition of condoms. A definition of what we mean by birth control would also be
clarifying.

cc: Chris Trenholm, Kristin Moore, Mustafa Menai, Kristine Andrews, Kassim Mbwana

FIRST FOLLOW-UP PRETEST SURVEY
QUESTION BY QUESTION APPENDIX

(Note: not every teen was asked all the debriefing questions due to time constraints.)
PART A
SECTION 1 – YOU AND YOUR BACKGROUND
• Q1.1 – Date of birth.
No problems.
• Q1.5 – Race.
One respondent out of 14 left this blank. He did not know how to answer, as he is
Hispanic and considers that his race. He was not sure what to put as an answer.
• Q1.6 – Learning about sexuality related topics in the last year.
There is some confusion about the terms “abstinence,” “communication,” and “to
have sex.”
- Four out of 14 respondents clearly reported not knowing at all what “abstinence”
meant. Most of the others understood that abstinence meant not to engage in some
kind of sexual activity. However, the range of meanings given by the ones who did
have an idea of what it meant included abstinence from “sexual intercourse,” from
“sex,” and from “sex” until marriage, as well as “being a virgin.” One respondent
thought it meant to slow down or stop the act of having sex.
Recommendation: Give a definition of abstinence in sub-question b. That is,
replace “Abstinence from sex?” with something like “Abstinence from sex,
which means not having sexual intercourse, oral sex or anal sex?” If we do
decide to define abstinence, we will need to decide if the definition should
address abstinence generally or until marriage specifically, and ensure the
definition is agreeable to the stakeholders in the study who support abstinence
education.
- Four respondents who answered this question were unsure about who the
communication in sub-question e. was meant to be between: Themselves and
parents/adults, themselves and partners, or possibly themselves and doctors.
Recommendation: Sub-question e. should be more specific and broken into two
questions:
“e. How to talk to your partner about whether to have sex or whether to
use birth control?”
“f. How to say no to sex?”

1

- While one respondent was not sure what “to have sex” (in sub-question e.) meant at
all, most respondents could give some definition. However, there does not seem to be
a uniform understanding of the phrase among the respondents. Some respondents
included all forms of sex in their definitions; some included only sexual intercourse
(penis in vagina). One respondent thought it was activities that could make someone
pregnant, and another thought it was things that go beyond using fingers and oral sex.
Recommendation: Give a definition of what we mean by “to have sex”
encompassing oral, anal and vaginal sex.
There is a potential lack of knowledge about methods of birth control. Few
respondents did not know at all. Most, when asked, mentioned condoms and/or pills
and nothing else. At least one respondent said they had indeed learned about other
methods, but did not think about them while answering the question. So for most, the
question may not capture whether they have received information about methods of
birth control other than condoms and pills in the last 12 months.
Recommendation: Give examples in the sub-question c., perhaps like this:
“Methods of birth control such as condoms, pills, the patch, Depo-Provera (the
shot), the ring, or other methods?” We would have to consider whether to list
withdrawal in this list.
• Q 1.7- Most helpful source of information on sexuality related topics:
Most respondents answered this question with “school,” followed by “family” and the
“media.” However, in interpreting the word “important,” four out of 14 respondents
chose the source that gave them the most information, rather than the source that was
the most helpful to them. For two respondents, the source of the greatest amount of
information was also the same as the source of highest quality information, but it is
unclear whether “highest quality” is the criterion on which we want them to base their
choice.
In addition, three respondents found it hard to choose just one source. One respondent
stated that in choosing a response, they were focusing on selecting all the relevant
sources of information, rather than choosing the source that was most helpful. This
was potentially an issue for the others, as well.
Recommendation: Think about how important it is for us to know the most
helpful source as opposed to knowing the source that provided the most
information. If it is important, may want to break this question into two parts:
a. Which source of information do you get most of your information from and
b. which source of information is the most helpful to you.
There was potential confusion about the term “media.” Most respondents knew what
it meant and had a good idea of what the term encompasses, however, one respondent
thought “media” referred to the library; while another thought it meant books. Two
respondents thought it meant only things on the internet.
Recommendation: Replace the word “media” with something else, such as
“newspapers, magazines, radio, TV or advertisements.”
Three participants suggested that instructions to mark (x) one response should be
made easier to see.
2

Recommendation: Reformat survey to make instructions easier to identify.
• Q 1.8- Unable to control the important things in life.
One respondent left this blank and wrote, “I don’t understand.” This 14 year old had
more trouble than most with vocabulary, but could be used as a good litmus test for
the questions with which respondents with difficulties reading will have most
problems.
SECTION 2 - FAMILY
• Q 2.1 & 2.4 – Identifying mother or mother figure (father or father figure).
These questions worked well. Most respondents found it helpful that the answer
categories were broken out. Two out of 13 respondents who answered this question
recommended adding older sister or brother to the answer categories, despite the
presence of the response option “some other adult.”
Recommendation: Add response option “older sister / brother.”
One respondent suggested separating stepmother and adoptive mother in the response
option “Your stepmother or adoptive mother (father),” as she lives with her stepdad
but is not adopted.
Recommendation: Separate step- and adoptive parents, or move adoptive to be
listed along with biological.

• Q 2.2 & 2.5 – Work status (full/part time) of parents or parent figures.
These questions worked well overall. There were a few minor issues: One respondent
did not know if someone who is self employed would fit into the part- or full-time
category. His mother is an “entrepreneur” who gets work from time to time, but she is
her own boss. He left this question blank. Another respondent’s mother works more
than 30 hours a week, but he marked part-time, because the hours are not regular and
sometimes she puts in those hours over a period of less than five days in a week. A
third respondent wondered how to respond since her father just received a job as a
full-time teacher for the fall, but had not started yet.
Recommendation: Add a category for “Yes, s/he works, but I don’t know how
many hours,” and change the “don’t know” option to “don’t know if s/he is
working.”
A fourth respondent highlighted a different type of issue. She lives with her adopted
parents, but does not think of her adoptive parents as any kind of parents or parental
figures. She said “I don’t know” for her mother and full time for her father, but it is
unclear about whom she is answering.
Recommendation: Clarify the people we want the respondents to answer about.

• Q 2.7-2.9 – Parental awareness of children activities in last month.
3

We asked what respondents thought of as “the last month.” Seven out of 14
respondents thought about “last month” in terms of the last four weeks. Five thought
of the last 30 days. These two conceptions are fairly similar for our purposes. Two
respondents thought of “last month” as the last calendar month, which might or might
not be the same time frame. In this case, it would be the same, since the pretest was
given at the start of a new month. In some cases it would not be. We do not think that
time frame will be a major problem with this question, and did not identify any other
problems with this question.
Recommendation: Leave questions as is.

SECTION 3 - VIEWS AND PERCEPTIONS
• Q 3.2 d – Possibility of having a good marriage.
There were no problems identified with this question. All 14 respondents answered
that they “strongly agree” or “agree” that having a good marriage seems possible for
them. This suggests that our pretest sample was more homogeneous than we expect
our study sample to be.
• Q 3.5 Refusal skills/stopping someone else’s sexual behavior towards you.
This question worked well overall. All but one out of 14 respondents understood that
“them” meant a single person in this context. When asked, most of the male
respondents thought that the sub-question a. regarding their ability to stop someone
from touching their chest was weird or not applicable.
Recommendation: Make sub-question a. for girls only by adding FOR GIRLS in
parentheses at the end of the sub-question. We suggest putting it at the end of
the sub-question, so that it is not the first thing all respondents see, and does not
cause boys to skip all three sub-questions in this series. Other options we
considered included putting FOR GIRLS first and adding FOR BOTH GIRLS
AND BOYS to the beginning of sub-questions b and c. This was too cluttered
looking and not consistent with the style of the rest of the survey. We also
considered having sub-question a. appear last in the list of sub-questions with
FOR GIRLS at the start of the sub-question. We chose not to do this because the
behaviors are progressively more invasive, and the flow of the content would be
negatively affected.
• Q 3.7 – Views about condom efficacy.
There were no issues with this question. All respondents thought that a definition of
condoms was unnecessary. Five out of 14 respondents thought a definition might be
helpful for others, but did not think they needed one themselves. Everyone said they
referred to condoms as “condoms,” and not by any other names.
• Q 3.7a, 3.10a & 3.13a – Degree of confidence in answer.
4

Most respondents thought the reason we were asking how confident they were was to
see how sure they were in their answers. A few extended the reasoning to surmise that
we wanted to make sure that they were confident so as to avoid bad outcomes for
themselves, and to see if the schools are teaching them well. Three respondents
suggested that the questions were meant to get them to think more carefully about
their answers. Two respondents answered these questions when they should have
skipped them, but continued with the survey, so these responses should be easy to
remove.
• Q 3.14e – Vaccine and HPV
Only one out of 14 respondents could accurately articulate what a vaccine is. None of
the respondents knew what HPV is or for what the acronym stands. A number of
respondents guessed that HPV is a disease like HIV. A couple of respondents thought
it was the same thing as cervical cancer (which is technically incorrect, as it’s the
virus that may cause cervical cancer). One respondent thought that HPV was
something girls get, which leads us to think that others may think that HPV affects
only females (which is incorrect, since it can cause genital warts in men).
Recommendation: Add the phrase “or shot” after vaccine in the question, as a
short, kid-friendly definition. In addition, add “(also known as Human
Papillomavirus)” after HPV in the question to specify what HPV is. We
considered adding other, longer definitions of both words, but did not want to
lengthen the question unduly.
• Q3.17 – Likelihood of having oral sex in the next year.
Most of the 14 respondents felt a definition of oral sex would be unnecessary, while
only three respondents thought it would be helpful to have a definition, especially one
that clarifies the difference between sexual intercourse and oral sex. When asked
what oral sex means, however, one respondent categorically did not know what oral
sex was, and another confused it with anal sex. Four out of 14 respondents seemed to
think that it referred only to a female putting a male’s penis in her mouth and not vice
versa. One person thought the question should emphasize that it’s something that both
sexes can perform on each other.
Recommendation: Provide definition of oral sex such as “when a person puts
their mouth on a female’s vagina or on a male’s penis.”
• Q3.18 - Likelihood of having sexual intercourse in the next year.
Regarding the term “sexual intercourse,” only one person did not distinguish between
the different types of sex, and included vaginal, oral and anal in the definition of
sexual intercourse. Regarding the response categories, two out of 14 respondents felt
it would be helpful to have a “not sure” or “I don’t know” category.
Recommendation: Leave question and responses as is.

5

• Q3.20 – Meaning of boyfriend/girlfriend and someone you’re going out with.
Among the 14 respondents, there was broad consensus that a boyfriend or girlfriend
was someone with whom one is romantically involved. However, the respondents
were pretty evenly divided on the meaning of “someone you’re going out with” and
whether that was the same thing as boyfriend or girlfriend. Half thought the two were
synonymous, while the rest thought the two could be different. Most of the
respondents preferred the question as asked (“Right now, do you have a boyfriend or
girlfriend – someone in particular you are going out with?”) to the alternatives
suggested: “Is there someone you are currently dating - that is, a girl or guy you like
and who likes you back?” or “Are you currently dating a girl or guy? By dating we
mean when you like a girl or guy and she or he likes you back. It doesn't have to
mean going on a ‘real’ date.”
Recommendation: Consider removing the second part of the question, “– someone
in particular you are going out with.”
• Q 3.21 – Frequency of dating.
Most of the 14 respondents said that “to have a date” was to arrange to be with
someone they are, or want to be, romantically involved with. A few respondents
suggested that it was possible for people who are just friends to go out on a date.
However, the respondents were pretty evenly divided on the issue of whether “having
a date” was the same thing as “going out with someone.” Half thought the two were
synonymous, while the rest thought the two could be different.
Regarding the number of dates in the last three months, most respondents found it easy to
come up with the number. Most thought about the number of times they had gone out
with someone. Four out of 14 respondents said that they had had zero dates in the past
three months.
Recommendation: Leave question as is, since the definition is clear, and adding
“going out with” could muddy the waters.
• Q 3.22 – Number of different dating partners in the last three months.
Only one out nine respondents had difficulty calculating the number of people they
had gone out on a date with in the last three months. Seven out of ten respondents
said they have only gone out on a date with one person in the last three months. This
suggests the question worked well for this sample. One respondent thought it would
be easier to answer 3.22 before 3.21. Only one person did not skip this question
appropriately, but he continued on with the survey, and his answer was consistent
(zero partners after saying he had no dated anyone).
Recommendation: Consider switching the order of 3.22 and 3.21. This might
confuse things if respondents think of group dates, though.
• Q 3.23 – Have you ever had sexual intercourse, oral sex, or anal sex?

6

Eight out of 14 respondents did not think they needed definitions of the different
kinds of sex in this question, while the rest thought it could be helpful for themselves
and/or others.
Eight out of 14 respondents stated they would feel unsafe filling out this survey in a
classroom setting due to privacy concerns. However, three respondents did state that
if the answers were kept confidential and others could not look at their answers, they
would feel safe.
Recommendations: Add oral reassurances to respondents that their answers will
be kept confidential and that the sections Bs are very hard to distinguish. We
can also experiment with dividers in Chicago to see if this improves reporting.
• Transition questions at the end of Part A
A. Directions/information clear enough for picking the right packet (B1 Sexually
Active or B2 Sexually Non-active)?
Only one out of 14 respondents reported difficulty picking the right packet. She
admitted it was because she was not reading the directions carefully, something we
think is somewhat common.
B. If the respondents is uncomfortable with the sexually active/non-active question
and skipped it, which packet would they have picked?
Of the nine respondents who answered this question, seven reported they would
choose the “correct” one. One respondent said they would choose the non-sexually
active section in a classroom setting, even though they are sexually active. One
respondent was unsure which packet they would choose if they skipped the question.
Recommendation: We can experiment with dividers in Chicago to see if this
improves reporting.
C. Other things that may be helpful in making the right choice?
Respondents did not offer any suggestions.
D. Concerned about privacy when filling out the survey in a classroom setting?
Out of seven respondents, five said they would not be concerned, while two said they
would. One respondent who was not concerned said it’s because the packets look
similar. Another respondent said that if there is enough space between desks and
others can’t see, he would not be concerned.

7

PART B1 – (SEXUALLY ACTIVE) SECTIONS 4 AND 5
SECTION 4
• Q 4.1 – Ever had sexual intercourse, oral sex or anal sex.
Four out of five respondents read the directions and understood that the information
they disclose will be kept confidential. Only one respondent reported skipping the
question at first because she thought it was an example, but then read it a second time
and understood correctly.
Recommendations: Consider adding a “START HERE” box so that respondents
realize the questions are not examples.
• Q 4.2 – Ever had sexual intercourse.
The skip was reported to be easy to follow. Five out of six respondents felt the
definition of sexual intercourse was adequate. Only one respondent found it confusing
in that he did not understand the word intercourse, and suggested we should have just
used the word sex instead. Though he left this question blank, he did skip correctly to
Q 4.3.
Whether the sex they experienced was consensual or non-consensual did seem to be a
factor in what respondents felt counted as sex. Out of four respondents, although one
said they would count it as sex regardless of consent status, one said it would be sex
only if consensual, and another said that it would matter but did not want to elaborate
further. One respondent seemed a bit confused and seemed unable to imagine a
scenario of non-consensual sex.
Recommendations: Elaborate the definition to indicate if we want them to
include or leave out instances of non-consensual sex. Consider putting the
question immediately after the definition, instead of on its own line, so that
respondents are more likely to read the definition.
• Q 4.3 – Date of first sexual intercourse.
Three out of five respondents reported it was easy to remember the first time they had
sexual intercourse. This ease may have been due in part to the fact that the “first
time” occurred very recently for them, that is, within the past few months. One
respondent said it was embarrassing to put the exact date down, but they did put it
down anyway. For the remaining two respondents, the dates were more difficult. One
respondent knew her age at first intercourse, but had difficulty figuring out the year
and month, which she calculated incorrectly. Another respondent said it was easy to
recall the year but not the month. Both of these respondents reported first sex as
having occurred much longer ago (2006 and 2009) than the other three respondents
(all in 2010).

8

Recommendation: Add a question about how confident they are in their
answers?
• Q 4.4 - Age at first sexual intercourse.
Four out of five respondents reported it was easy to remember their age at time of
first sexual intercourse. For three of them, this may have been due to the fact that the
first time was very recently, within the past few months. When asked to compare this
question with the one before it (Q 4.3. Date of first sexual intercourse), three
respondents preferred this one, one liked the date question better, and one respondent
was indifferent between the two. The two who had a harder time with the date
preferred this one. But, of those two, one got the year of first sex correct, and could
only guess his age (12 or 13), while the other got her age correctly and calculated the
date incorrectly. The other three got both questions “right.” So, there is no clear
“best” way to collect this data based on this pretest.
• Q 4.5 - Had sexual intercourse more than one time.
Four out of five respondents followed the skip correctly. One respondent answered no
to this question, but did not skip as instructed, and filled out the questions in between.
Two respondents who answered yes indicated that their answers would still have been
yes if they had had sex with just one person, but more than one time. One person
chose to skip this question and skipped the others in between as if he had answered
no.
Recommendation: Leave as is. Answers to questions that should be skipped can
be deleted.
• Q 4.8 – Types of birth control used at last sex.
Two of the six respondents skipped several of the six sub-questions in this item.
• Q 4.9 - Frequency of sexual intercourse in last three months.
Out of three respondents, two recalled the frequency quite easily. One respondent said
it was easy as they had been having sex with just one person in the last three months.
The third respondent thought we were asking about the last three weeks instead of last
three months. He said it would be hard to remember the frequency during a period of
more than three weeks. His answer to this question was one, so his issue seems to be
the duration of the question, not the frequency of sex.
• Q 4.10 - Frequency of condom use in last three months.

9

Out of three respondents who should have answered this, one respondent could not
read the word “intercourse” and skipped the question (this is the same respondent
who could not read it earlier in the survey and skipped question 4.2). Another
respondent stated that they always use condoms, but could not come up with a
number. She did not seem able to connect the number of times she reported having
sexual intercourse in the last three months to this question. She wrote “don’t knowalways” as her answer. A third respondent did not seem to have any trouble.
• 4.12 - In the last 3 months, how often did you use any type of birth control?
Out of two respondents, one said they used some type of birth control just once, while
the other respondent reported never using any type. However, both had reported
frequent use of condoms in the past three months in previous questions. When
debriefed about what types of birth control they thought about when answering this
question, they both said birth control pills.
Recommendation: The placement of this question could have suggested we did
not mean condoms here. Either add, “including condoms,” or give a definition or
examples of types of birth control.
• Q 4.13 – Frequency of purely sexual relationships in last 12 months.
Five out of five respondents understood the concept of a purely sexual relationship
quite well. One respondent pointed out that the word “relationship” may be confusing
as others may think if the object is just sex, then it’s not really a relationship.
Recommendation: Leave as is.
• Q 4.14 – Ever had oral sex.
The definition we gave matched well with the understanding of the term for the four
respondents who responded to the debriefing questions about this item. Consensual
versus non-consensual nature of the sex experienced was a factor in what one
respondent feels counts as sex. One said it would be sex only if consensual.
Recommendation: Elaborate the definition to indicate if we want them to include
or leave out instances of non-consensual sex?
• Q 4.15 – Date of first oral sex.
It was easy to answer for all four respondents, probably because they started having
oral sex only recently, although one respondent said he was not really sure of his
answer. One respondent said it was easy to figure out the first time he had oral sex
since he had had it only once.
• Q 4.16 – Number of oral sex partners.

10

The three respondents who answered this question said it was not hard to answer,
probably because they had had oral sex with just one person.
• Q 4.17 – Frequency of oral sex in the past three months.
One of two respondents who answered debriefing questions about this confused this
question with the number of times she had had sexual intercourse. She suggested we
underline “oral.” One respondent answered it easily, probably because he has not had
oral sex in the last three months.

• Q 4.22 – Frequency of anal sex in last three months.
None of the 6 respondents had ever had anal sex.
SECTION 5
• Q 5.1- Ever taken a pregnancy test.
Only one of six respondents reported having taken a test, while one male respondent
did not know if his partner had taken a test or not. The others said they had not. Out
of four respondents who answered debriefing questions about this item, three felt it
was a relevant question. All three had had sexual intercourse. One respondent did not
think it was relevant because he had not had sexual intercourse (only oral sex).
• Q 5.2 – FOR BOYS Ever gotten someone pregnant.
All four male respondents answered this question, and the two who were debriefed
reported that the question was easy to find. Both girls skipped it.
• Q 5.3 – FOR GIRLS Ever been pregnant.
Both female respondents answered this question and thought that the question was
easy to find. All four male respondents skipped it. One female respondent found this
whole section irrelevant, especially if the respondent had only had oral sex. However,
this view of relevancy was not shared by a male respondent who had had only oral
sex.
• Q 5.8 – STD history.
Of three respondents, all reported not knowing at least one of the STDs listed.
Chlamydia, syphilis, genital herpes and HPV were named specifically as ones
respondents didn’t know.
11

Respondents noted that they answered or would have answered “no” or “don’t know”
to all the STDs they did not know. One respondent explained that she knew she does
not have an STD after going to the doctor, while another said that even though he
hasn’t been tested he would like to remain “positive,” (presumably meaning
“optimistic”).
Recommendation: Change all references in the survey to HPV to “Human
Papilloma virus, also known as HPV or genital warts.”
Q 5.9 – Opinions on birth control.
Of five respondents, two respondents thought they had enough experience to answer
this series of questions. One respondent thought that he did not have enough
experience to answer these questions since he has never bought/needed to buy birth
control (had never had sexual intercourse, only oral sex). Another thought he did not
have enough experience to have an opinion about birth control items. Both did answer
all the questions though.
Respondents understood what the phrase “heat of the moment” meant. When asked
about alternative wording regarding whether it is “hard to use birth control” versus
“impossible to use birth control” when caught in the heat of the moment, one
respondent said that she preferred “hard to use.”
Recommendation: Consider adding a response option for “don’t know,” even
though there is already a column for neither agree nor disagree.

PART B2 – (NON-SEXUALLY ACTIVE) SECTIONS 4 AND 5
SECTION 4
• Q 4.4 – Temporary housing.
There were no problems with this question; it worked well.
• Q 4.5 – Living in one or multiple homes.
There were no problems with this question; it worked well. We must note, however,
that all of the respondents live in one home only.
• Q 4.5 – Marking all the people who live with the respondent in the respondent’s
home.
There were no problems with this question; it worked well. Most respondents
followed the arrows correctly and found it helpful to have the categories broken out.
One respondent wondered where to put her step-siblings, but put them with other
unrelated people.
• Q 4.6 - Number of people living in household.
12

There were no real problems with this question; it worked well. One respondent
wondered whether or not to include himself, and ultimately did.
• Q 4.7 – Number of homes.
No problems.
• Q 4.8 – Consider one of the homes as main, or consider all homes as equal.
No problems.
• Q 4.9 – Nights spent in each home.
No problems.
• Q 4.10 – People who move with respondent from home to home.
No problems.
• Q 4.11 – Status of respondent’s home(s) as group home or halfway house.
Seven out of eight respondents did not know what a group home or halfway house
was. Guesses ranged from houses with multiple families to houses that were split in
the middle or where half the house is occupied by someone else.
Recommendation: Give definitions of group home and halfway house, including
terms for these that are more familiar to youth.
• Q 4.12 – Parent’s living arrangement.
There were no real problems with this question; it worked well. One respondent
answered with her biological parents in mind, whom she doesn’t really know but
thinks of as her parents rather than her adoptive parents. Seven out of eight
respondents were able to distinguish this item from 4.5, which asked about where
they themselves lived.
Items 4.4-4.12 were easy to answer for most respondents possibly due to the sample
we interviewed. Most of the sample belonged to relatively affluent, highly educated
small families where they lived with their married, biological parents and siblings in
one home.

SECTION 5
• Q 5.1 and 5.2 – Onset of puberty.
Of the eight respondents who answered, all boys and girls were able to follow the
skips correctly. All the girls knew what “menstrual period” meant, and some
contributed alternative terms such as “monthly visitor” or “Aunt Flo.” Two of the
four boys who answered paused when thinking about the onset of puberty. One of the
13

boys was not sure about when puberty started for him, and another had to think about
a memorable moment when the changes started occurring.
• Q 5.6 – Fear of physical abuse by dating partner.
No issues were raised. Most respondents defined dating as having a
girlfriend/boyfriend. All except one respondent thought that dating was the exact
opposite of never having dated anyone. The one exception pointed out that a person
may really like another and does everything with them that is done by people who are
dating, but the other person may think of the relationship as just friends.
• Q 5.7. – Spoken with doctor or nurse about sex, birth control or STDs in last year.
For all seven respondents, birth control refers to methods for preventing pregnancy.
However, for three, birth control means “the pill” and nothing else. No one mentioned
condoms.
Two respondents defined sex to include oral and anal sex, while five defined it as
sexual intercourse. One also included the consequences of sex as well, such as
pregnancy.
Two out of seven respondents thought we should include health teachers in the list of
people talked with, one respondent thought we should include guidance counselors,
and two others thought we were referring to all people they would talk to.
Recommendation: Define what is meant by “birth control” in the question, and
offer examples. Define what we mean by “sex.” Be clearer on who we want the
respondents to include as people they spoke with, just a doctor or nurse, or other
as well.
• Q 5.8 – Likelihood of using condom if having sexual intercourse before marriage.
Two female respondents did not think the question applied to them, however one
specified that that was because she thought she was too young. One girl specifically
noted that the question would apply to girls, because they have a choice regarding
condom use and should voice it.
Another respondent said that she was not sure of the timeframe for this question, so
thought about college and after as a potential time.
Recommendation: Consider adding “the first time” to the end of the question, to
help respondents focus on the timeframe.
• Q 5.9 – Reasons for NOT having sexual intercourse.
Three out of nine respondents left 5.9.f. blank. 5.9.f. offers the following reason not
to have sex: “My boyfriend / girlfriend would lose respect for me.” All three did not
have a boy/girlfriend, and therefore felt they could not answer.
When asked, four respondents thought there should be an added category of “I don’t
want to.”
14

Three respondents suggested other reasons for people their age choosing not to have
sex. Some included: would be too embarrassed, fear of ridicule by peers, and not
having a girlfriend or boyfriend.
Recommendation: Take 5.9.f. out of the list and make it a standalone question or
add a N/A response category for it. Consider adding “I don’t want to” as a
reason. Consider a reason gay and lesbian respondents could answer to.
COMMON SECTIONS 6 AND 7
SECTION 6 - ALCOHOL AND DRUG USE
• Q 6.1 - Number of days had an alcoholic drink in the last 30 days.
Only one out of six sexually active respondents responded with something other than
zero to this question. One respondent could not read the word “alcoholic” (the same
respondent who could not read “intercourse” in earlier questions). However, he was
able to read other terms in the question such as beer and wine and was able to guess
correctly what we were asking and answered appropriately. Only the respondent who
had had any alcohol in the last three months said that his consumption differed over
the course of the year (in baseball season, he would drink less). The others said their
responses were typical of their consumption year-round. No respondent said their
response was affected by the fact that the activity was illegal.
Five of the nine non-sexually active respondents answered zero to this question. One
respondent answered >25 incorrectly – it should have been zero, and later
acknowledged the reason was not having read the whole question. Of the rest, recall
of number of days was reported to be easy. Four respondents stated their answers
were not affected by the fact that this activity is illegal, while three respondents
thought that other people might let their answers be affected by the illegality of the
activity.
Recommendation: Emphasize or add to the reassurance in the introduction of
the question “….Please remember, everything you say will be kept private.”
• Q 6.3 – Number of days used marijuana in the last 30 days.
One sexually active respondent could not read the word “marijuana” (the same
respondent who could not read “alcoholic” and “intercourse” in earlier questions).
However, he still was able to read other terms in the question such as weed and pot
and was able to guess correctly what we were asking and answer appropriately. For
the one respondent who had used marijuana in the last 30 days, the illegality of the
activity did not seem to affect his response. There were no significant suggestions of
alternative terms other than weed or pot.
For non-sexually active respondents, two of the nine respondents answered other than
zero. All respondents stated their answers were not affected by the fact that this
activity is illegal, although one respondent thought that other people might let their
answers be affected by the illegality of the activity. There were no significant
suggestions of alternative terms other than weed or pot.
15

Two respondents out of the 15 who answered this question (one sexually active and
the other non-sexually active) skipped 6.3. Likely they skipped from 6.1 to 6.4
inadvertently.
Recommendations: Emphasize or add to the reassurance in the introduction of
the question. Consider formatting to reduce the risk of people skipping 6.3.
• Q 6.4 - Ever used any other type of illegal drug, prescription drugs that were not
prescribed for you, or an inhalant.
All six sexually active respondents answered no to this question. Of three respondents
who were debriefed, two reported that they did not think that the illegality of this
activity would affect their responses. One thought that respondents might not be
honest, but she thought people would exaggerate their drug use instead of hide it. She
noted, “Where I come from, people won’t be scared to talk about drugs or alcohol.”
She did think we should clarify that we wouldn’t tell police or teachers about people’s
responses. Regarding if there were any drugs respondent thought they should not
include, one respondent mentioned that he did not include cough syrup.
All nine non-sexually active respondents answered no to this question. Six were
debriefed. One mentioned that he did not consider prescription, legal drugs in
answering. It is unclear if this was because he had not used them in a way not
prescribed for him. Another wanted clarification on what “prescription” meant and a
second thought “prescription” meant “medicine drugs.” At least one other respondent
thought of ADHD medicine that “everyone” in her school took for exams. From their
comments, it seems there may be a need for defining prescription drugs.
One respondent said that the illegality of the activity would have affected his answer,
if he had used any of these drugs.
Recommendation: This question contains a lot of different elements. Simplify
and clarify it. Explain prescription drugs and the fact that they are supposed to
be had only by the person they are prescribed for. Also, emphasize or add to the
reassurance in the introduction to the question.
SECTION 7 – FRIENDS AND RELATIONSHIPS
• Q 7.1 – Pressure from friends to have sex.
One out of six sexually active respondents skipped this question.

16