Instrument 2. Cognitive Interview Protocol

OMB No.: 0970-0355

Expiration Date: 08/31/2024

Instrument 2. SRAENE Co-Regulation Measures Pilot Study: Youth Self-Assessment Survey: Cognitive Interview Protocol

THE PAPERWORK REDUCTION ACT OF 1995

This collection of information is voluntary and will be used to provide the Administration for Children and Families with information to help refine and guide the development of a survey on self- and co-regulation in the area of adolescent pregnancy prevention. Public reporting burden for this collection of information is estimated to average 90 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0355, Exp: 08/31/2024. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Heather Zaveri at hzaveri@mathematica-mpr.com

Summer 2024

Youth Self-Assessment Survey: Cognitive Interview

Discussion Protocol

The session will be a facilitated discussion about the Youth Self-Assessment Survey, to gather feedback from youth about the survey questions and response options. Youth will begin the session by taking the survey. This will be followed by a group discussion about the survey questions. Illustrative probes are included before the survey questions to encourage this discussion, but moderators may adjust probes depending on timing and relevance. The groups may be held in-person or virtually.

Check in participants

1. RECORD ATTENDANCE IN A STAND-ALONE FILE THAT CAN BE DELETED/SHREDDED

2. ONCE EVERYONE HAS ARRIVED, START THE INTRODUCTION

Introduction

Thanks for helping us today! My name is [NAME], and this is my colleague, [NAME]. We work for a research company called Mathematica. Mathematica was hired by the Administration for Children and Families, which is an agency in the federal government within the U.S. Department of Health and Human Services that supports the health and well-being of people.

We’re here today because we want to get your feedback and thoughts about a new survey we have developed. This survey will eventually be given to students who are taking teen pregnancy prevention, sexual health, and healthy relationship education classes, either at school or outside of school. We want to learn how well these survey questions capture the experiences and opinions of youth in grades 9-12. Your participation today will help us make the survey better for later when other youth take the surveys in their classes. You can help us make sure the questions are clear and use words that people will understand, make sense based on your experiences, and help us learn if there are any topics you think are missing. Although we don’t expect you to speak for everyone, we consider each of you an expert in how well our survey works for people your age. Thank you again for your help.

Our session today will take no more than an hour and half. In a few minutes I’m going to ask you to each take a survey. After everyone is done with the survey, we will then have a group discussion so I can hear your thoughts and opinions about the questions on the survey. I have some questions I want to ask you all and we will walk through the questions together and you should chime in anytime to provide your thoughts and opinions.

Informed assent information [in-person group- assents collected in-person at arrival/virtual collected at time of parent consent]

Before we get started, I’d like to be sure you are aware of a few things.

• [When you arrived/Before being allowed to join the virtual group] you were asked to sign a form, called an assent form. Your parent or caregiver has already provided permission for you to participate, but I want to make sure to cover the key points that were in that form so you know what your rights are for participating in this discussion today.

• First, your participation is completely voluntary. That means you can decide if you want to participate and can stop participating at any time without any reason or consequence.

• Your responses in the discussion will not be shared with anyone outside of the study team.

• During the discussion, you can decide not to answer any questions that I ask the group.

• We ask that everyone in this group respect each other’s privacy by NOT talking about the details of our discussion with anyone outside the group. However, we want everyone to understand that because this is a group discussion, we cannot guarantee your privacy of what you say in front of each other.

• When we share what we learned during today’s discussion we will NOT use your name or any other information that could identify you. We will only share information about you if required by law—that means that if we learn that you have been hurt or are in danger or if you tell us that you plan to seriously hurt yourself or someone else, then, by law, we must report it to the appropriate legal authorities.

• We will audio/video-record and take notes during our discussion. This is to help us remember all of your opinions and ideas when we make improvements to the survey questions. We will not share this recording with anyone outside of our study team and will destroy it once the study is finished.

Does anyone have any questions at this point?

Administer Youth Self-Assessment Survey

Let’s get started. First, I would like for you to complete the survey. This is the survey that we are testing and we want your feedback on.

As you work through the survey questions, please make any notes in the margins or circle any questions that you think should be worded differently, any words that you think people your age may not know, anything we are missing on the lists, if the question or answer choices don’t match to your opinions or experiences or you are having a hard time picking one of the answer choices. We are interested in any and all feedback you have for us. There are no right or wrong answers. Once everyone completes the survey, which may take about 10-15 minutes, I will then ask you some questions to get your overall thoughts about it, and then we’ll walk through most of these questions and I will ask you questions about them.

[PASS OUT SURVEYS: IF ELECTRONIC: Please scan this QR code to pull up the survey.

Go ahead and begin. When you are done, please turn it over so I know you’ve completed.

Discussion group guidelines

I’m excited to hear your feedback about the survey so that we can improve it for when other people take it in the future! We’re now going to move into the discussion part of today’s session.

1. We understand that participating in a group like this might be new to some of you. Before we begin our discussion, we want to establish a few group rules. Please feel free to add to this if you have other things we should consider as a group:

• There are no right or wrong answers to our questions. We just want to know your true opinions—all feedback is welcome!

• Remember to be respectful of everyone’s opinions, even if they are different from your own. We are really interested to hear if you have different thoughts and opinions about these survey questions, and welcome that discussion. We just want to always be kind and respectful.

• Please do not share what we discuss today outside of this room. It is up to each of you to respect one another’s privacy and honor this request. 

2. Let’s go around and each say your name and your favorite (movie, food, TV show). You can also say your pronouns if you would like.

3. Some of the topics we talk about today may be things you are not used to talking about. If we ask anything that you don’t want to answer or that makes you feel uncomfortable – please just let me know and we can skip that question.

4. As we go through specific questions, you don’t need to tell us what your answer on the survey was or would be. Our goal for today is to get your feedback on how these questions were worded and your thought process for answering these questions, so you can help us improve the questions.

5. IN PERSON: Each of you have a green and red card. We will use these throughout the discussion to get feedback on the survey questions you liked and did not like. For example, sometimes I might ask you to look at a specific question and ask everyone to hold up the green card if you like the question or the red card if you don’t like it, then we can discuss why folks felt the way they did. Remember there is no right or wrong answer and we are not trying to get everyone to agree on something. It’s okay if everyone has different thoughts and ideas. It is possible you may not feel either way about a question, and that is okay too. If that is the case, you don’t need to hold up a card.

6. VIRTUAL: We are going to ask you to sometimes tell us your opinion by clicking on the thumbs up or thumbs down emojis. Let’s practice doing this together right now. Move your cursor to the bottom of your screen, to the Reactions button. If you don’t see it, look for 3 dots, and click on those 3 dots. You will then see some emojis pop up just above the Reactions button. Click on the thumbs up. Great, looks like everyone did that. Now click on the thumbs down. [PROVIDE SUPPORT IF NEEDED]. Perfect. We will use these throughout the discussion to get feedback on the survey questions you liked and did not like. For example, sometimes I might ask you to look at a specific question and ask everyone to click the thumbs up emoji if you like the question or the thumbs down emoji if you don’t like it, then we can discuss why folks felt the way they did. Remember there is no right or wrong answer and we are not trying to get everyone to agree on something. It’s okay if everyone has different thoughts and ideas. It is possible you may not feel either way about a question, and that is okay too. If that is the case, you don’t need to click on an emoji.

7. Do you have any questions before we begin or anything to add to our group rules?

DATA COLLECTION PROTOCOL

USE FOR COGNITIVE INTERVIEW DISCUSSION

BEGIN DISCUSSION. SESSIONS ARE BEING RECORDED. CHECK THAT RECORDING HAS BEGUN. GO TO DATA COLLECTION PROTOCOL.

High level questions

Let’s start by discussing your overall thoughts on the surveys.

1. If you were to describe your experience responding to the survey questions in one word, what would it be?

2. Were there topics that you think the questions overemphasized or were given too much attention?

3. Were there topics that you think needed more emphasis or more attention?

4. Anything else to add about your experience taking the surveys?

Probes

Now let’s look at what is called Question 2, this question includes several items. [IN PERSON: Raise up either your green or red card/[VIRTUAL: Click thumbs up or thumbs down emojis] to express your feelings about this set of items. [PROBE FURTHER BASED ON THE CARDS HELD UP/EMOJIS].

Did anything stick out to you in particular, either positive or negative?

Let’s talk about some parts of question 2.

• First, look at the answer choices. They asked you to pick whether each statement was very hard, hard, a little hard, a little easy, easy, or very easy.

  • Did you find it difficult or easy to pick an answer?

  • What do you think the middle choice, a little hard, a little easy, meant? Was it helpful to have that choice or not helpful?

  • Look at the list of statements now, these are a-t. Let’s take a few seconds to look at those and focus on the answer choices. Did you feel like you had an answer choice for each statement? If no, what answer choice would you have needed or liked to have? [PAUSE TO ALLOW TIME FOR THEM TO REVIEW AND THINK]

• Now let’s shift and focus on the statements. Were there statements that you had a hard time answering? What were they? Why was it hard for you? [PROBE IF NEEDED: For example, maybe you didn’t know what it meant or you didn’t know which answer to pick]. Did anyone else have a challenge with this statement? What about other statements? [CONTINUE PROBING FOR DEPTH AND COVERAGE].

• Should anything about these be changed? If yes, what should be changed? What do others think?

• Do you think these statements are relevant, or make sense, to you and others your age? Which ones are not relevant or don’t make sense for others your age?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• These questions ask about how people can manage the things they think about and how they feel to help them reach their goals. Are there other ways of managing thoughts and feelings that people use that we did not ask about? Did the questions we ask include more about thoughts or feelings, or was it a balanced combination of thoughts and feelings?

• Let’s look at letter ‘o”- it says, “know what I’m feeling.” What does that phrase mean to you?

  • What comes to mind when we talk about what someone is “feeling”? What are things you would think of? [MODERATOR: IF NEEDED PROBE ON EMOTIONS, FEELINGS, INCLUDING HUNGER, COLD, ETC.]

• Let’s look at letter “p”- it says, “Find ways to make myself study when my friends want to go out.” I want to hear people’s thoughts about this one. What does this mean? Do you think it belongs in this list or doesn’t belong in this list? Why/why not? Do you think everyone will understand this and can answer it or not? What if the question asked, “find ways to make myself study even when my friends want to do other things.” Would that question be easier or more difficult to answer? Tell me more.

2. Below are questions about ways people may feel or act. There are no right or wrong answers. Please do your best to answer honestly.

	SELECT ONE ONLY
How easy or hard is it to…	Very hard	Hard	A little hard, a little easy	Easy	Very easy
a. Set goals for myself.	1 	2 	3 	4 	5 
b. Find a way to stick with my goals, even when it’s tough.	1 	2 	3 	4 	5 
c. Keep track of lots of things going on around me, even when I’m feeling stressed.	1 	2 	3 	4 	5 
d. Resist getting pulled into other people’s drama.	1 	2 	3 	4 	5 
e. Calm myself down when I’m too excited to focus.	1 	2 	3 	4 	5 
f. Stop myself from acting on my feelings without thinking first.	1 	2 	3 	4 	5 
g. Consider all the positives and negatives before making a decision.	1 	2 	3 	4 	5 
h. Resist doing something when I know I shouldn’t.	1 	2 	3 	4 	5 
i. Wait for what I want.	1 	2 	3 	4 	5 
j. Remain calm when things go wrong for me.	1 	2 	3 	4 	5 
k. Resist saying something that I know I will later regret.	1 	2 	3 	4 	5 
l. Think carefully before making a decision.	1 	2 	3 	4 	5 
m. Stay away from situations that could bring trouble.	1 	2 	3 	4 	5 
n. Calm myself down when I’m stressed.	1 	2 	3 	4 	5 
o. Know what I’m feeling.	1 	2 	3 	4 	5 
p. Find ways to make myself study even when my friends want to go out.	1 	2 	3 	4 	5 
q. Talk calmly with someone I disagree with.	1 	2 	3 	4 	5 
r. Know when I start to feel frustrated.	1 	2 	3 	4 	5 
s. Keep my reactions in check when I’m upset.	1 	2 	3 	4 	5 
t. To be a good friend, even when I’m in a bad mood.	1 	2 	3 	4 	5 

Probes

Now let’s look at the next question, question 3, which has 6 items lettered a-f.

• What do you think about these questions?

• Do you think these statements are relevant, or make sense, to you and others your age? Which ones are not relevant or don’t make sense for others your age?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• Let’s look at letter f, it says, “Naming my emotions helps me figure out what matters to me.” What does this mean? Do you think it belongs in this list or doesn’t belong in this list? Why/why not? Do you think everyone will understand this and can answer it or not?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD/THUMBS UP OR DOWN EMOJI FOR EACH QUESTION. For question 3 let’s use the [IN PERSON: green and red cards/VIRTUAL: thumbs up and thumbs down emojis] to share how you feel about the questions. [IN PERSON: Hold up the green card if you like the question and red if you dislike/VIRTUAL: Click on the thumbs up emoji if you like the question or the thumbs down emoji if you don’t like it].

  • What did you like/not like about it?

  • QUESTION SPECIFIC QUESTIONS (IF NOT ALREADY DISCUSSED):

  • Let’s look at letter “e”: I can name the emotions I’m feeling. What does that mean to folks? There is no right or wrong answer, just tell me what this means to you and what you think of when you read this.

  • OTHER POTENTIAL PROBES BASED ON COLOR CARDS/THUMBS UP/DOWN: Is the question unclear? Is there a better way to word this?

3. Please rate how often you do the following:

	SELECT ONE ONLY
	Almost never	Sometimes	Often	Almost always
a. When I’m having a hard time paying attention, I take a few deep breaths to refocus.	1 	2 	3 	4 
b. When I am feeling stressed, I listen to my body’s signals.	1 	2 	3 	4 
c. When I’m in the middle of an argument with my parent or caregiver, I take a break to calm myself down.	1 	2 	3 	4 
d. When I’m feeling overwhelmed, I can calm myself down.	1 	2 	3 	4 
e. I can name the emotions I’m feeling.	1 	2 	3 	4 
f. Naming my emotions helps me figure out what matters to me.	1 	2 	3 	4 

Probes

Now let’s look at question 4, and the 4 statements within the question labeled a-d.

• First, let’s look at the answer choices again. They asked you to pick whether you strongly disagree, disagree, agree, or strongly agree to the statements.

  • Did you find it difficult or easy to pick an answer?

  • Did you feel like you had an answer choice for each statement? If no, what answer choice would you have needed or liked to have? [PAUSE TO ALLOW TIME FOR THEM TO REVIEW AND THINK]

• Now let’s shift and focus on the statements. Were there statements that you had a hard time answering? What were they? Why was it hard for you? [PROBE IF NEEDED: For example, maybe you didn’t know what it meant or you didn’t know which answer to pick]. Did anyone else have a challenge with this statement? What about other statements? [CONTINUE PROBING FOR DEPTH AND COVERAGE].

• Should anything about these be changed? If yes, what should be changed? What do others think?

• Do you think these statements are relevant, or make sense, to you and others your age? Which ones are not relevant or don’t make sense for others your age?

• Do you think people your age will feel comfortable providing honest answers to these questions?

4. How much do you agree or disagree with these statements?

	Select one only
	Strongly disagree	Disagree	Agree	Strongly agree
a. There are adults at this school I could talk with if I had a personal problem.	1 	2 	3 	4 
b. If I tell a teacher that someone is bullying me, the teacher will do something to help.	1 	2 	3 	4 
c. I am comfortable asking my teachers for help with my schoolwork.	1 	2 	3 	4 
d. There is at least one teacher or other adult at this school who really wants me to do well.	1 	2 	3 	4 

• NOTE Q5 OF THE SURVEY INSTRUMENT IS NOT ADMINISTERED TO THE COGNITIVE INTERVIEW PARTICIPANTS,

Probes

Now let’s look at question 6, which is made up of rows a through e. Please do not share your own response to this question, as this may be very personal and I don’t want anyone to feel like they need say how they answered. We’re going to talk about these statements and the response choices generally.

• Now let’s shift and focus on the statements. Were there statements that you had a hard time answering? What were they? Why was it hard for you? [PROBE IF NEEDED: For example, maybe you didn’t know what it meant or you didn’t know which answer to pick]. Did anyone else have a challenge with this statement? What about other statements? [CONTINUE PROBING FOR DEPTH AND COVERAGE].

• Should anything about these be changed? If yes, what should be changed? What do others think?

• Do you think these statements are relevant, or make sense, to you and others your age? Which ones are not relevant or don’t make sense for others your age?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• Some of these questions are pretty similar to questions that are asked earlier in the survey. Were these questions harder to answer, easier to answer, or about the same as the questions above?

• Let’s look now at the answer choices listed here. Did those work ok for you when you were picking your answers for each of these listed below or did they not work ok?

• Now let’s look at each the statements. GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD/THUMBS UP OR DOWN EMOJI FOR EACH QUESTION to share how you feel about the statements. What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS/EMOJIS: Is the question unclear? Is there a better way to word this?

6. How confident are you that you could…

	Select one only
	Not at all confident	A little confident	Somewhat confident	Confident	Very confident
a. Consider all the positives and negatives before making a decision about sex.	1 	2 	3 	4 	5 
b. Think carefully before making a decision about sex	1 	2 	3 	4 	5 
c. Stop yourself from acting on your feelings without thinking first before making a decision about sex	1 	2 	3 	4 	5 
d. Tell your partner your feelings about what you do and do not want to do sexually	1 	2 	3 	4 	5 
e. Know what you are feeling when faced with a decision about sex.	1 	2 	3 	4 	5 

Probes

Now let’s look at question 7, which is made up of rows a through e. Please do not share your own response to this question, as this may be very personal and I don’t want anyone to feel like they need say how they answered. We’re going to talk about these statements and the response choices generally.

• Now let’s shift and focus on the statements. Were there statements that you had a hard time answering? What were they? Why was it hard for you? [PROBE IF NEEDED: For example, maybe you didn’t know what it meant or you didn’t know which answer to pick]. Did anyone else have a challenge with this statement? What about other statements? [CONTINUE PROBING FOR DEPTH AND COVERAGE].

• Should anything about these be changed? If yes, what should be changed? What do others think?

• Do you think these statements are relevant, or make sense, to you and others your age? Which ones are not relevant or don’t make sense for others your age?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• Let’s look now at the answer choices listed here. Did those work ok for you when you were picking your answers for each of these listed below or did they not work ok?

• Now let’s look at each the statements. GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD/THUMBS UP OR DOWN EMOJI FOR EACH QUESTION to share how you feel about the statements. What did you like/not like about it?

OTHER POTENTIAL PROBES BASED ON COLOR CARDS/EMOJIS: Is the question unclear? Is there a better way to word this?

7. How strongly do you agree or disagree with each of the following statements?

	Select one only
	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
a. At your age right now, having sex could create problems.	1 	2 	3 	4 	5 
b. At your age right now, it is okay for you to have sex if you use birth control, like a condom, the pill, etc.	1 	2 	3 	4 	5 
c. A person should only have sex if they are married.	1 	2 	3 	4 	5 
d. At your age right now, it is okay to have sex if you are dating the same person for a long time	1 	2 	3 	4 	5 
e. Teens should wait to have sex until they are out of high school.	1 	2 	3 	4 	5 

Probes

Now let’s look at questions, 8, 9, 10, 11, and 12.

• GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD/THUMBS UP OR DOWN EMOJI FOR EACH QUESTION. For question 7/8/9/10 let’s use the [IN PERSON: green and red cards/VIRTUAL: thumbs up and thumbs down emojis] to share how you feel about the questions. [IN PERSON: Hold up the green card if you like the question and red if you dislike/VIRTUAL: Click on the thumbs up emoji if you like the question or the thumbs down emoji if you don’t like it].What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS/EMOJIS: Is the question unclear? Is there a better way to word this?

8. In the past month, how often did you feel like you could share your thoughts and feelings with your parent(s) or caregivers? (STREAMS)

m None of the time 1

m Some of the time 2

m Most of the time 3

m All of the time 4

9. How important is it to you to talk with your parents/guardians about your decisions related to having sex? Please answer for the parent/guardian whom you feel most comfortable talking to.(MPC Eval)

 Not at all important 1

 Somewhat important 2

 Important 3

 Very important 4

10. How comfortable are you talking with your parents/guardians about your decisions related to having sex? Please answer for the parent/guardian whom you feel most comfortable talking to.

 Not at all comfortable 1

 Somewhat comfortable 2

 Comfortable 3

 Very comfortable 4

11. In general, how much pressure, if any, do you feel from your friends to have sex? (MPC, PREP)

m A lot of pressure 1

m Some pressure 2

m A little pressure 3

m No pressure 4

12. If your dating partner wanted to have sex, but you didn’t, you would find it hard to say “no.”

m Strongly agree 1

m Agree 2

m Disagree 3

m Strongly disagree 4

Probes

Our final question! Let’s look at question 13.

• Tell me what you think this question is asking in your own words.

• Should anything about this question be changed? What do others think?

13. How confident are you that you could say no to drinking or using drugs when you don’t want to? (STREAMS)

m Completely confident 1

m Very confident 2

m Confident 3

m A little confident 4

m Not at all confident 5

Focus Group/Interview Wrap-up:

Those were all the questions I had, but I want to make sure we don’t miss any more feedback from you to help us improve this survey.

Please take a moment to look back through the survey questions. Do you have any additional comments? Did we miss anything in our discussion that you would like to mention now? This can be about your general thoughts, feedback, or opinions on the surveys and the experience of answering the questions, rather than about specific questions.

Thank you so much for joining the discussion today and sharing your thoughts with us. [IF VIRTUAL: If you have any follow up questions, we added our email address in the chat and can stay on a few more moments. Otherwise, you will receive an email with a thank you note and your incentive for participating.] [IN-PERSON: You may now leave and check out at the front desk to receive your incentive for participating.]

Thanks, and have a wonderful [day/afternoon/evening].