Examining Equity of the Administration for Children and Families' Healthy Marriage and Responsible Fatherhood (HMRF) Client Surveys

Instrument 3:

Healthy Marriage Youth Focus Group and Interview Protocol

HM Youth Focus Group or Interview Introduction

The session will be a facilitated discussion about surveys that are completed by Healthy Marriage youth program clients, such as whether and how relevant the questions are to participants’ lives and whether the questions are respectful of different life experiences, cultures, values, and beliefs. Illustrative probes are included before the survey questions to encourage this discussion, but moderators may adjust probes depending on timing and relevance.

Check in participants

1. RECORD ATTENDANCE IN A STAND-ALONE FILE THAT CAN BE DELETED/SHREDDED

2. ASK IF THEY HAVE THEIR SURVEY PACKET NEAR BY

3. ONCE EVERYONE HAS ARRIVED, START THE INTRODUCTION

Introduction

Thanks for helping us today! My name is [NAME], and this is my colleague, [NAME]. We work for a research company called Mathematica. Mathematica was contracted by the Administration for Children and Families (or ACF), which is part of the federal government. ACF provides funds for healthy marriage and relationship education classes, such as [PROGRAM NAME] and other services for people your age.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #:0970-0630, Exp: 3/31/2027.

We’re here today because we want to get your feedback and thoughts about the surveys that clients in healthy relationship classes take at the beginning and end of workshops and how to improve them. We want to know how well the questions in these surveys reflect the experiences, values, and beliefs of people from different races, ethnicities, backgrounds, and cultures. You can help us make sure the questions are clear, make sense based on your experiences, and help us learn if there are any topics you think are missing. Your feedback will help us improve the questions we ask other youth who participate in the healthy relationship classes that use these surveys.

Although we don’t expect you to speak for everyone, we consider each of you an expert in how well our survey works for people your age and with similar life experiences. Thank you again for your help.

[SPANISH-SPEAKING GROUP: We are also interested in your thoughts on the Spanish translation. During our discussion, please tell us anything that you think could be improved.]

For your reference, we’ve compiled the selected questions into one survey packet. This includes a few questions from surveys you took at the beginning of the program and the survey you just completed. If you completed the surveys on a computer, you might not have been asked all these questions because the surveys are programmed to skip questions that do not apply to your situation, based on your answers to earlier questions. Our discussion will take about an hour today.

Informed assent information [if assent not already collected]

Before we get started, I’d like to be sure you are aware of a few things.

• [IF SITE USING HARD COPY ASSENT FORMS]: Please open your survey packet and take out the top form. This is the assent form. Your parent or guardian has already provided permission for you to participate, but you can now decide for yourself. Please take a moment to read through it right now. I will then cover the key points.

• [IF SITE USING ELECTRONIC ASSENT FORMS]: Please use your phone to scan the QR code showing on the screen right now. This is the assent form. Your parent or guardian has already provided permission for you to participate, but you can now decide for yourself. Please take a moment to read through it right now. I will then cover the key points.

• First, your participation is completely voluntary. That means you can decide if you want to participate and can stop participating at any time without any reason or consequence.

• During the discussion, you can decide not to answer any questions that I ask the group.

• Your responses will not be used in a way that allows anyone outside the research team to identify you.

• We ask that everyone in this group respect each other’s privacy by NOT talking about the details of our discussion with anyone outside the group. However, we want everyone to understand that because this is a group discussion, we cannot guarantee your privacy of what you say in front of each other.

• When we share what we learned during today’s discussion we will NOT use your name or any other information that could identify you. We will only share information about you if required by law—that means that if we learn that you have been hurt or are in danger or if you tell us that you plan to seriously hurt yourself or someone else, then, by law, we must report it to the appropriate legal authorities.

• We will video-record and take notes during our discussion. This is to help us remember all of your opinions and ideas when we make improvements to the survey questions. We will not share this recording with anyone outside of our study team and will destroy it once the study is finished.

Does anyone have any questions at this point?

• Please sign the assent form and [IF HARD COPY: (NAME) will come by and pick them up and mail them back to us/IF ELECTRONIC: Please sign it electronically and then press submit so that we receive it.]

Informed assent information [if form already signed]

Before we get started, I’d like to be sure you are aware of a few things.

• First, your participation is completely voluntary. That means you can decide if you want to participate and can stop participating at any time without any reason or consequence.

• During the discussion, you can decide not to answer any questions that I ask the group.

• Your responses in the discussion will not be used in a way that allows anyone outside the research team to identify you.

• We ask that everyone in this group respect each other’s privacy by NOT talking about the details of our discussion with anyone outside the group. However, we want everyone to understand that because this is a group discussion, we cannot guarantee your privacy of what you say in front of each other.

• When we share what we learned during today’s discussion we will NOT use your name or any other information that could identify you. We will only share information about you if required by law—that means that if we learn that you have been hurt or are in danger or if you tell us that you plan to seriously hurt yourself or someone else, then, by law, we must report it to the appropriate legal authorities.

• As we described in the assent form you signed, we will video-record and take notes during our discussion. This is to help us remember all of your opinions and ideas when we make improvements to the survey questions. We will not share this recording with anyone outside of our study team and will destroy it once the study is finished.

Does anyone have any questions at this point?

Demographic survey questions [if not previously administered]

Next, we would like to ask that you please complete a very short set of questions that ask about your basic demographics. These questions are very important for us to help us make sure that we are collecting information that represents the wide diversity of people who participate in these kinds of workshops. [IF HARD COPY: Please remove the second sheet from your survey packet, called Demographic Questions/[IF ELECTRONIC: Please scan this QR code to pull up the Demographic Questions. Please take a moment to complete these questions.

Thank you. [IF HARD COPY: (NAME will come by and pick them up and mail them back to us/IF ELECTRONIC: Please hit the submit button so that we receive it.]

Focus group guidelines [focus group only]

1. Let’s go around and each say your name and your favorite (movie, food, TV show). You can also say your pronouns if you would like.

2. Some of the topics we talk about today may be things you don’t talk about a lot with your friends or family. If we ask anything that you don’t want to answer or that makes you feel uncomfortable – please just let me know and we can skip that question.

3. Our goal for today is to get your feedback on how these questions were worded and your thought process for answering these questions, so you can help us improve the questions. As we go through specific questions, you don’t need to tell us what your answer on the survey was or would be.

4. Each of you have a green and red card in front of you. We will use these throughout the discussion to get feedback on the survey questions you liked and did not like. For example, sometimes I might ask you to look at a specific question and ask everyone to hold up the green card if you like the question or the red card if you don’t like it, then we can discuss why folks felt the way they did. Remember there is no right or wrong answer and we are not trying to get everyone to agree on something. It’s okay if everyone has different thoughts and ideas. It is possible you may not feel either way about a question, and that is okay too. If that is the case, you don’t need to hold up a card.

5. We understand that participating in a group like this might be new to some of you. Before we begin our discussion, we want to establish a few group rules. Please feel free to add to this if you have other things we should consider as a group:

• There are no right or wrong answers to our questions. We just want to know your true opinions so we can help make the surveys better for people your age and with similar life experiences—all feedback is welcome!

• Remember to be respectful of everyone’s opinions, even if they are different from your own. We are really interested to hear if you have different thoughts and opinions about these survey questions, and welcome that discussion. We just want to always be kind and respectful.

• Please do not share what we discuss today outside of this room. It is up to each of you to respect one another’s privacy and honor this request. 

• Do you have any questions before we begin or anything to add to our group rules?

Interview introduction [interview only]

Before we start our discussion, I want to go over a few more things.

• Some of the topics we talk about today may be things you don’t talk about a lot with your friends or family. If we ask anything that you don’t want to answer or that makes you feel uncomfortable – please just let me know and we can skip that question.

• Our goal for today is to get your feedback on how these questions were worded and your thought process for answering these questions, so you can help us improve the questions. As we go through specific questions, you don’t need to tell us what your answer on the survey was or would be.

  • There are no right or wrong answers to our questions. We just want to know your true opinions so we can help make the surveys better for people your age and with similar life experiences—all feedback is welcome!

• Do you have any questions before we begin?

BEGIN DISCUSSION. SESSIONS ARE BEING RECORDED. CHECK THAT RECORDING HAS BEGUN. GO TO DATA COLLECTION PROTOCOL.

DATA COLLECTION PROTOCOL

USE FOR EITHER FOCUS GROUP OR INTERVIEW

Note on probes

The probes in the data protocol are written for focus groups. Interviewers should modify the language to suit individual interviews.

High level questions

Let’s start by discussing your overall thoughts on the surveys. For this first part, think about the exit survey you just completed, as well as the surveys you took at the beginning of the program.

1. If you were to describe your experience responding to the survey questions in one word, what would it be?

2. Overall, how relevant were the questions to your life and your goals?

3. Were there topics that you think the questions overemphasized or were given too much attention?

4. Were there topics that you think needed more emphasizing or additional questions?

5. Were there important topics about relationships, family, or other areas that were missing?

6. Thinking about the overall tone of the questions, would you say they were more positive, negative, or mixed? For example, do you think the questions seemed overly positive or overly negative? Why?

• As we discuss specific questions in the next part of the conversation, please feel free to suggest where and how you think the tone or wording of questions could be improved.

7. How comfortable were you answering the survey questions at the beginning of the program? What about at the end?

8. How did you feel about the time needed to respond to the questions? Too long, too short, just right? Why?

9. Anything else to add about your experience taking the surveys?

ENT/EXIT-A. ATTITUDES ABOUT MARRIAGE AND RELATIONSHIPS

Probes

Now let’s look at the “Attitudes about Marriage and Relationships” section. Raise up either your green or red card to express your feelings about this section of the survey. [PROBE FURTHER BASED ON THE CARDS HELD UP]. Did anything stick out to you in particular, either positive or negative?

Let’s look at question A1, and the statements about marriage.

• Do you think anything about the questions should be changed? If yes, what should be changed? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Do you think these statements are relevant, or make sense, to your life and your goals for the future? Which ones are relevant? Which ones are not relevant? What about to other people your age?

• Do you think that these statements are respectful of your culture and racial or ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• Are there any attitudes about marriage or relationships that you think are missing? What other relationship or marriage goals do you think people your age and with your life experiences have?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION: For A1X, let’s use the green and red cards to share how you feel about the question. Hold up your green card if you like the question and red if you dislike it.

• What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there an easier or better way to word this?

First, we would like to hear about your attitudes about marriage.

A1. How much do you agree or disagree with the following statements about marriage?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. There is only one true love out there who is right for me to marry	1 	2 	3 	4 
b. In the end, feelings of love should be enough to sustain a happy marriage	1 	2 	3 	4 
c. Living together before marriage will improve a couple’s chances of remaining happily married	1 	2 	3 	4 
d. It is okay to live with a boyfriend/girlfriend without being married	1 	2 	3 	4 
e. It is okay to live with a boyfriend/girlfriend without a plan to be married	1 	2 	3 	4 
f. It is okay to have kids without being married	1 	2 	3 	4 
g. It is okay to have kids without a plan to be married	1 	2 	3 	4 
h. It is better for children if their parents are married	1 	2 	3 	4 
i. Living together is just the same as being married	1 	2 	3 	4 

Probes

Now let’s look at the questions in A2 and the options for responding to them.

• What do you think about these questions?

• How did you feel about being asked these questions?

• Do you think anything about the questions should be changed? If yes, what should be changed? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Do you think these questions are relevant, or make sense, to your life and your goals for the future? What about to other people your age?

• Do you think that these questions are respectful of your culture and racial and ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For A2x, let’s use the green and red cards to share how you feel about the questions. Hold up the green card if you like the question and red if you dislike.

• What did you like/not like about it?

• QUESTION SPECIFIC QUESTIONS (IF NOT ALREADY DISCUSSED):

• For letter “a” what does healthy mean to you?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

A2. When you think of your future, what do you think are the chances that…

	SELECT ONE RESPONSE PER ROW
	ALMOST NO CHANCE	SOME CHANCE BUT PROBABLY NOT	A 50-50 CHANCE	A GOOD CHANCE	ALMOST CERTAIN
a. You will have a healthy marriage with one person for life?	1 	2 	3 	4 	5 
b. You will live with a boyfriend/ girlfriend without being married?	1 	2 	3 	4 	5 
c. You will have a child without being married?	1 	2 	3 	4 	6 

Probes

Now let’s look at A3.

• Did anything stick out to you in particular, either positive or negative? What do others think? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?]

• How did you react to being asked this question? How do you think people your age will react to being asked about some of the statements in this question? Do you think people your age will feel comfortable providing honest answers to these questions?

• Do you think these questions are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• Do you think that these questions are respectful of your culture and racial and ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• Are there any aspects of healthy or unhealthy relationships that you think are missing here, especially for people your age and with your lived experiences?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. FOR A3X, Let’s use the green and red cards to share how you feel about the questions. Hold up the green if you like the question and red if you dislike.

• What did you like/not like about it?

• ITEM SPECIFIC QUESTIONS (IF NOT ALREADY DISCUSSED):

  • Let’s look at A3b, what does “call each other names” mean to you? [AFTER DISCUSSION, IF NEEDED: Can you think of a better way to ask this?]

  • Now let’s look at A3e. Can someone explain what it means to “not argue”? Is that what others thought as well?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

A3. In a healthy relationship, how important is it that couples…

	SELECT ONE RESPONSE PER ROW
	NOT AT ALL IMPORTANT	A LITTLE IMPORTANT	PRETTY IMPORTANT	VERY IMPORTANT
a. Do not cheat on each other?	1 	2 	3 	4 
b. Do not call each other names?	1 	2 	3 	4 
c. Do not threaten each other?	1 	2 	3 	4 
d. Do not push, shove, hit, slap, or grab each other?	1 	2 	3 	4 
e. Do not argue?	1 	2 	3 	4 
f. Encourage each other when life is hard?	1 	2 	3 	4 
g. Enjoy spending time together?	1 	2 	3 	4 
h. Do not fear each other?	1 	2 	3 	4 

Probes

Now let’s look at A4.

• Should anything about this question be changed? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Do you think these statements are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• Is there anything else about communication in a relationship that you think we are missing?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For A4X, let’s use the green and red cards to share how you feel about the questions. Hold up the green if you like the question and red if you dislike.

• What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

A4. How much do you agree or disagree with the following statements about relationships?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. In a healthy relationship, it is essential for couples to talk about things that are important to them.	1 	2 	3 	4 
b. Even in a good relationship, couples will occasionally have trouble talking about their feelings.	1 	2 	3 	4 
c. A relationship is stronger if a couple doesn’t talk about their problems.	1 	2 	3 	4 

Probes

Now let’s look at A5.

• Did anything stick out to you in particular, either positive or negative? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• How did you react to being asked this question? How do you think people your age will react to being asked about some of the statements in this question? Do you think people your age will feel comfortable providing honest answers to these questions?

• Do you think these statements are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For A5x, let’s use the green and red cards to share how you feel about the questions. Hold up the green card if you like the question and red if you dislike.

• What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

A5. How much do you agree or disagree with the following statements about relationships?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. A person who makes their partner angry on purpose deserves to be hit.	1 	2 	3 	4 
b. Sometimes physical violence, such as hitting or pushing, is the only way to express your feelings.	1 	2 	3 	4 
c. Violence between dating partners is a personal matter, and people should not interfere.	1 	2 	3 	4 
d. It’s okay to stay in a relationship even if you’re afraid of your boyfriend/girlfriend.	1 	2 	3 	4 

Probes

Now let’s look at A6.

• Should anything about this question be changed? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Are there other types of relationships that should be included here?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For A5x, let’s use the green and red cards to share how you feel about the questions. Hold up the green card if you like the question and red if you dislike.

• What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

A6. Since completing the program, have you…

	SELECT ONE RESPONSE PER ROW
	YES	NO	I AM NOT IN THIS TYPE OF RELATIONSHIP
a. Ended a relationship that was emotionally unhealthy or abusive?	1 	2 	na 
b. Ended a relationship that was just not working for you?	1 	2 	na 
c. Ended a relationship that was physically unhealthy or abusive?	1 	2 	na 

Section wrap up

Thank you so much for the very helpful feedback. Before moving on to the next set of questions, does anyone have anything else to share here?

Alright, now let’s move to the next section on “Attitudes About Sex”. Again, I’ll have some specific questions but please feel free to bring up anything else you would like.

ENT/EXIT-B. ATTITUDES ABOUT SEX

Probes

Let’s start with overall feelings about this section of the survey. Raise up either your green or red card to express your feelings about the questions on attitudes about sex. [PROBE FURTHER BASED ON THE CARDS HELD UP].

Now let’s turn to question B1.

• Did anything stick out to you in particular, either positive or negative? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• How did you react to being asked this question? How do you think people your age will react to being asked about some of the statements in this question? Do you think people your age will feel comfortable providing honest answers to these questions?

• Do you think these statements are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• Do you think that these questions are respectful of your culture and racial and ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For B1X, let’s use the green and red cards to share how you feel about the questions. Hold up green if you like the question and red if you dislike.

• What did you like/not like it?

• ITEM SPECIFIC QUESTIONS (IF NOT ALREADY DISCUSSED):

• Let’s look at B1a. When you read this, what does “someone they love” mean to you?

• At B1b, what does lifelong commitment mean to you?

• Can you tell me what B1f means, in your own words? What do others think?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

B1. How much do you agree or disagree with the following statements about sex?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. A person should only have sex with someone they love.	1 	2 	3 	4 
b. A person should only have sex if they are married or made a lifelong commitment.	1 	2 	3 	4 
c. MALE: I would be upset if I got someone pregnant at this age. / FEMALE: I would be upset if I got pregnant at this age.	1 	2 	3 	4 
d. I would feel comfortable having sex with someone I was attracted to but didn’t know very well.	1 	2 	3 	4 
e. Two people who are in love do not need to use condoms/ birth control.	1 	2 	3 	4 
f. At my age right now, having sexual intercourse would create problems.	1 	2 	3 	4 
g. At my age right now, it is okay to have sexual intercourse if I use protection, like a condom, the pill, etc.	1 	2 	3 	4 

Probes

• Now let’s look at B2. What did you think about this question? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?]

• How did you react to being asked this question? How do you think people your age will react to being asked about some of the statements in this question? Do you think people your age will feel comfortable providing honest answers to these questions?

• Can someone tell me what B2b means, in their own words? What do others think?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH ITEM. For B2X let’s use the green and red cards to share how you feel about each question. Hold up green if you like the question and red if you dislike.

• What did you like/not like about it?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

B2. How much do you agree or disagree with the following statements about sex?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. If my boyfriend/girlfriend wanted to have sex, but I didn’t, I would find it pretty hard to say “no”.	1 	2 	3 	4 
b. I feel comfortable saying “no” even if my friends are having sex.	1 	2 	3 	4 

Section wrap up

Thank you so much for the very helpful feedback. Before moving on to the next set of questions, I have a few wrap up questions for you all.

• Do you think these questions covered concepts that are important to people your age when deciding if and when they want to have sex?

• Is there anything that should be dropped?

• What other aspects of sexual relationships are we missing?

• Is there anything we could do to make youth more comfortable answering these questions?

Alright, now let’s move to the next section called “Relationship Skills”.

ENT/EXIT-C. RELATIONSHIP SKILLS

Probes

Let’s start with overall feelings about the questions on relationship skills. Raise up either your green or red card to express your feelings about this set of questions. [PROBE FURTHER BASED ON THE CARDS HELD UP].

As we start to review individual questions, some questions are based on relationship status. We are still going to walk through all of them and want to hear your feedback regardless of your relationship status.

Now let’s look at C1.

• What types of relationships do you think people would consider a “yes” response here?

• How do you think other people your age might feel about answering this question? Do you think people your age will feel comfortable providing honest answers to it?

• Are there other ways to ask this question?

C1. Do you have a boyfriend/girlfriend?

1  Yes GO TO C2

0  No GO TO C1a

Probes

Now let’s look at C1a.

• Did anything stick out to you in particular, either positive or negative? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Do you think these questions are relevant, or make sense, to your life right now? What about to others your age who might not be in a relationship?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH ITEM. For C1X, let’s use the green and red cards to share how you feel about the question. Hold up green if you like the question and red if you dislike.

• What did you like/not like about it?

• ITEM SPECIFIC QUESTIONS (IF NOT ALREADY DISCUSSED):

  • What does “conflict” mean to you in C1a_a? What do others think?

  • What does a romantic relationship mean to you?

  • What does it mean to make a romantic relationship last? (C1a_b)

  • What does C1a_c mean to you, in your own words?

  • What does C1a_e mean to you, in your own words? What does “bad relationship” mean to you? Do others agree/disagree? Why?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

[ASK IF BOYFRIEND/GIRLFRIEND = NO]

C1a. How much do you agree or disagree with the following statements?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. I believe I will be able to effectively deal with conflicts that arise in my relationship.	1 	2 	3 	4 
b. I feel good about my ability to make a romantic relationship last.	1 	2 	3 	4 
c. I am very confident when I think of having a stable, long-term relationship.	1 	2 	3 	4 
d. I have the skills needed for a lasting, stable romantic relationship.	1 	2 	3 	4 
e. I am able to recognize the warning signs of a bad relationship.	1 	2 	3 	4 
f. I know what to do when I recognize the warning signs of a bad relationship.	1 	2 	3 	4 

Probes

Now let’s look at C1b.

• Did anything stick out to you in particular, either positive or negative? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• How did you react to being asked this question? How do you think people your age will react to being asked about some of the statements? Do you think people your age will feel comfortable providing honest answers to these questions?

• Do you think these statements and response options are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH QUESTION. For C1X, let’s use the green and red cards to share how you feel about the questions. Hold up green if you like the question and red if you dislike it.

• What did you like/not like about it?

• For “b”—Do you all think this is a good or bad thing to do when faced with a conflict? Why? What about other questions in this list?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

[ASK IF BOYFRIEND/GIRLFRIEND = NO]

C1b. How good or bad are you at each of the following?

	SELECT ONE RESPONSE PER ROW
	I AM EXTREMELY GOOD AT THIS	I AM GOOD AT THIS	I AM BAD AT THIS	I AM EXTREMELY BAD AT THIS
a. Admitting that you might be wrong during a disagreement	1 	2 	3 	4 
b. Avoiding saying things that could turn a disagreement into a big fight	1 	2 	3 	4 
c. Accepting another person’s point of view even if you don’t agree with it	1 	2 	3 	4 
d. Listening to another person’s opinion during a disagreement	1 	2 	3 	4 
e. Working through problems without arguing	1 	2 	3 	4 

Probes

Now let’s look at C2.

• What did you think about these statements or response options? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?] What do others think?

• Do you think these statements and response options are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• Do you think that these statements are respectful of your culture and racial and ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH ITEM. For C2X, let’s use the green and red cards to share how you feel about the questions. Hold up green if you like the question and red if you dislike.

• What did you like/not like about this question?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this? Do you think people your age will feel comfortable providing honest answers to these questions?

[ASK IF BOYFRIEND/GIRLFRIEND = YES]

C2. Please indicate how often the following things happen with your boyfriend/girlfriend. My boyfriend/girlfriend…

	SELECT ONE RESPONSE PER ROW
	NONE OF THE TIME	SOME OF THE TIME	HALF OF THE TIME	MOST OF THE TIME	ALL OF THE TIME
a. Makes me feel good about myself	1 	2 	3 	4 	5 
b. Pressures me to do risky things I don’t want to do	1 	2 	3 	4 	5 
c. Wants to control what I do	1 	2 	3 	4 	5 
d. Tries to make me look bad	1 	2 	3 	4 	5 
e. Puts down my physical appearance or how I look	1 	2 	3 	4 	5 
f. Insults or criticizes my ideas	1 	2 	3 	4 	5 
g. Blames me for his/her problems	1 	2 	3 	4 	5 

Probes

Now let’s look at C3.

• How did you react to being asked these questions? How do you think people your age will react to being asked about some of the topics?

• Do you think people your age will feel comfortable providing honest answers to these questions?

• Do you think these questions are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• FOR ANY QUESTIONS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH ITEM. For C3, let’s use the green and red cards to share how you feel about the questions. Hold up green if you like the question and red if you dislike.

• What did you like/not like about it?

• What do you think counts as a serious disagreement in this question? How do you think others would define that?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

[ASK IF BOYFRIEND/GIRLFRIEND = YES]

C3. When you have a serious disagreement with your boyfriend/girlfriend, how often do you…

	SELECT ONE RESPONSE PER ROW
	NONE OF THE TIME	SOME OF THE TIME	HALF OF THE TIME	MOST OF THE TIME	ALL OF THE TIME
a. Just keep your thoughts and feelings to yourself?	1 	2 	3 	4 	5 
b. Discuss your disagreements?	1 	2 	3 	4 	5 
c. End up throwing things or hitting something?	1 	2 	3 	4 	5 
d. Keep arguing until you get your way?	1 	2 	3 	4 	5 
e. Yell or shout?	1 	2 	3 	4 	5 
f. Give each other the silent treatment?	1 	2 	3 	4 	5 

Probes

Now let’s look at C4.

• Did anything stick out to you in particular, either positive or negative? [IF MEMBERS OF THE GROUP INDICATE THAT THE QUESTIONS ARE NEGATIVE: Do you have suggestions to make the wording more positive?]

• Do you think these statements are relevant, or make sense, to your life right now? What about to others your age who might be in a relationship?

• Do you think that these statements are respectful of your culture and racial and ethnic identity? If not, which questions, and why?

• Do you think that these statements are respectful of people with different backgrounds? If not, which questions, and why?

• FOR ANY ITEMS NOT DISCUSSED, GO THROUGH EACH ONE, ASKING YOUTH TO SHOW THEIR GREEN OR RED CARD FOR EACH ITEM. For C4x, let’s use the green and red cards to share how you feel about the questions. Hold up green if you like the question and red if you dislike.

• What did you like/not like about it?

• In C4b, what does “really matter” mean to you?

• OTHER POTENTIAL PROBES BASED ON COLOR CARDS: Is the question unclear? Is there a better way to word this?

[ASK IF BOYFRIEND/GIRLFRIEND = YES]

C4. How often do the following things happen with your boyfriend/girlfriend?

	SELECT ONE RESPONSE PER ROW
	NONE OF THE TIME	SOME OF THE TIME	HALF OF THE TIME	MOST OF THE TIME	ALL OF THE TIME
a. My boyfriend/girlfriend can count on me to be there when he/she needs me	1 	2 	3 	4 	5 
b. My boyfriend/girlfriend and I talk about the things that really matter.	1 	2 	3 	4 	5 
c. I am comfortable sharing my thoughts and feelings with my boyfriend/girlfriend	1 	2 	3 	4 	5 

Section wrap up

Thank you so much for the very helpful feedback. Before moving on to the next set of questions, I have a few wrap up questions for you all.

• Do these statements cover the relationship skills you think are important for people your age?

• What other aspects of relationship skills are we missing?

• Is there anything we could do to make youth more comfortable answering these questions?

F. PROGRAM PERCEPTIONS

Probes

Now let’s look at questions F1 – F2.

• Are the questions clear? Are there better ways to word the questions or structure the response options?

• Are there other questions about your experiences in the program that you think should be included?

• Anything else we have not already discussed for these statements and response options? What do others think?

F1. How much do you agree or disagree with the following statements about the program?

	SELECT ONE RESPONSE PER ROW
	STRONGLY AGREE	AGREE	DISAGREE	STRONGLY DISAGREE
a. Since attending the program, I better understand what makes a relationship healthy.	1 	2 	3 	4 
b. I have learned new skills in this program that I plan to use in my relationships.	1 	2 	3 	4 
c. I am confident in my abilities to use the skills and knowledge presented in this program.	1 	2 	3 	4 

F2. On a scale from 1 to 5, overall, how helpful was the program to you?

1  1 – not at all

2  2

3  3

4  4

5  5 – extremely helpful

Entrance/exit survey wrap up

Thank you so much for the very helpful feedback on these questions that people answer when they start and end workshops. Did we miss anything in our discussion that you would like to mention now?

Next, we’ll discuss the survey questions that people answer when they enroll in the program.

ACS. SELECTED QUESTIONS

Probes

Now we’re going to look at a few questions that people answer only when they enroll in the program.

• As you read back over these, does anyone have any feedback on how the questions are worded? What about how people might feel responding to them?

• Look at question D1. What response options are missing from this list?

• For question D2. What do you think of these options? Which ones are relevant for people your age? What is missing?

• Look at question D3. Would someone talk through how they think about answering this question?

A1. How do you describe yourself?

SELECT ONE ONLY

1  Female

2  Male

4  Other (specify)

D1. How or where did you hear about this program?

SELECT ALL THAT APPLY

1 □ My school offered the program or class

2 □ Newspaper ad, billboards, or a flyer

3 □ Radio ad, TV commercial, or news story

4 □ Internet ad or social media such as Facebook, Twitter

5 □ Government agency, such as the Office of Child Support Enforcement, TANF, WIC, child welfare (CPS), parole/probation office, other agency

6 □ Community organization, such as a school, hospital, maternity clinic, doctor’s office, place of worship, Head Start, or Healthy Start center

7 □ Program offered in prison or criminal justice facility

8 □ Program staff in the community or at a community event, such as a street fair

9 □ Word of mouth (friends, family, acquaintances)

10 □ Other

D2. Why did you choose to enroll in this program?

SELECT ALL THAT APPLY

1 □ To learn about being a better parent

2 □ To learn how to improve my personal relationships, such as with my partner or co-parent

3 □ To find a job or a better job

4 □ To meet a school requirement

5 □ My friends were coming

6 □ My spouse/partner asked me to come

7 □ My parole/probation officer told me to enroll in a program like this

8 □ A court ordered me to enroll in a program like this

9 □ None of the above

D3. What is the primary reason you chose to enroll in this program?

SELECT ONE ONLY

1  To learn about being a better parent

2  To learn how to improve my personal relationships, such as with my partner or co-parent

3  To find a job or a better job

4  To meet a school requirement

5  My friends were coming

6  My spouse/partner asked me to come

7  My parole/probation officer told me to enroll in a program like this

8  A court ordered me to enroll in a program like this

9  None of the above

Focus Group/Interview Wrap-up

Now let’s look back over all four sections. We covered topics around attitudes about marriage and relationships, attitudes about sex, relationships skills, and other areas. As you might remember, we want to know how well the questions in these surveys reflect the experiences, values, and beliefs of people from different backgrounds, races, ethnicities, and cultures.

Please take a moment to look back through the survey questions. Do you have any additional comments? Did we miss anything in our discussion that you would like to mention now? This can be about your general thoughts, feedback, or opinions on the surveys and the experience of answering the questions, rather than about specific questions.

Thank you so much for joining the discussion today. If you have any follow up questions, we added our email address in the chat and can stay on a few more moments. Otherwise, you will receive an email with a thank you note and your incentive for participating in today’s group. Thanks, and have a wonderful [day/afternoon/evening].