Adolescent Development Products Feedback Survey and We Think Twice Digital Media Campaign

Attachment 1. Example Discussion Prompts for Online CMNTY Platform

DISCUSSION PROMPTS FOR REMOTE TESTING:

MESSAGE AND PRODUCT INFORMATION GATHERING

Note to OMB: This document includes examples of potential questions we may ask across the series of remote, online testing activities. We will not ask all of these questions in one 20-minute activity. Rather, we will mix the questions and activities based on the campaign development stage and to ensure we do not overburden participants. Some of the questions also will be customized to reflect the message or product we are testing; we have noted customizable pieces of questions with brackets []. Participants will only be asked to commit 20 minutes or less of their time per activity.

++

Public reporting burden for this collection of information is estimated to take an average of up to 20 minutes per response, including the time for reviewing instructions and providing the information requested. You are not required to respond to any request for information unless it displays a currently valid OMB control number.
_____________________________________________________________________________________

Welcome Page of the Online Community: We Think Twice Insiders

Welcome to The We Think Twice Insiders

We want to bring people together, online, to achieve more than we could on our own. That’s why we've invited you to join our online community. Together, we want to think and talk about relationships, friends, sex, school, social media, growing up, and more. We're building a campaign for teens and young adults like you, and we need YOUR help. Participate in as few or as many activities as you like or have time for! Earn points and gift cards for your time.

GET STARTED! 

1. Go to your profile and update your settings (note to OMB: this is the Appendix A – Screening Instrument) 

2. Add an avatar image. It can be anything - your Bitmoji, a photo of your pet, or a photo of you - completely up to you!

3. Under 18? Have your parent or guardian complete this parent consent form.

4. Then complete your own participant consent form.

Questions? DM a CMNTY moderator or email WTT@rti.org.

About the Campaign

The Administration for Children and Families is sponsoring this project. We are talking with young people across the U.S. about how we can work together to design a national campaign for youth, with youth. The campaign wants to support you in achieving a healthier, happier future by avoiding some of the potential downsides of sexual activity—things like sexually transmitted diseases and teen pregnancy.

The purpose of these online activities is to get your input, ideas, and feedback on campaign materials. You are the experts - on life, struggles, and wins you experience as a youth. It’s your voice that matters, and your voice will direct what this campaign looks like, sounds like and where it will be.

Instructions for each activity

There are no right or wrong answers. We want to know your opinions and what you think about the issues we will be discussing. If at any time you are uncomfortable with the questions, you can choose not to answer or stop participating in the activity. If you bring up a friend or other person you know as an example in your responses, please do not use their last name. You can submit a private message to the site administrator at any time if you have questions about this activity.

1. Insight Questions on Topics and Media Behaviors

Questionnaire Format

1. Have you ever seen information online or on social media about teen pregnancy prevention (e.g., abstinence, safer sex)? Yes/No

[If Q1 = No, skip to Q5]

2. Have you ever seen videos online or on social media related to teen pregnancy prevention? Yes/No

[If Q2 = No, skip to Q4]

3. What did you like and/or dislike about these videos? Open-ended

4. Have you ever shared information found online or on social media about teen pregnancy prevention? By sharing we mean re-posting to your personal social media page or sending information (e.g., posts, videos, photos, etc.) to others. Yes/No

5. Have you seen any videos online or on social media related to setting goals for a better future? Yes/No

[If Q5 = No, skip to Q7]

6. What did you like and/or dislike about these videos? Open-ended

7. What kind of videos do you typically look for online or on social media (e.g., Instagram, YouTube and Snapchat)? Please select all that apply. Checklist.

   • Funny

   • Informational/educational (e.g., tutorial, product review, etc.)

   • Vlogs (video blogs)

   • Entertainment (e.g., sports, music, tv shows, movies, etc.)

   • Videos posted by friends

   • Other: ______________

8. What kind of videos do you typically share with others? Open-ended

9. Do you like/dislike video ads on Instagram, YouTube and Snapchat? Like/Dislike slider

10. How often do you watch video ads on Instagram, YouTube and Snapchat? Never, Rarely, Sometimes, Often, Always

[If Q10 = Never, skip to Q12]

11. Describe the types of video ads you watch on Instagram, YouTube and Snapchat. Open-ended

12. Which channels/influencers do you follow or subscribe to on Instagram, YouTube and Snapchat? By influencers, we mean individuals who have a large online or social media following. Open-ended

13. Do you watch Instagram TV? Yes/No

[If Q13 = No, skip to Q15]

14. How do you discover new shows/episodes? Open-ended

15. Below is a list of products that we may develop for our campaign website. Which of the following products would you engage with (e.g., watch, read, listen to, click on, etc.)? Please mark all that apply. Checklist.

• Click-to-reveal photos

• Image transition sliders

• Listicle

• Music video

• Live video

• Vlog

• Interview

• Interactive video

• Choose-your-own-adventure story

• Interactive timeline

• How-to content

• Online calculator

• Interactive map

• Podcast

• Quiz

16. Which product would you most like to engage with? Please select one. Multiple-choice

• Click-to-reveal photos

• Image transition sliders

• Listicle

• Music video

• Live video

• Vlog

• Interview

• Interactive video

• Choose-your-own-adventure story

• Interactive timeline

• How-to content

• Online calculator

• Interactive map

• Podcast

• Quiz

17. Why did you choose this product as your favorite? Open-ended

18. Below is a list of potential names for new products that we might develop. Which of the following names would you engage with (e.g., watch, read, listen to, click on, etc.)? Please mark all that apply. Checklist.

• 5 Ways to Beat Boredom

• Challenging Negative Self-Talk

• Conflict Resolution 101 video

• Money Saving Hacks

• What Does Success Mean to You?

• Think You Know How Much These Items Cost?

• Video: A Day in the Life

• Video: Healthy Relationships Chat

19. Which product name would you most like to engage with? Please select one. Multiple-choice

• 5 Ways to Beat Boredom

• Challenging Negative Self-Talk

• Conflict Resolution 101 video

• Money Saving Hacks

• What Does Success Mean to You?

• Think You Know How Much These Items Cost?

• Video: A Day in the Life

• Video: Healthy Relationships Chat

20. Why did you choose this title as your favorite? Open-ended

21. Would you visit a website with .ORG in the URL? Yes/No

[If Q22 = Yes, skip to Q23]

22. You said you would not visit a website with .ORG in the URL. Why not? Open-ended

23. Would you visit a website with .GOV in the URL? Yes/No

[If Q23 = Yes, skip to Q25]

24. You said you would not visit a website with .GOV in the URL. Why not? Open-ended

25. What are your favorite websites to visit? List up to 5. Open-ended

Journal Activity (individuals answer questions; no one else can see responses)

Prompts:

1. Are our core campaign topics relevant to you? Thinking about the campaign “buckets” (Dating and Relationships, Smart Choices, Mental Health and COVID-19, and Goals and Success), how relevant are the core campaign topic areas to you and others your age?

2. What are myths you’ve heard about sexual health?

3. Do you think that your generation is actually making strides in body positivity?

4. How would you describe your current relationship with your siblings (if you have any)?

5. What might you say to a friend struggling with their mental health?

6. Where do you see yourself in the next year? In 5 years?

Discussion Forum Activity (participants answer discussion board questions and can see each other’s responses)

Prompts:

1. What kind of jobs are great first jobs for teens?

2. How do you organize yourself when preparing to apply for college?

3. What characteristics make a good leader? What kind of leader do you want to be? What kind of leader would you want to follow?

Poll Activity

Prompts:

1. How are you spending your winter break?

   1. Relaxing

   2. Catching up on school work

   3. College applications

   4. Traveling

   5. Spending time with friends/family

2. Do you have a bank account open?

   1. Yes

   2. No

3. Where do you get your news?

   1. Social media

   2. Podcasts

   3. Word-of-mouth

   4. School

   5. Regular news

Challenge Activity

Prompts:

1. Women in history: Name as many impressive women from history as you can.

2. Historical black figures: Who are some impressive black historical figures?

3. Share We Think Twice Contest Ideas!

4. Topic brainstorm dumps: What topics has We Think Twice not yet covered relevant to you and your peers?

2. Message Feedback

Questionnaire Format

1. Please select how much you disagree or agree with each of the following statements about [insert specific message on SRA topic]:

	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly Agree	Prefer Not to Answer
I liked this message overall.	1	2	3	4	5	99
This message grabbed my attention.	1	2	3	4	5	99
This message was confusing.	1	2	3	4	5	99
This message was convincing.	1	2	3	4	5	99
This message said something important to me.	1	2	3	4	5	99
This message spoke to me.	1	2	3	4	5	99
I learned something new from this message	1	2	3	4	5	99
This message gave me good reasons to [insert desired behavior]	1	2	3	4	5	99
I do not like this message.	1	2	3	4	5	99
This message was offensive.	1	2	3	4	5	99
This message was easy to understand.	1	2	3	4	5	99

3. Product or Material Feedback

Questionnaire, Poll, or Discussion Forum Format

Now I want to show you a sample [insert product] for our campaign. First, I want your feedback on a few different options.

[Show multiple creative concepts for product/material.]

1. Which of these concepts do you like best? Multiple Choice [A/B/C/etc.]

2. What do you like about these concepts? Dislike? Open-ended

3. How would you change these concepts? Open-ended

Now, we want your feedback on the overall look and feel for the [insert product/material], as well as the content.

[Show product or material.]

1. What are your initial impressions of this [product/material]? Open-ended.

2. What do you like? Dislike? Open-ended

3. In your own words, what is the main idea this [product/material] is trying to get across? Which words or phrases grab your attention? Open-ended

4. What feelings do you have in reaction to this [product/material]? Anything positive? Anything negative? Open-ended

5. What do you think about the overall look and feel of this [product/material]? What do you think about the photos/graphics, colors, and font? Open-ended

6. Using the heat map (an interactive feature where youth can click on a spot in the image and leave a note), mark up to 5 areas of the image that you like, and up to 5 that you don’t like. Leave a brief comment for each explaining why.

7. What do you think about the content and text in this [product/material]?

8. What do you think this [product/material] is asking you to do? Open-ended or multiple choice

9. Do you feel motivated to do this behavior? Yes/No/Maybe

10. If you saw this [insert product/material], what’s the first thing you would do? Open-ended

11. Based on this [product/material] alone, would you be interested in seeing more content from this campaign? Yes/No/Maybe

12. How can we improve this [product/material]? Open-ended

13. How would you describe the tone of the [product/material]? You can select more than one or add your own answer. Checklist

    • Uplifting

    • Friendly

    • Empowering

    • Judgmental

    • Shaming

    • Other:_____________________

14. Would you share this [product/material] with others? Yes/No

[If Q13 = No, skip to Q15]

15. Who would you share it with and how? Open-ended

[End questionnaire]

16. Why would you not share it? Open-ended

Additional/unstructured questions to assess WTT products, resources, etc.

Explanation of CMNTY features. Respondents answer using:

• Slider Scale: Youth respond by sliding a marker along a spectrum to indicate their level of agreement with a statement.

• Heatmap: Youth respond by clicking anywhere on an image with a marker labeled “I like it” or “I dislike it”. They then leave a brief comment explaining why they liked/disliked that portion of the image.

• Matrix: Youth respond by clicking To what degree do you agree/disagree with the following statement(s): strongly disagree, disagree, neither, agree, strongly agree”

• Multiple choice: Youth either choose one or multiple response options to a question.

• Open text: Youth write their response in a fillable box.

Feedback on message/idea/draft product

Matrix, Slider, or Multiple Choice: To what degree do you agree/disagree with the following statements (Responses: strongly disagree, disagree, neither, agree, strongly agree)?

• I liked the overall message/idea/design/content.

• This message/idea/design/content grabbed my attention.

• This message/idea/design/content was confusing.

• What about it was confusing (open text)?

• I connected with this message/idea.

• What would you change about this message/idea/design/content (open text)?

• What additional ideas do you have for this product (open text)?

• What is your initial impression of the product (open text)?

• To what degree does the product encourage you to reflect on your own choices about [insert topic here]? (Highly likely, somewhat likely, neutral, not likely, not at all likely)

  • Please explain your answer (open text)

• How likely are you to use the product once it's published? (Highly likely, somewhat likely, neutral, not likely, not at all likely)

  • Please explain your answer (open text)

• Which, if any, words in the document are worded in a way that is confusing or not teen-friendly? How would you re-word it? (open text)

• What was the message you took away from reading/using this product? What do you think is the overall goal of this product? (open text)

Feedback on Visuals

• Of the concepts/visuals (graphics, video, format) displayed, which do you like best (multiple choice) and why (open text)?

• What would you change about the visuals displayed (open text)?

• Use the heatmap feature to mark the areas of the following visual/graphic that you like. Then mark the areas you don't like. Explain why for both.

• How might you make these images more diverse/inclusive of all kinds of teens (open text)?

Appendix A: Screening Instrument for Online Testing Platform Participants

Additional context for OMB: This is a form that each user completes when joining our online testing platform (CMNTY.com). None of this information except for the user’s username is visible to other community members. This information is not linked to the data we collect or analyze about the campaign from participants. The collection of name and email address are required by the online platform to be able to contact participants about new activities and opportunities to provide feedback. The participant characteristics excluding name, date of birth, and email address, are aggregated and anonymized and added to our database of participant characteristics to adequately describe our sample and ensure we are engaging a diverse sample of adolescents in the U.S. Note: We are asking for both date of birth and age on our online platform to double-validate and avoid errors and non-eligible participants.

Note: Other members in the community will only see your username and avatar image. You do not need to add an avatar image if you don’t want to. All other information is only visible to the project team.

• Username (this does not need to be your real name; it can be anything):

• Email address (we will only use this to send you updates about new activities and send you electronic gift cards to reward you for your participation):

• First name:

• Last name:

• Date of birth:

• Age (for double-verifying that they are eligible):

• Zip code: 

• What sex were you assigned at birth, on your original birth certificate?

  • Male

  • Female

• How do you describe your gender identity? MARK ONLY ONE ANSWER.

  • Male

  • Female

  • Transgender

  • Not Sure

  • Something else (specify) _________________

  • I prefer not to answer.

• Which of the following best represents how you identify? MARK ONLY ONE ANSWER.

• Gay (or lesbian)

• Straight, that is not gay (or lesbian)  

• Bisexual

• Something else (specify) ___________________

• Not sure

• I prefer not to answer.

• Please indicate your race. You may select one or more races.

  • White

  • Black or African American

  • Asian

  • Native Hawaiian or Other Pacific Islander

  • American Indian or Alaska Native

  • Other race [Specify: ___________________________]

  • Don’t know

  • Prefer not to answer

• Are you of Hispanic or Latino origin? Select one.

  • Yes

  • No

• What type of school do you attend? 

  • Public

  • Private

  • Charter 

  • Boarding

  • College

  • Other [Specify: _______________________]

16