SSB - Administration for Children and Families Youth Programs – SOGI Items

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Administration for Children and Families Youth Programs – Sexual Orientation and Gender Identity (SOGI) Items

Pre-testing of Evaluation Data Collection Activities

0970 - 0355

Supporting Statement

Part B

October 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Caryn Blitz, Tia Brown

Part B

B1. Objectives

Study Objectives

The Administration for Children and Families (ACF) is currently engaged in efforts to improve participant survey questions on sexual orientation and gender identity (SOGI) to better represent the range of program participants, as well as to align with Executive Order (EO) 14075 (Advancing Equality for Lesbian, Gay, Bisexual, Transgender, Queer, and Intersex Individuals. This agency-wide initiative aims to select SOGI items to be used with youth served by ACF, including youth participating in the Personal Responsibility Education Program (PREP) and the Sexual Risk Avoidance Education Program (SRAE).

This information collection request includes pretesting SOGI items with youth through virtual cognitive interviews and in-person focus group discussions. The pretest with youth will explore how youth understand the SOGI items, the relevance of the language used, and evaluating the sensitivity of the revised items and usefulness in real world application. Pretests will be conducted with both English- and Spanish-speaking youth.

ACF proposes the following guiding questions for the SOGI items pretest with youth:

• Do the youth respondents easily understand the questions and response options?

• Are the items inclusive and sensitive to youth from various backgrounds?

• Do the items accurately and adequately capture how youth self-identify?

Generalizability of Results

The proposed youth pretests are intended to inform instrument development for ACF data collections (including the Personal Responsibility Education Program (PREP) and SRAE participant surveys approved under OMB #: 0970-0497 and 0970-0536, respectively) by obtaining feedback from youth on the SOGI items. It is not designed to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

Data collected under this generic clearance will be used to inform the selection of SOGI measures for youth in ACF programs and the revision of PREP and SRAE participant surveys, which will further ACF’s learning agenda. Cognitive interviews will explore how youth interpret the SOGI items, including how they understand and operationalize the SOGI measures. Cognitive interviews with Spanish-speaking youth will also confirm translation of items are interpreted as intended. The cognitive interviews will be conducted with youth ages 12-19 who test new content and engagement strategies for a market research vendor, Canadian Viewpoint (CanView). Working with Canview, who has access to a wide range of participants, allows for efficient recruitment of a diverse testing pool. The in-person focus groups will help assess the sensitivity of the revised items and real-world application. The focus groups will be conducted with youth who are currently receiving ACF programming.

Group cognitive interviews have been recommended over traditional one-on-one cognitive interviews with a teen population (Norris et al 2014)¹, with the researchers noting that the brainstorming process typical of focus groups allowed them to “find and fine tune alternative wording for items that were unclear or confusing,” and that the measures they revised through this process were completed without difficulty in a later study. This research team also employed red and green cards to signify likes and dislikes in part of their group interviews and found that their population of teen girls was very engaged in making use of the cards; our team plans to employ this same tactic with the in-person focus groups and to mimic this process using “thumbs up” and “thumbs down” reactions in the virtual cognitive interviews. The iterative nature of the data collection effort, beginning with cognitive interviews after which the measures will be revised prior to the focus groups, will enable the research team to test modifications made to the measures based on initial feedback from youth.

As noted in Supporting Statement A, because this is an initial pilot study, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Limitations will be described in any written products associated with this pilot study. 

B2. Methods and Design

Target Population

The target populations for this generic information collection request are youth ages 12 through 19 for the cognitive interviews and focus groups.

Sampling and Site Selection

The study team will conduct up to 30 virtual small-group cognitive interviews, with each group comprised of up to 5 youth, for a total of up to 150 youth. We plan to conduct cognitive interviews with youth across three rounds of data collection; the first two rounds will be conducted with English speaking participants, and the third round will be conducted with Spanish-speaking participants to assess translation and cultural appropriateness of the SOGI items. Each youth participant will be asked to participate in one 60-minute cognitive interview. To maximize participation and the ability to engage a diverse sample, cognitive interviewees will be recruited by a market research vendor, Canview. Canview manages a non-probability-based panel of members across the U.S. from various geographical areas and with a wide range of demographic characteristics. Selecting study participants through Canview allows for highly targeted recruitment that is efficient and cost-effective.

To recruit minor youth, Canview will contact parents/guardians of eligible youth with a description of the study, planned use of data, and confirm their consent. (Appendix B). To recruit youth aged 18 or older, Canview will contact participants directly with a description of the study, planned use of data, and study consent form. Canview will send an email to potential groups of participants (Appendix A) to inform them of the pretest and link them to the eligibility screener (Instrument 1). The screener will collect information on key selection criteria, including age, grade, gender identity, sexual orientation, and primary language. The screener will also collect information on other demographics, including race and ethnicity. Canview will share the de-identified screener responses with the study team, who will carefully review the screener data and conduct purposive, non-probability-based recruitment to ensure that participants are selected based on key criteria. We will also use the screener data to ensure that a diverse range of other demographics are represented across the discussion groups, though we cannot guarantee that cognitive interview participants will exactly reflect the demographics of youth served by ACF programming. Table B.1 provides a breakdown of the targeted number of youth per group for the virtual cognitive interviews, along with the key selection criteria.

Table B.1. Target recruitment numbers and criteria for virtual interview participants

				Number of youth				Total number of groups
Grade	Gender identity	Sexual Orientation		Round 1 target (English)	Round 2 target (English)	Round 3 target (Spanish)	Total
7th and 8th	Cisgender male	Straight		5		5	10	2
7th and 8th	Cisgender female		Straight	5	0	5	10	2
7th and 8th	Any+		LGBTQ+	10	10	10	30	6
9th and 10th	Cisgender male		Straight	5	0	5	10	2
9th and 10th	Cisgender female		Straight	5	0	5	10	2
9th and 10th	Any+		LGBTQ+	10	10	10	30	6
11th and 12th	Cisgender male		Straight	5	0	5	10	2
11th and 12th	Cisgender female		Straight	5	0	5	10	2
11th and 12th	Any+		LGBTQ+	10	10	10	30	6
Total				60	30	60	150	30

⁺ “Any” includes individuals who identify as any gender (cisgender males and females, transgender males and females, non-binary, non-confirming, questioning, etc.)

For the in-person youth focus group discussions, the study team will work with three operating divisions within ACF (the Family and Youth Services Bureau, the Office of Trafficking in Persons, and the Office of Refugee Resettlement) to identify grantee sites from which to recruit youth for the pretest. For the focus groups, we will review the revised versions of the English questions, as well as translated versions of the questions in Spanish, Haitian Creole, and Q’eqchi. Haitian Creole and Q’eqchi are the languages most highly requested for translation for youth by ACF programs, including PREP and SRAE, and the Office of Refugee Resettlement and the Office of Trafficking in Persons. In recruiting youth for the focus groups, we will work with each operating division to identify program sites that serve youth who speak these languages. All youth participating in the programs offered by the selected sites will be eligible to participate in the focus group discussions, provided they assent and that the consent requirements of the site and its Institutional Review Board (IRB) are met. We will coordinate with the program sites to collect parental consent and youth assent for those under 18, or participant consent for those 18 or older (Appendix B). We expect that across the three divisions and their selected sites, approximately 90 youth will agree to participate in the in-person focus groups. We anticipate that the youth program participants will be middle and high school students between the ages of 12 and 19. Program participants involved in data collection will be from a convenience sample. Consequently, like the cognitive interview participants, they may not be representative of the population that all ACF youth programs serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

To develop Instrument 1, the Youth Screener for Cognitive Interviews, the study team considered what demographic variations would support the collection of data from a diverse pool of youth, identifying characteristics of interest as grade level, age, race and ethnicity, home language, gender, sexual orientation, location (US State), and urbanicity. Collecting youth’s grade level will allow us to organize groups by grade level to ensure that youth feel they are among peers in each interview grouping. Collecting youth’s age will allow us to implement appropriate consent and assent procedures, as minor youth will have a different consent process than youth who may be 18 years of age or older. Collecting race and ethnicity will allow us to ensure diversity in the sample, as youth from different racial/ethnic backgrounds may respond differently to the items presented; for example, potentially including Native American youth in the sample will allow us to hear culturally informed opinions on the potential inclusion of a “Two Spirit” response option. Collecting youth’s home language will allow us to identify youth to participate in Spanish-language testing of the measures. Collecting youth’s gender and sexual orientation will allow us to ensure diversity in the sample and organize interview groupings of LGBTQ youth. Lastly, collecting location and urbanicity will allow us to ensure diversity in the sample, as there may differences in youth’s familiarity with and understanding of SOGI terms by geographic region or urbanicity. We do not propose collecting information on household income as youth are not a reliable source of such information.

To prepare the list of SOGI measures that will be presented to youth for cognitive testing, the study team scanned available SOGI measures and identified a set of items aligned with ACF’s research priorities, the latest recommendations from OMB and NASEM (Appendix C), and current practices for collecting SOGI measures from youth in the United States. To develop Instrument 2, the Cognitive Interview and Focus Group Protocol, we considered the overall goals of the cognitive interviews and focus groups; namely, to develop an understanding of how well these measures represent the identities of youth aged 12-19, how clear and understandable the measures are to diverse populations of youth, and collect youth feedback on ways to improve the measures. Aligned with current best practices in cognitive interviewing, we developed high-level questions to ask each group of youth to consider thinking across all of the measures that were presented, as well as a set of questions specific to each measure presented. To support efforts to identify the most ideal measure for youth, we also selected a set of comparison questions intended to allow youth to review measures side by side and vote on their preferred options, with set discussion questions aimed to gather insight into the rationale for these preferences.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection and Mathematica will oversee all data collection and ensure quality control measures are in place and followed for each data collection activity. To ensure an efficient and standardized data collection process, Mathematica study staff will participate in a project-specific data collection training. The training will cover the expectations and process for working with the market research vendor and cover all aspects of the data collection. All staff will be trained on best practices for collecting high-quality data and procedures around data privacy and security.

Participant screening for virtual cognitive interviews. To recruit youth participants for the virtual cognitive interviews, Mathematica will work closely with the market research vendor, Canview. Canview will conduct outreach to their panel members, and administer via a web survey a short, 5-minute eligibility screener (Instrument 1). This screener is designed to allow the Mathematica study team to purposively select study participants based on key criteria (age, grade, gender identity, sexual orientation, and primary language), ensuring all participants are eligible for the study, and that variation exists on other demographics. The data collection process will be monitored to maintain high standards of participant selection and to ensure that the study participants accurately reflect the desired variation in the pretest population.

Cognitive Interviews and youth focus group discussions. The study team will collect data by moderating all small group cognitive interviews and focus group discussions, following a data collection protocol (Instrument 2). The cognitive interviews will be held on a virtual platform such as WebEx or Zoom, and all participants, including the experienced cognitive interviewers, will log into the platform. Cognitive interviews will be conducted with up to 5 participants per group. The youth focus group discussions will be conducted in-person, on-site for select programs. During the cognitive interviews and focus group discussions, the participants will begin by reviewing several SOGI items, followed by a guided discussion using the relevant protocol. Interviews and discussions with youth who speak a language other than English will be led by moderators who are fluent in the youth’s primary language. A notetaker will take live notes and the sessions will be audio recorded to ensure accurate collection of data. Following each discussion group, the moderator and notetaker will debrief on the session together. The notetaker will finalize the notes, which will be reviewed by the moderator to ensure accuracy and completeness.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

For the virtual cognitive interviews, respondents will be purposively selected from a pool of eligible respondents (eligibility determined by Instrument 1). During the in-person focus groups, we will track the total number of youth at each site who provided consent to participate in the pretest.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the participant pretest data collection.

B6. Production of Estimates and Projections

The goal of the proposed pretest is to collect information from youth to support ACF’s effort in developing SOGI measures to better represent program participants. The data collected will document (1) whether the questions are age appropriate, and (2) whether and how the questions can be interpreted by respondents, and 3) the relevance and inclusivity of the language used.

The data will not be used to generate population estimates, either for internal use or for dissemination. Policy decisions will not be made based on the data collected for this pre-test study, as the data are not representative.

B7. Data Handling and Analysis

Data Handling

For the virtual cognitive interviews, participants will be recruited from a market research vendor (Canview). The vendor maintains all personally identifiable information (PII) for their panel members. For the interviews, Canview will share the demographics collected from the Youth Screener for Cognitive Interviews (Instrument 1) needed for recruitment with the study team, but this will not include any other identifying information about participants. At the time of the cognitive interviews, Canview will provide a list of first names of participants, without any other identifying information.

For the in-person focus groups with youth, we will work in collaboration with select program offices within ACF to recruit youth who have participated in a PREP, SRAE, or other relevant ACF program. For that recruitment effort, ACF may need to provide first names and email addresses to the contractor to coordinate data collection. We expect the sites will manage the consent form process, and youth and parent names will not be linkable to the focus group data. Following the focus groups, all names and contact information will be destroyed. Data collected from the focus group discussions will not be linkable by the PII.

The cognitive interviews and youth focus group discussions will be audio-recorded for notetaking, with the consent of all group participants. We will wait to begin recording the discussion until after everyone has introduced themselves. The transcribed notes will not include any names.

All notes and recordings will be stored on Mathematica’s secure network. No one outside the study team will have access to the data. Only Mathematica staff working directly on this project have access to the project folder on the network where recordings will be saved. All recordings will be destroyed as soon as they have been transcribed and notes will be destroyed per contract requirements.

Data Analysis

This project will not employ complex data analytic techniques.

The study team will analyze the cognitive interview and focus group discussion data qualitatively, using feedback from the participants to assess and improve the clarity and comprehension of the SOGI questions and response options. This will include coding the interview and discussion notes to identify recurring themes or patterns across the items, as well as comments that may indicate potential issues with a particular item. The analysis will focus on understanding the context and reasons behind any challenges in comprehension or responding to the SOGI items and will determine what revisions can be made to improve the clarity and relevance of the items.

Data Use

ACF is currently engaged in efforts to improve the PREP and SRAE participant entry and exit survey questions, and to select questions on SOGI for use across ACF youth programs to better represent how youth participants self-identify, as well as to align with EO 14075. Findings from the pretests will inform the development of the final SOGI measures for the PREP and SRAE youth participant entry and exit surveys and to support the agency-wide initiative at ACF to select SOGI items for youth.

After conducting both the cognitive interviews and the focus group discussions, the study team will develop an internal ACF memorandum describing the pre-test methodology and results, including proposed changes to the measures, and justification for the proposed changes.

ACF will use the results internally to inform subsequent information collection requests. Results of these methodological studies may be made public through methodological appendices or footnotes, reports on data collection development, data collection user guides, descriptions of respondent behavior, and other publications or presentations describing findings of methodological interest. The results of these pre-testing activities may be prepared for presentation at professional meetings or publication in professional journals. When necessary, results will be labeled as exploratory in nature and any limitations will be described.

B8. Contact Persons

Table B.2 lists the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Staff responsible for study

Name	Affiliation	Email address
Caryn Blitz	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Caryn.Blitz@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Brian Goesling	Mathematica	bgoesling@mathematica-mpr.com
Lara Hulsey	Mathematica	lhulsey@mathematica-mpr.com
Melissa Thomas	Mathematica	mthomas@mathematica-mpr.com
Maya Reid	Mathematica	mreid@mathematica-mpr.com

Attachments

Appendix A: Outreach to participants (English and Spanish)

Appendix B: Consent and Assent forms (English and Spanish)

Appendix C: SOGI Questions for Pretest (English and Spanish)

Instrument 1: Youth Screener for Cognitive Interviews (English and Spanish)

Instrument 2: Cognitive Interview and Focus Group Discussion Protocol (English and Spanish)

1 Norris AE, Torres-Thomas S, Williams ET. Adapting cognitive interviewing for early adolescent Hispanic girls and sensitive topics. Hisp Health Care Int. 2014;12(3):111-9. doi: 10.1891/1540-4153.12.3.111. PMID: 25239207; PMCID: PMC4459492. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4459492/

8