Survey of Earned Doctorates Sexual Orientation and Gender Identity Question Experiment Plan

ATTACHMENT 10

Survey of Earned Doctorates Sexual Orientation and
Gender Identity Question Experiment Plan
Revised 5/10/2023

1

2024 SED SOGI Data Collection Experiment Plan
In the 2024 SED, NCSES will continue its research on how to collect sexual orientation and gender
identity (SOGI) data from doctorate recipients that would yield most accurate and usable data while
attempting to address privacy and confidentiality concerns. The experimental design proposed below
uses compound random assignment to assess the efficacy of different questions for measuring gender
identity and sexual orientation.
The experiment plan includes the 2023 NCSG biological sex assigned at birth question, the 2023 National
Survey of College Graduates (NSCG) gender identity question and a variation (G1 and G3), and other
variations of SOGI questions (SOGI1, G2, SO1, SO2, and SO3) based on the results of the SED SOGI
cognitive interviews and other current SOGI research. Proposed wording and response options for each
item are shown on page 4.

Experimental Design
Concept

Question location

Sex Assigned At Birth

C14_S

Gender identity

C14_G

Path 1

Paths 2 – 10
SAB
G1 – 2023 NSCG (25%)
G2 – Y/N OPTION (25%)

SOGI1 (25%)

Sexual orientation

G3 – GENDNSCG REV (25%)
SO1 – OMB REV1 (25%)
SO2 – Y/N OPTION (25%)
SO3 – OMB REV2 (25%)

C14_SO

Sample distribution

25%

75%

Key aspects of the design and analysis plan
•

The SED SOGI experimental questions will be asked near the end of the survey prefaced by a
statement explaining that the SED is testing different approaches to improve the way these
questions are collected, and let respondents know that some of the questions may seem redundant.
Placing the SOGI experimental questions near the end will mitigate any concerns about data loss due
to potential survey breakoff.

•

Respondents in all experimental paths will first be asked the biological sex assigned at birth question
followed by the SOGI experimental questions.

•

Respondents in path 1, the two experimental questions will be displayed on a single screen. The first
question on this screen will be the biological sex assigned at birth question. The second question will
be combined sexual orientation and gender identity question that asks if the respondent identifies as
LGBTQ+ (SOGI1).

•

Respondents in paths 2 – 10 will see two separate screens with different experimental SOGI
questions. The first screen will ask the biological sex assigned at birth (SAB) and one of the three

2

experimental gender identity questions (G1 – G3). The second screen will ask one of the three sexual
orientation questions (SO1 – SO3).
•

Respondents will be randomly assigned to one of the 10 treatment conditions composed of the
combinations of biological sex assigned at birth, gender identity, and sexual orientation questions to
maximize power for comparisons across question variations and minimize differences based on
combinations of items seen (i.e., context effects). The design will:
o require 7 distinct new screens be programmed (1 for the combination of SAB and SOGI1, 3 for
the combination of SAB and G1 – G3, and 3 for SO1 – SO3)
o yield 10 paths through the 7 screens (for example, path 1 is SAB/SOGI1; path 2 is SAB/G1 and
SO1; path 3 is SAB/G1 and SO2; path 4 is SAB/G1 and SO3; path 5 is SAB/G2 and SO1; etc.)

•

After the screens displaying both a biological sex assigned at birth question and gender identity
question, respondents will receive a prompt to confirm their responses to these questions when
there presents a concern of data quality, consistent with the current design of SED and the best
practice to collect SOGI information.

•

The design also will test survey functionality added to address concerns respondents may have
about data confidentiality, privacy, and relevance of the SOGI data. The SED cognitive tests results
suggest that many SOGI minorities would weigh the relevance of SOGI data and concerns about data
confidentiality and privacy when deciding whether to answer SOGI questions.
o SOGI screens will feature a link providing an explanation of the relevance of the SOGI and how
the data will be protected.
o Respondents who do not answer SOGI1 or the sexual orientation question (SOGI1 and SO1 –
SO3) or select “I prefer not to answer” will be directed to a follow up screen that gives them
another opportunity to answer the question and to check a checkbox that allows them to opt
out of sharing responses with their doctoral institution.

•

Finally, all respondents will be asked to provide brief feedback on the SOGI questions they were
asked using Likert scales and/or comment boxes.

•

The following data will be examined to assess the relative efficacy of the SOGI questions:
o
o
o
o

Response distribution
Item nonresponse
Changes in response
Clicks on explanatory links

o
o
o
o

3

Use of opt out checkboxes
Timing data
Incidence of breakoff
Respondent feedback

Biological Sex Assigned At Birth Test Question (SAB)
What sex were you assigned at birth, on your original birth certificate?
O Male
O Female

Gender identity and sexual orientation test questions
SOGI1

G1 – 2023 NSCG
How do you currently
describe yourself?
Select all that apply.
 Male
 Female
 Transgender
 I use a different term: ____

G2 – Y/N OPTION

G3 – GENDNSCG REV

Do you consider
yourself to be a gender
minority (e.g.,
transgender, nonbinary)?

How do you currently describe
yourself?

O Yes

Select all that apply.
 Man
 Woman
 Transgender

O No
O I prefer not to answer

 Non-binary, Gender nonconforming, Genderfluid,
Genderqueer
 I use a different term: ____
 I prefer not to answer

Click here to learn why these questions are asked and how data will be protected.
(At the bottom of the 3 screens that combine SAB with G1 – G3, the following text will
appear upon clicking the link)
“The SED is asking these questions to be more inclusive and to study the differences in
educational or postgraduation outcomes among minority groups. Your response will be kept
confidential, used for experimental research only, and reported in a format that does not
lead to individual identification.”
SO1 – OMB SO REV1
Do you consider
yourself to be
LGBTQ+?

Which of the following best
represents how you think of
yourself?

O Yes
O No
O I prefer not to
answer

O Straight or heterosexual
O Gay, lesbian, bisexual,
queer or another
orientation
O I prefer not to answer

SO2 – Y/N OPTION
Do you consider
yourself to be a sexual
minority (e.g., gay,
lesbian, bisexual)?

SO3 – OMB SO REV2
Which of the following best
represents how you think of
yourself?
Select all that apply

O Yes

 Straight or heterosexual

O No

 Gay or lesbian
 Bisexual
 I use a different term:____
 I prefer not to answer

O I prefer not to answer

Click here to learn why this question is asked and how data will be protected.
(At the bottom of the SOGI1 screen and SO1 – SO3 screens, the following text will appear upon clicking the link)
“The SED is asking this question to be more inclusive and to study the differences in educational or
postgraduation outcomes among minority groups. Your response will be kept confidential, used for
experimental research only, and reported in a format that does not lead to individual identification.”

4

Rationale of the proposed SED SOGI experimental test questions
SAB: This question is the same as the biological sex assigned at birth question in the 2023 NSCG so that data
collected from SED and NSCG respondents can be compared.
SOGI1: This question is based on a question cognitively tested by Dr. Anthony Pho and discussed in his 2022
AAPOR presentation 1. This question is less burdensome than the separate gender identity and sexual orientation
questions while yielding an estimate that can be reported. This may be preferrable unless distinction between
sexual and gender minorities and distinguishing subgroups within these minority populations are necessary and
possible.
G1 – 2023 NSCG: This question is same as the gender identity question in the 2023 NSCG and will be used to
compare the data collected from SED and NSCG respondents, and the data from the test question G3. Comparing
with G3 will allow us to detect differences due to the more detailed response options including an “I prefer not to
answer” option.
G2 – YES/NO OPTION: This question has simple Yes or No response options to compare the data collected from
the SED respondents with the data from the test questions G1 and G3. This question is less burdensome than the
other gender identity questions and may yield an estimate that can be reported.
G3 – GENDNSCG REV: This question is a revised version of the 2021 NSCG bridge panel gender identity question
tested in the 2022 SED cognitive interviews. The cognitive interviews found that many SOGI minority participants
(16 of 39, 44%) found it challenging to respond to gender identity questions that only allowed a single response
option because more than one option applied to them. Also, some SOGI minority participants (11 of 39, 28%)
stated that “male” and “female” connote biological sex so the inclusion of these terms in questions about gender
identity made the intent of the questions unclear to them. Replacing these with “man” and “woman” would make
the question clearer to these participants because the response options align with the question wording. The
majority of the SOGI minorities preferred this question with more detailed response options. Although we may
not be able to report the detailed options due to data confidentiality and/or reliability concerns, comparing the
collapsed gender minority data in G3 with G1 will allow us to (1) use the accurate definition of “transgender”
consistently, (2) maintain the comparability for the analysis regarding the “transgender” option across conditions
and (3) detect differences due to the additional response options including an “I prefer not to answer” response
option.
SO1 – OMB REV1: This question is based on the presentation of the sexual orientation question provided in the
OMB SOGI best practices document2. In this question, multiple sexual orientation minority response options,
including ‘queer or another orientation’, are collapsed into a single option. In addition, an “I prefer not to answer”
response option is included to address potential privacy concerns of SED respondents. Some participants in the
SED cognitive interviews (7 of 58, 12%) felt uncomfortable sharing their sexual orientation because it was
perceived to be highly personal information. This question can be compared with SO3 to evaluate the impact of
the response options change as both questions have same question stem.
SO2 – YES/NO OPTION: This question has simple Yes or No response options to compare the data collected from
the SED respondents with the data from the test questions SO1 and SO3. This question is less burdensome than
the other sexual orientation questions and may yield an estimate that can be reported.
SO3 – OMB REV2: This question takes elements from the sexual orientation question provided in the OMB SOGI
best practices document3 and the 2021 NSCG bridge panel sexual orientation question tested in the SED cognitive
interviews. The SO3 question stem is the same as SO1 to test the differences in the response options. The
1

How do People Want to be Asked About Their Sexual Orientation and Gender Identity on National Health Surveys?
Presented at the American Association for Public Opinion Research Annual Meeting, May 18-21, 2022
2
3

https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf
https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf

5

response options in SO3 are disaggregated as they are in the OMB question but modified based on SED cognitive
testing of SOGI questions.

6