SED Sexual Orientation and Gender Identity Question Experiment Plan

ATTACHMENT 10

Survey of Earned Doctorates Sexual Orientation and
Gender Identity Question Experiment Plan

1

2024 SED SOGI Data Collection Experiment Plan
In the 2024 SED, NCSES will continue its research on how to collect sexual orientation and gender
identity (SOGI) data from doctorate recipients that would yield most accurate and usable data while
attempting to address privacy and confidentiality concerns. The experimental design proposed below
uses compound random assignment to assess the efficacy of different questions for measuring biological
sex, gender identity, and sexual orientation.
The experiment plan includes the current SED binary sex question (SEDSEX), the 2023 NCSG biological
sex at birth question and a variation (S1 and S2), the 2023 NSCG gender identity question and a variation
(G1 and G3), and other variation of SOGI questions (SOGI1, G2, SO1, SO2, and SO3) based on the results
of the SED SOGI cognitive interviews and other current SOGI research. Proposed wording and response
options for each item are shown on page 3.

Experimental Design
Concept
Sex/Gender
Biological
sex

Question
location

Path 1 - 2

C14

C14_S
C14_G

S1: 2023
NSCG
(12.5%)

S2: 2023
NSCG REV
(12.5%)

SOGI1 (25%)
C14_SO

Sample distribution

Paths 12 – 20

SEDSEX

Gender
identity
Sexual
orientation

Paths 3 – 11

S1: 2023 NSCG
(37.5%)

S2: 2023 NSCG REV
(37.5%)

G1: 2023 NSCG (12.5%)

G1: 2023 NSCG (12.5%)

G2: Y/N OPTION (12.5%)

G2: Y/N OPTION (12.5%)

G3: GENDNSCG REV (12.5%)

G3: GENDNSCG REV (12.5%)

SO1: OMB REV (12.5%)
SO2: Y/N OPTION (12.5%)
SO3: SEXIDNSCG REV (12.5%)

SO1: OMB REV (12.5%)
SO2: Y/N OPTION (12.5%)
SO3: SEXIDNSCG REV (12.5%)

37.5%

37.5%

25%

Key aspects of the design and analysis plan
•

The current SED biological sex question (SEDSEX) will be asked near the end of the survey so it can
be asked prior to the the SOGI experimental questions (see Attachment 1). The questions will be
prefaced by a statement explaining that the SED is testing different approaches to improve the way
these questions are collecting. Placing the SEDSEX and the SOGI experimental questions near the
end will mitigate any concerns about data loss due to potential survey breakoff.

•

Respondents in all experimental paths will first be asked the current SED binary sex question
(SEDSEX) followed by the experimental questions.

2

•

Respondents in path 1 – 2, the two experimental questions will be displayed on a single screen. The
first question on this screen will be one of the experimental biological sex at birth questions (S1 or
S2). The second question will be combined sexual orientation and gender identify question that asks
if the respondent identifies as LGBTQ+ (SOGI1).

•

Respondents in paths 3 – 20 will see two separate screens with different experimental SOGI
questions. The first screen will ask one of the six combinations of the experimental biological sex at
birth (S1 and S2) and gender identity questions (G1 – G3). The second screen will ask one of the
three sexual orientation questions (SO1 – SO3).

•

Respondents will be randomly assigned to one of the 20 treatment conditions composed of the
combinations of biological sex at birth, gender identity, and sexual orientation questions to
maximize power for comparisons across question variations and minimize differences based on
combinations of items seen (i.e., context effects). The design will:
o require 11 distinct new screens be programmed (1 for the combination of S1 and SOGI1, 1 for
the combination of S2 and SOGI1, 6 for the combinations of S1 – S2 and G1 – G3, and 3 for SO1 –
SO3)
o yield 20 paths through the 11 screens (for example, path 1 is S1/SOGI1; path 2 is S2/SOGI1; path
3 is S1/G1 and SO1; path 4 is S1/G1 and SO2; path 5 is S1/G1 and SO3; path 6 is S1/G2 and SO1;
etc.)

•

After the screens displaying both a biological sex at birth question and gender identity question,
respondents will receive a prompt to confirm their responses to these questions when there
presents a concern of data quality, consistent with the current design of SED and the best practice
to collect SOGI information.

•

The design also will test survey functionality added to address concerns respondents may have
about data confidentialiy, privacy and relevance of the SOGI data. The SED cognitive tests results
suggest that many SOGI minorities would weigh the relevance of SOGI data and concerns about data
confidentiality and privacy when deciding whether to answer SOGI questions.
o SOGI screens will feature a link providing an explanation of the relevance of the SOGI and how
the data will be protected.
o Respondents who do not answer SOGI1 or the sexual orientation question (SOGI1 and SO1 –
SO3) or select “I prefer not to answer” will be directed to a follow up screen that gives them
another opportunity to answer the question and to check a checkbox that allows them to opt
out of sharing responses with their doctoral institution.

•

Finally, at the end of survey, all respondents will be asked to provide feedback about privacy
concerns related to the SOGI questions they were asked.

•

The following data will be examined to assess the relative efficacy of the SOGI questions:
o
o
o
o

Response distribution
Item nonresponse
Changes in response
Clicks on explanatory links

o
o
o
o

3

Use of opt out checkboxes
Timing data
Incidence of breakoff
Respondent feedback

Biological Sex at birth test questions
S1 – 2023 NSCG

S2 – 2023 NSCG REV

What sex were you assigned at birth, on your original
birth certificate?

What sex were you assigned at birth, on your original
birth certificate?

O Male
O Female

O Male
O Female
O Something else: _____

Gender identity and sexual orientation test questions
SOGI1

G1 – 2023 NSCG
How do you currently
describe yourself?
Select all that apply.
 Male
 Female
 Transgender
 I use a different term: ____

G2 – Y/N OPTION
Do you consider
yourself to be a gender
minority (e.g.,
transgender, nonbinary)?
O Yes
O No
O I prefer not to answer

G3 – GENDNSCG REV
How do you currently
describe yourself?
Select all that apply.
 Man
 Woman
 Transgender
 Gender non-conforming
 Non-binary
 Genderfluid
 Genderqueer
 I use a different term: ____
 I prefer not to answer

Click here to learn why these questions are asked and how data will be protected.

(To be shown at the bottom of the 6 screens that randomly combine S2 – S3 with G1 – G3. The
following text will appear upon clicking the link)

“The SED is asking these questions to be more inclusive and to study the differences in
educational or postgraduation outcomes among minority groups. Your response will be
kept confidential, used for experimental research only, and reported in a format that does
not lead to individual identification.”
SO1 – OMB SO REV
Do you consider
yourself to be
LGBTQ+?

Which of the following best
represents how you think of
yourself?

O Yes
O No
O I prefer not to
answer

O Straight or heterosexual
O Lesbian, gay, bisexual,
queer or another
orientation
O I prefer not to answer

SO2 – Y/N OPTION

SO3 – SEXIDNSCG REV

Do you consider yourself
to be a sexual minority
(e.g., gay, lesbian,
bisexual)?

Regardless of your sexual
experience, what is your
sexual identity or
orientation?

O Yes
O No
O I prefer not to answer

O Straight or heterosexual
O Lesbian, gay, bisexual,
queer or another
orientation
O I prefer not to answer

Click here to learn why this question is asked and how data will be protected. (To be shown at the bottom of the
SOGI1 screen and SO1 – SO3 screens. The following text will appear upon clicking the link)

“The SED is asking this question to be more inclusive and to study the differences in educational or
postgraduation outcomes among minority groups. Your response will be kept confidential, used for
experimental research only, and reported in a format that does not lead to individual identification.”

4

Rationale of the proposed SED SOGI experimental test questions
S1 – 2023 NSCG: This question is the same as the biological sex at birth question in the 2023 NSCG so
that data collected from SED and NSCG respondents can be compared. The question stem is the same as
the test question S2. By comparing the S1 and S2 data we will be able to assess the impact of the
addition of a “something else (specify)” response option.
S2 – 2023 NSCG REV: This question is same as the question S1 with a third option “something else
(specify)”. In the SED cognitive interviews, 57% (35 of 61) of participants thought that adding a third
response option to this question would make the question more inclusive. Question data can be
compared with S1 to measure the impact of the addition of a “something else (specify)” response
option.
SOGI1: This question is based on a question cognitively tested by Dr. Anthony Pho and discussed in his
2022 AAPOR presentation 1. This question is less burdensome than the separate gender identity and
sexual orientation questions while yielding an estimate that can be reported. This may be preferrable
unless distinction between sexual and gender minorities and distinguishing subgroups within these
minority populations are necessary and possible.
G1 – 2023 NSCG: This question is same as the gender identity question in the 2023 NSCG, and will be
used to compare the data collected from SED and NSCG respondents, and the data from the test
question G3. Comparing with G3 will allow us to detect differences due to the more detailed response
options including an “I prefer not to answer” option.
G2 – YES/NO OPTION: This question has simple Yes or No response options to compare the data
collected from the SED respondents with the data from the test questions G1 and G3. This question is
less burdensome than the other gender identity questions and and may yield an estimate that can be
reported.
G3 – GENDNSCG REV: This question is a revised version of the 2021 NSCG bridge panel gender identity
question tested in the 2022 SED cognitive interviews. The cognitive interviews found that many SOGI
minority participants (16 of 39, 44%) found it challenging to respond to gender identity questions that
only allowed a single response option because more than one option applied to them. Also, some SOGI
minority participants (11 of 39, 28%) stated that “male” and “female” connote biological sex so the
inclusion of these terms in questions about gender identity made the intent of the questions unclear to
them. Replacing these with “man” and “woman” would make the question clearer to these participants
because the response options align with the question wording. The majority of the SOGI minorities
preferred this question with more detailed response options. Although we may not be able to report the
detailed options due to data confidentiality and/or reliability concerns, comparing the collapsed gender
minority data in G3 with G1 will allow us to detect differences due to the additional response options
including an “I prefer not to answer” response option.
SO1 – OMB REV: This question is based on the sexual orientation question provided in the OMB SOGI
best practices document 2. In this question, two separate minority sexual orientation response options
are collapsed into a single option, including ‘queer or another orientation’, and uses an “I prefer not to
answer” response option added to address potential privacy concerns of SED respondents. Some
participants in the SED cognitive interviews (7 of 58, 12%) felt uncomfortable sharing their sexual
orientation because it was perceived to be highly personal information. This question can be compared
1

How do People Want to be Asked About Their Sexual Orientation and Gender Identity on National Health
Surveys? Presented at the American Association for Public Opinion Research Annual Meeting, May 18-21, 2022
2

https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf

5

with SO3 to evaluate the impact of the question stem change as both questions have same response
options.
SO2 – YES/NO OPTION: This question has simple Yes or No response options to compare the data
collected from the SED respondents with the data from the test questions SO1 and SO3. This question is
less burdensome than the other sexual orientation questions and and may yield an estimate that can be
reported.
SO3 – SEXIDNSCG REV: This question is a revised version of the 2021 NSCG bridge panel sexual
orientation question tested in the SED cognitive interviews. The question stem is the same as the
cognitive test question but with the response options collapsed to address SOGI minority respondents’
potential concerns about privacy. The data from this question can be compared with the test question
SO1 to measure the impact of the question stem change as both questions have same response options.

6