Supporting Statement for NCSES - Methodological Study to Test Education Data Collection Methods for the 2021 Survey of Earned Do

Memorandum

Date: October 30, 2019

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for approval of a methodological study to test education data collection methods for the 2021 Survey of Earned Doctorates (SED)

The National Center for Science and Engineering Statistics (NCSES) requests approval to conduct a methodological study for the 2021 Survey of Earned Doctorates (SED) under the generic clearance for improving survey projects (OMB control number 3145-0174). NCSES is considering revising the SED education section to reduce response burden and improve data quality by improving the way degree history and field of study codes are collected in the SED web instrument. This study will employ both online pretesting and traditional cognitive testing to obtain information on the optimal order of the degree history questions and the best method to collect and code field of study.

Background

The SED is an annual census survey of new recipients of research doctorates from U.S. institutions, sponsored by the National Science Foundation, the National Institutes of Health, the Department of Education, and the National Endowment for the Humanities. Research doctoral degrees are oriented toward preparing students to make original intellectual contributions in a field of study.

The SED collects educational history, education-related debt, financial support during graduate studies, information about postgraduation plans, and personal demographic information for each research doctorate recipient, primarily using the web survey mode. The two-phase study outlined in this request will test possible changes to the way educational data are collected from respondents using the SED web instrument.

Changes Being Tested

Two changes being considered are as follows:

1. Revision of field of study (FOS) question wording and how the FOS code is collected with the goals of improving the accuracy of the FOS data and making it easier to identify newly emerging fields.

   • Condition A (Similar to the FOS coding method currently used in the SED): As respondents start typing their field of study verbatim text in the search textbox, the instrument pulls up the field names that include the characters typed until a field is selected by the respondents. If no field name matches what the respondents type, respondents are asked three follow-up questions to find the most appropriate FOS code.

   • Condition B (Similar to the method used in the National Survey of College Graduates): After the respondents type the field of study verbatim text and click “Enter,” the instrument searches for the verbatim text that matches a specific FOS code. If no verbatim match is found, the respondents are asked three follow-up questions to find the most appropriate FOS code.

2. Revision of the degree history question order, with the goal of improving data quality by finding the best way to collect additional postsecondary degrees (in addition to the doctorate just received), while minimizing respondent burden.

   • Condition C (Same as current SED): Respondents are asked to report their doctoral degree, and other postsecondary degrees they may have earned in the following order: associate’s degree, bachelor’s degree, master’s degree, and professional doctorate. For each degree type they earned, the respondents provide details about the most recent degree and the first degree, if they have more than one.

   • Condition D: Similar to Condition C, but respondents are asked to report other postsecondary degrees in reverse order: professional doctorate, master’s degree, bachelor’s degree, and associate’s degree.

   • Condition E (Similar to the Early Career Doctorates Survey): Respondents are asked to list all postsecondary degrees they have earned. Follow-up questions about each degree are based on the order of degrees listed. If the respondents did not list a bachelor’s or master’s degree, they will be asked if they earned one.

The methodological study described in this memorandum will be conducted to test data collection methods that would result in improvements to the SED questionnaire. The proposed study will enable NCSES to:

• understand respondents’ thought processes when answering education related questions,

• understand respondents’ comprehension of field of study terms used in the questions,

• evaluate the memory demands of the degree history questions,

• evaluate respondents’ ability to make decisions and judgments in answering questions,

• determine appropriate presentations of response categories,

• assess the navigational problems respondents face within the web instrument, and

• identify sources of response burden and respondent stress.

Test Design

In an effort to control the costs associated with questionnaire pretesting and cognitive testing, the proposed pretesting of questions will be conducted in two phases. The Phase 1 test will be based on an online convenience sample and the Phase 2 test will be cognitive interviewing with an interviewer either onsite or via video conference. Both test phases will include qualitative and quantitative techniques using a web instrument.

NCSES has recently conducted an exploratory study to evaluate the effectiveness and utility of using Amazon’s Mechanical Turk (MTurk), an online survey platform, specifically with an eye toward future use for pretesting questionnaires. Based on NCSES’ assessments, MTurk is determined to be one of the most promising crowdsourcing platforms for use in this testing effort. The MTurk platform has a larger available sample than other crowdsourcing platforms. It also has the most extensive features, allowing requesters a great deal of control over who may participate in their surveys.

The MTurk platform also allows NCSES to target college-educated respondents. Because the SED web instrument collects a detailed education history, online panel members with multiple postsecondary degrees will provide the most relevant and useful data for this study. A 2016 data collection by the Pew Research Center found that 87% of MTurk workers located in the United States had an educational attainment of some college or more.¹ A 2010 study by researchers at the University of California, Irvine found 65% of all MTurk workers, regardless of location, had an educational attainment of some college or more.² The MTurk platform should allow NCSES to recruit a large number of cases with the appropriate background for testing the SED instrument (see below for details on how NCSES will recruit MTurk workers).

Web Instrument

The test web instruments for the different conditions are shown in Attachments 1-6. Conditions A and C are very close to the current version of the SED instrument. To evaluate alternative methods of collecting data on field of study and degree history, we will randomly assign and administer slightly revised versions (Conditions B, D, and E) of the questions to the participants. Each respondent will be randomly assigned to receive either Condition A or B. Each respondent will also be randomly assigned to receive either Condition C, D, or E.

Phase 1 Online Test

We will recruit MTurk sample members with a graduate degree using the recruitment materials shown in Attachment 7. These advertisements will be posted on MTurk forums to alert MTurk members to the opportunity (called Human Intelligence Task (HIT) on the forums).

Those who participate in the Phase 1 Test will complete the education section of the SED web survey. The web survey will be hosted on the survey contractor’s (RTI) secure servers.

Interspersed in the web instrument will be probe questions asking participants how they arrived at their responses. Attachment 1 shows an overview of the control and test versions of the FOS collection and coding methods. Attachment 2 shows an overview of the degree history data collection method tests. Attachments 3, 4, 5, and 6 show the control and test versions of the degree history data collection methods.

Phase 2 Test With an Interviewer

The Phase 2 Test will be cognitive interviews conducted with an interviewer while respondents complete the web survey in person or via video conference.

We will recruit current doctoral students or doctorate holders at large research universities near RTI offices, such as the University of Chicago, the University of Maryland, the University of Illinois at Chicago, Georgetown University, the University of North Carolina, North Carolina State University, and Duke University.

We will post advertisements (Attachment 7) on student listservs. Interested individuals will first take a short 1-minute screener to determine whether they are eligible (Attachment 8), and those determined to be eligible will be contacted to participate in the study.

Screened cases will be grouped into three tiers:

• Tier 1 participants are doctorate holders and doctoral students whose degree or program is in interdisciplinary or complex research fields.

• Tier 2 participants are doctorate holders and doctoral students who hold or are pursuing joint degrees (e.g., MD/PHD).

• Tier 3 participants are doctorate holders and doctoral students whose degree or program is in typical doctorate fields.

If we have more than enough eligible participants who are current doctoral students or doctorate holders, we will give preference to those with complex doctoral fields of study.

Outcome Measures

FOS Coding Methods in Conditions A and B

Evaluation of the best FOS coding method will include the following measures:

• Compare the proportion of uncoded FOS verbatim responses (FOS name was not selected in Condition A or FOS was not auto-coded in Condition B)

  • Compare the proportion of remaining uncoded FOS cases after follow-up questions, where a lower proportion of uncoded FOS cases is better

• Compare participants’ responses to probe questions

  • Average reported difficulty in answering (Attachment 3 probe questions 4 & 11; 5-point scale), where less difficulty is better

  • Open-ended responses

    1. Source(s) of difficulty in finding FOS code (probe questions 5 & 12)

    2. Suggestions on question wording changes (probe questions 6 & 13)

    3. Reason for providing specific FOS verbatim response (probe questions 7 & 15)

    4. Other comments (probe questions 8, 9, 10, & 16)

    5. Factors considered in answer (probe question 14)

• Compare interviewer observations (Phase 2 test only)

  • Observed difficulty in participants answering the FOS questions (probe questions 1 & 2), where less observed difficulty is better

  • Observed attention in reading questions (probe question 3)

  • Description of FOS code selection process (probe questions 3 & 10)

Degree History Reporting Methods in Conditions C, D, and E

Evaluation of the best degree history reporting method will include the following measures:

• Compare item nonresponse rates, where lower rate is better

• Number of degrees reported, where more reported degrees is better

• Compare responses to probes

  • Average reported difficulty in answering (Attachments 4-6 probe question 1; 5-point scale), where less difficulty is better

  • Open-ended responses

    1. Source(s) of difficulty (probe question 2)

    2. Suggestions on question wording changes (probe question 3)

    3. Other issues (probe question 4)

    4. Response process (probe questions 5 & 6)

    5. Record uses in reporting degree history (probe questions 7 & 8)

    6. Overall comments (probe questions 9, 10, & 11)

Test Participants

To explore differences between the two designs of the FOS coding question and the three designs of the education history questions, we will recruit up to 3,000 participants from MTurk for the Phase 1 Test and screen up to 250 doctoral students or doctorate holders to recruit 60 participants for the Phase 2 Test. The data will provide sufficient quantitative and qualitative insights to inform the final design for the 2021 SED.

Burden Hours

The total burden hours for this study are estimated to be 815 hours:

• Phase 1 Test: 15 minutes per participant on average

• Phase 2 Test Screener: 1 minute per participant on average

• Phase 2 Test: 60 minutes per participant on average

Test Phase	Estimated Burden	Number of Participants	Total Burden
Phase 1	.25 hour	3,000	750 hours
Phase 2 Screener	.02 hour	250	5 hours
Phase 2	1 hour	60	60 hours
Total		3,310	815 hours

Payment to Participants

Phase 1 Test participants who complete the survey will receive $5.00. This amount is slightly higher than a typical MTurk panel payment but necessary to maximize the number of advanced degree holders to participate in the study. Phase 2 Test participants will receive $40 for completing the cognitive interview. Phase 2 Test participants who complete the cognitive interviews in person will receive $40 in cash immediately at the end of the interview. For cognitive interviews completed via videoconference, a $40 check will be mailed to the participants after the interview. These amounts are industry standards and have been used by NCSES in the past.

Test Phase	Payment per Participant	Number of Participants	Total Cost
Phase 1	$5.00	3,000	$15,000
Phase 2	$40.00	60	$2,400
Total			$17,400

Informed Consent

At the beginning of the survey, participants will be informed of the OMB control number, the expected survey completion time, and the voluntary nature of the study. In addition, participants will be informed that the data they provide in this study will reside on a server outside of the NCSES domain and that NCSES cannot guarantee the protection of survey responses (see ‘Introduction’ in Attachment 3).

Testing Schedule

The tentative schedule for cognitive testing is planned as below.

Date	Activity/Deliverable
10/31/2019	OMB submission for approval
11/15/2019	OMB clearance
11/28/2019	Launch Phase 1
01/31/2020	Complete Phase 1 evaluation
11/30/2019	Launch Phase 2
01/31/2020	Complete Phase 2 evaluation
02/28/2020	Final report

Contact Person

Kelly Kang

Project Officer

Survey of Earned Doctorates

Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

kkang@nsf.gov

703-292-7796

Attachments

Attachment 1: SED FOS Coding Method Test (Conditions A and B) Overview

Attachment 2: SED Degree History Data Collection Method Test Overview

Attachment 3: SED Degree History Data Collection Method Doctorate and Dissertation Research Questions

Attachment 4: SED Degree History Data Collection Method Test Condition C

Attachment 5: SED Degree History Data Collection Method Test Condition D

Attachment 6: SED Degree History Data Collection Method Test Condition E

Attachment 7: Test Participant Recruitment Materials

Attachment 8: Test Participant Eligibility Screening

1 Hitlin, P. (2016). “Research in the Crowdsourcing Age, A Case Study.” Pew Research Center. https://www.pewinternet.org/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/

2 Ross, J., Irani, L., Silberman, M. Six, Zaldivar, A., and Tomlinson, B. (2010). "Who are the Crowdworkers?: Shifting Demographics in Mechanical Turk." Proceedings of the Association for Computing Machinery (ACM) Conference on Human Factors in Computing Systems (CHI 2010).

7