9-10-2020 Memo

Memorandum

Date: September 10, 2020

To: Margo Schwab, Desk Officer

Office of Management and Budget

From: Emilda B. Rivers, Director

National Center for Science and Engineering Statistics

National Science Foundation

Via: Suzanne Plimpton, Reports Clearance Officer

National Science Foundation

Subject: Request for Approval for Cognitive Interviews for the National Training, Education, and Workforce Survey

The National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation requests approval to conduct cognitive interviews under the generic clearance for survey improvement projects (OMB number 3145-0174). This memorandum describes the justification and plans for this cognitive testing of a new federal survey, the National Training, Education, and Workforce Survey (NTEWS). The NTEWS will be a biennial sample survey sponsored by NCSES and by the National Center for Education Statistics (NCES) within the Department of Education.

Background

NCSES has a legislative mandate to collect, analyze, interpret, and disseminate data on the U.S. science and engineering workforce, disaggregated by detailed demographic subgroups such as sex, race/ethnicity, and disability status. NCES has legislative mandate to collect, collate, analyze, and report complete statistics on the condition of American education. Under these mandates, the NTEWS serves to measure and understand two research concepts that are of national interest: 1) the effects of a diverse credentialing system (certifications, licenses, postsecondary certificates and work experience programs) on career opportunities and 2) the education, training, and career paths of skilled technical workers to meet America’s workforce demands to stay globally competitive in an increasingly scientific and technological era. The nature of work has changed dramatically over the decades with the pervasiveness of science and technology, making work-related credentials and skilled technical workers increasingly important to U.S. economic competitiveness, national security, and scientific progress. Currently, there is a lack of information about these workers, making it difficult for policymakers, employers, educational institutions, and researchers to understand or address issues affecting this component of the overall workforce.

The NTEWS is a new survey that plans to collect information previously collected in the Adult Education and Training Survey (ATES), an inactive, OMB-approved survey conducted by NCES. From March to August 2018, NCES conducted three rounds of cognitive testing on credentials questionnaire items previously collected in the ATES. This cognitive research (OMB #1850-0803 v.227 & v.228) was originally planned for the 2019 and 2021 administration of the ATES. When ATES was terminated prior to its 2019 administration, NCSES adopted much of the ATES content for NTEWS.

At the conclusion of the 2018 cognitive testing, unresolved issues for some questionnaire items required further refinement. In addition, the NTEWS plans to include some questionnaire items derived from NCSES’s National Survey of College Graduates (NSCG). Since the NTEWS is a general population survey (compared to NSCG being a college graduate population survey), these NSCG-based questionnaire items require testing to justify their inclusion in the NTEWS. NCSES requests approval to conduct up to 90 cognitive interviews (3 rounds, using both virtual and in-person interviews) to test the planned survey content prior to the beginning of the 2021 NTEWS data collection effort.

Purpose

The NTEWS is a voluntary, nationally representative survey targeting individuals ages 16-75 who are not enrolled in high school. This specific population allows for coverage of employment-eligible adults in the U.S. The NTEWS draws survey content from three surveys: 1) NSCG, 2) ATES, and 3) the Census Bureau’s American Community Survey (ACS). One of the benefits of drawing from these three surveys to create the NTEWS is that most of the survey content has undergone multiple rounds of cognitive testing. However, this memorandum presents the following shortcomings that require additional cognitive testing prior to the inclusion of this content on the NTEWS:

1. In January 2020, NCSES and NCES closely evaluated each question for quality and utility according to OMB’s federal survey guidance, and as a result, reduced the number of survey questions as well as modified the wording of some survey questions from the ATES version tested in 2018.

2. The questions from the NSCG have been tested and fielded on cases who have a bachelor’s or higher (i.e., college-educated population) and not on a general population.

3. The questions from the ATES have been tested and fielded on a general population in full production. However, during the 2018 cognitive testing, some questions were still identified as problematic and require additional testing.

4. The questions from ACS have been tested and fielded on a general population but were not tested in combination with the questions from NSCG and ATES during the 2018 cognitive testing. At that time, NCSES and NCES believed that demographic data from the ACS could be appended to the NTEWS datafile and would not need to be collected in the NTEWS. However, since that time, the Census Bureau’s ACS governing committee concluded that, due to privacy and confidentiality concerns, the ACS data could not be appended to the NTEWS datafile.

Methodology

The cognitive testing will address the concerns above. Cognitive testing techniques will assess whether: (1) question wording is understandable by participants; (2) participants are able to provide answers to the questions and are able to retrieve the information; and (3) the navigation through the survey is clear. The following sections describes NCSES’s methodology for NTEWS cognitive testing.

Target Population for Testing

NCSES identified the following target population for testing based on two characteristics:

1. Skilled Technical Workforce (STW) Status:

   1. Skilled Technical Workers – Defined as adults who have less than a bachelor’s degree.

   2. Non-Skilled Technical Workers – Defined as adults who have a bachelor’s degree or higher.

2. Credentials Status:

   1. Adults with vocational certificates

   2. Adults with licenses

   3. Adults with certifications

   4. Adults with previous participation in a work experience program

   5. Adults with a combination of credentials

These characteristics result in ten categories of individuals eligible to participate in the cognitive interviews. Although non-STW individuals are eligible to participate in testing, recruitment will prioritize individuals who are considered a part of the STW. Non-STW participants will comprise no more than one-third of the total interviews. In addition to these characteristics, NCSES will also attempt to balance participants across broad demographic characteristics that include age, race, and employment status.

NTEWS Questionnaire

Attachment A contains the full NTEWS questionnaire. A list of the sections, in the order of the questionnaire, is presented below. NCSES anticipates revising some of the content in response to findings from the testing.

• Employment Status

• Current Employment

• Licenses for Work

• Certifications for Work

• Work-Experience Programs

• Education Level

• Vocational Certificates

• Enrollment

• Background (demographic questions)

NCSES aims to keep the interview to no more than 75 minutes to keep participants engaged and prevent fatigue from the interview length. Given the number of questions that require probing, the first round of testing will not include the Education Level, Enrollment, or Background sections. As less probing becomes necessary, NCSES will include these sections and eventually test the full NTEWS questionnaire. For the first round of interviews, NCSES will use two different testing versions of the questionnaire (see Table 1) to probe and collect information on why individuals seek and obtain their credentials.

Table 1. Testing Questionnaires for Round 1

Questionnaire Versions
Vocational Certificates	Licenses and Certifications
Employment Status	Employment Status
Current Employment	Current Employment
Licenses for Work	Licenses for Work
Certifications for Work	Certifications for Work
Vocational Certificates	Work Experience Programs

Participants who report having a vocational certificate at the time of recruitment, regardless of any other credentials they may have, will receive the Vocational Certificates Questionnaire (Attachment B), which is guided by the Vocational Certificates Protocol (Attachment C). All other interview participants, who will only have licenses, certifications, or both, will receive the Licenses and Certifications Questionnaire (Attachment D), which is guided by the Licenses and Certifications Protocol (Attachment E). NCSES may introduce additional probes if testing uncovers unforeseen problems that need to be investigated further. The Recruitment section of this memo provides recruitment targets for testing both versions of the questionnaire.

Sample

To conduct the cognitive testing, NCSES will complete up to 90 interviews across three rounds of cognitive testing (no more than 30 interviews per round). Participants will be aged 18-75.

NCSES will use recruitment vendors that maintain both national and local panels for qualitative research projects. The vendors’ panels and recruitment databases are nonprobability based; members self-select into the panel/database and will volunteer for the study. All panels feature a diverse group of members in terms of race, gender, age, occupation, and urbanicity.

NCSES expects that the recruitment vendors will be sufficient for recruitment. However, in cases where alternative recruitment strategies are needed (for example, if a partner has exhausted the panel in a particular geographic area), the NCSES will directly recruit additional participants¹ using networks of personal and professional contacts and community organizations in the greater Washington, DC area, Chicago, New Jersey, Pennsylvania, and Delaware.

Recruitment

NCSES’s recruitment vendors use a variety of methods to contact potential participants, including posting research studies on their websites, as well as emailing and calling panel members. For this testing, vendors will make initial contact with panel members using two different methods: 1) “blast” calls or emails to all members of in the vendors’ panel databases and 2) targeted outreach, using demographic information on panel members, to email or call those panel members who closely match the study criteria (see Attachment F for recruitment materials, including contact scripts and FAQs). In this initial contact, panel members will be invited to access a web link to answer online, eligibility pre-screening questions. Responses to these questions will allow the vendors to identify potentially eligible panel members with whom to follow up. The vendors will then contact these potentially eligible panel members and administer a detailed screener to confirm eligibility for the study (see Attachment G for the pre-screening questions and the detailed screener questionnaire). Although the detailed screener questions are the largely the same for virtual and in-person interviews, additional eligibility questions particular to mode of the interview are necessary. The virtual interview screener will include questions about having Internet access, a webcam-enabled computer, and the ability to download and use a videoconferencing application. The screener for the in-person interview will ask about the ability to travel to the interview location.

After confirming a potential participant’s eligibility, the recruitment vendor will schedule the participant for an interview. To minimize cancellations and no-shows, the vendor will contact scheduled participants via telephone or email before the scheduled interview to remind them of the upcoming appointment (see Attachment F for the reminder contact scripts). Table 2 shows the targeted number of completed interviews based on participants’ characteristics for Round 1.

Table 2. Completed Interview Targets for Round 1

	STW Participants (N > 20 completed interviews)	Non-STW Participants (N < 10 completed interviews)
Participants with vocational certificates	N = ~ 9	N = ~3
All others	N = ~13	N = ~5

To ensure maximum flexibility in scheduling, NCSES will recruit participants and schedule virtual interviews on a rolling basis. If NCSES determines that Rounds 2 and 3 will include in-person interviews, all recruitment for in-person interviews in a geographic area will take place prior to conducting interviews in that location. Regardless of interview mode, NCSES expects that recruiting participants and conducting the cognitive interviews for each round will take about four weeks.

Interview Mode

Due to social-distancing requirements in response to the COVID-19 pandemic, NCSES will conduct all Round 1 interviews virtually, using videoconferencing applications such as WebEx or Zoom. If social distancing measures are still in place after the completion of Round 1, subsequent rounds will continue using videoconferencing applications. If social distancing measures have been lifted after the completion of Round 1, subsequent rounds will be conducted using either face-to-face interviews or virtual interviews.

In order to ensure that virtual interviews can gather the most information possible from survey response behavior and non-verbal cues, participants, as determined through the screener, will be required to have: 1) Internet access, 2) a webcam, and 3) the ability to download and use a videoconferencing application. For Round 1, participants will only be eligible to participate using a computer, not a smartphone or tablet. After Round 1, NCSES will evaluate if this requirement can be relaxed and still obtain the richness of data desired.

Questionnaires for Virtual and Face-to-face Interviews

Across all three rounds of testing, NCSES will be testing the paper-and-pencil (PAP) versions of the questionnaire. For virtual interviews, the PAP questionnaire will be a fillable, electronic version of the printed paper questionnaire (such as a PDF) that participants access and complete during the interview. For face-to-face interviews, participants will fill out a printed paper questionnaire.

Conducting Interviews

All interviews will be recorded. Regardless of mode, each testing session will begin by obtaining participants’ written consent, and in the case of virtual interviews, both written consent and verbal assent to record the interview (Attachment H).

Interviewers will conduct the cognitive interview using established cognitive interview practices. Probing will be retrospective at the end of each section of the questionnaire being tested unless the participant has major issues and needs help to continue to the next question. In these situations, the interviewer may use concurrent probing. Interviews will not employ the think-aloud method. Participants will be permitted to fill out the questionnaire without voicing their response process.

Alternative Question Testing

In addition to testing the content of the NTEWS questionnaire, NCSES seeks to understand if there are any differences in response quality for questions that are similar to those on other NCSES surveys. For in-person interviews in Rounds 2 and 3, if time permits, the end of session will include the testing of these alternative versions of questions found on other NCSES surveys. Attachments I and J contain the alternative versions of the questions and the protocols, respectively.

Also, through the information gained from the Round 1 interviews, NCSES will determine if any employment-related questions should be modified to assess the impact of the coronavirus pandemic. Any question modifications will be tested in Round 2 or 3, time permitting.

Payments to Participants

As a thank you to participants for their time, NCSES will offer participants an incentive of $50 in the form of cash, check, or gift card, depending on participant preference. The amount is based on the amount typically used for 1-hour in-person cognitive interviews conducted by Federal agencies (the proposed interview will likely take 75 minutes). Although the current interviews will be conducted on-line, and thus do not include travel costs which are a key driver of the typical incentive amount ($40 for 1 hour interview), this higher incentive is requested as a transition amount as NCSES works with Federal Statistical System colleagues to identify more effective, efficient, and cost-effective measures to conduct cognitive interviewing in a manner consistent with the social distancing requirements necessitated by the current pandemic.

The incentive payments will be made after the interview is completed. For virtual interviews, participants will be mailed the incentive within one week after the completion of the interview. For in-person interviews, the participants will receive the incentive immediately at the end of the interview.

Response Burden

Approximately 178 burden hours are estimated for this project. Table 3 summarizes the expected burden for recruiting participants and conducting the interviews.

Table 3: Burden Calculations for Cognitive Testing

Task	Time (minutes)	N		Hours
Initial eligibility pre-screening (reading message, connecting to web screener, and responding)	5	360		30.0
Confirming eligibility through detailed screener	10	120		20.0
Testing session reminder (call or email)	5	90		7.5
Completing testing session (downloading app, providing consent, accessing the questionnaire, and completing interview)	80	90		120.0
Total Burden Hours			177.5

Schedule

NCSES aims to complete the cognitive testing in approximately three months. By January 2021, NCSES expects to finalize the content for the 2021 NTEWS instrument. Table 4 shows a tentative schedule for testing, which is subject to change to accommodate any unforeseen circumstances (e.g., national policy changes due to COVID-19).

Table 4. Estimated Testing Schedule with Milestones

Milestone	Duration	Target Dates
NCSES receives OMB approval for NTEWS cognitive testing	3 weeks	September 11, 2020
Round 1 – recruit participants and conduct cognitive interviews	4 weeks	September 14 – October 9, 2020
NCSES finalizes changes to the questionnaire and prepares for next round	1 week	October 12 – October 16, 2020
Round 2 – recruit participants and conduct cognitive interviews	4 weeks	October 19 – November 13, 2020
NCSES finalizes changes to the questionnaire and prepares for next round	1 week	November 16 – November 23, 2020
Round 3 – recruit participants and conduct cognitive interviews	4 weeks	November 26 – December 18, 2020
NCSES decides on final changes to the questionnaire and finalizes the 2021 NTEWS content	6 weeks	December 21 – January 22, 2021

Contact Person

John Finamore

Program Director, Human Resources Statistics Program

National Center for Science and Engineering Statistics

National Science Foundation

jfinamor@nsf.gov

(703) 292-2258

List of Attachments

Attachment A. 2021 NTEWS Instrument

Attachment B. NTEWS Instrument_ Vocational Certificates

Attachment C. NTEWS Interview Protocol_ Vocational Certificates

Attachment D. NTEWS Instrument_ Licenses and Certifications

Attachment E. NTEWS Interview Protocol_ Licenses and Certifications

Attachment F. NTEWS Recruiting Materials

Attachment G. NTEWS Eligibility Screening: Pre-screening Questions and Detailed Screener

Attachment H. NTEWS Consent Form

Attachment I. NTEWS Alternative Question Testing_Questions

Attachment J. NTEWS Alternative Question Testing_Protocol

1 As part of the questionnaire development process, NCSES sought feedback on the NTEWS survey content through multiple presentations to the federal statistical and academic researcher communities. Outreach included presentations to the Council of Professional Associations on Federal Statistics, the Census Bureau Household Surveys Sponsors, and the George Washington University’s Non-Degree Credential Research Network (NCRN). In addition, NCSES held detailed conversations with a subset of the NCRN academic researchers to discuss specific question wording issues that helped in the formation of the final questionnaire for use in this cognitive testing. For future cognitive testing efforts associated with subsequent NTEWS survey cycles, NCSES will attempt to use these professional and personal networks to reduce future dependency on recruitment vendors.

9

Cover Memo _NTEWS Cognitive Interview