2020/22 Beginning Postsecondary Students Longitudinal Study (BPS:20/22) Cognitive and Usability Testing Update

2020/22 Beginning Postsecondary Students Longitudinal Study (BPS:20/22)

Cognitive and Usability Testing

Attachments I - IV

OMB # 1850-0803 v. 267

Submitted by

National Center for Education Statistics

U.S. Department of Education

April 2020

Attachments

Attachment I – Recruitment Procedures and Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Cognitive Interview

Attachment IV – Interview Protocol

Contents

Attachment I. Recruitment Procedures and Materials 3

Identification of Respondents 3

Sample Recruitment Advertisements 3

Initial Contact Recruitment E-mail/Letter to Individuals 5

Recruitment E-mail/Letter to Organizations 6

Recruitment Phone Script for Potential Participants/Respondents 7

General Social Media Post 8

Specific Social Media Post 9

General Flyer 10

Specific Flyer 12

Frequently Asked Questions (FAQs) 14

Confirmation E-mail for In-person Participants 16

Confirmation E-mail for Remote Participants 17

In-person Participant Confirmation Phone Script 18

Remote Participant Confirmation Phone Script 19

Thank You E-mail/Letter to Participants/Respondents 20

Attachment II. Eligibility Screening Questions 21

E-mail with Self-Screener to Interested Participants who Call EurekaFacts 22

Eligibility Screener (Self-Administered and Over the Telephone) 23

Attachment III. Consent to Participate in Cognitive Interview 29

Consent Form for Participants 29

Attachment IV. Interview Protocol 31

Interview Protocol Guide 31

Attachment I. Recruitment Procedures and Materials

Identification of Respondents

EurekaFacts will locate potential participants through multiple outreach/contact methods and resources, such as cold calling from directory listings, advertisements within organizations and education institutions, referrals and snowball sampling, and social media posts. Based on experience, EurekaFacts recommends recruitment efforts to be conducted by directly targeting and contacting college students. Our approach to recruitment will utilize four primary vehicles:

• Usage/purchase of student e-mails available from a highly reputable commercial vendor that provides similar direct marketing outreach services for major national corporation and consumer brands.

• Placement of advertisements in highly selective/targeted venues to include Facebook and Google ads where they are restricted to users in university environments or with .edu websites.

• Use of classified advertisements or more general announcements to appear in online and print versions of campus newspapers as well as university-based student affairs/career centers.

Utilize social media and interaction with campus staff and student groups to supplement outreach and amplify visibility and credibility of the recruitment effort.

In response to these outreach/contact methods and resources, potential participants will inform EurekaFacts of their interest to participate in the study by calling or e-mailing the EurekaFacts contact information found in the outreach/contact materials or following a link to the self-administered screener.

In addition to EurekaFacts identification of respondents, the eligibility screener will also be advertised on Amazon’s MTurk crowdsourcing platform, in an effort to quickly recruit potential participants. Qualified “workers” based on MTurk’s premium qualifications, such as education level, will be able to view the request upon logging into their worker account.

EurekaFacts will review the responses to the self-administered screeners of potential participants to identify eligible participants resulting from EurekaFacts’ outreach/contact methods and MTurk “worker” task completions. Potential participants will then be offered the option to participate in either an in-person interview or a remote interview.

Platform advertisement

Survey description: Paid Online Study Opportunity – Seeking students who attended a college, university, or trade school within the past three years. Tap here to get started <INSERT LINK TO SCREENER>

Time allotted: 5 minutes

Expires 05/31/2020

Qualifications required: Resides in U.S. and enrolled in postsecondary education

Figure 1. Example advertisement from a previous study

Sample Recruitment Advertisements

Advertisements conducted by EurekaFacts will target individuals qualified for BPS:20/22, describing briefly the purpose of the testing and the opportunity for participants to contribute to the development of a survey. The advertisements will identify the need for individuals who attended a college, university, or trade school within the past three years, the time commitment of the interview, and the incentive amount offered for participation.

Sample advertisements using a variety of methods are presented below. The main method will include direct e-mails/ letters sent to individuals and organizations with members who may qualify. Advertisements to organizations will specify the recruitment of first-time beginning students since January 1, 2017 and the initial advertisements to individuals will reference attendance in “the past three years” so that this eligibility requirement is not directly provided to potential participants. The e-mail or letter will also include a flyer and an FAQ page. In addition, social media post language has been included for online advertisements. As part of the recruitment process, recruitment scripts, confirmation phone scripts and e-mails, interview invitation e-mails, and thank you e-mails may be used and are provided in this document.

Initial Contact Recruitment E-mail/Letter to Individuals

IF SENT AS E-MAIL–Subject: Paid Interview Study Opportunity

<DATE>

Dear <NAME>,

The U.S. Department of Education’s National Center for Education Statistics (NCES) has contracted EurekaFacts, a social science research firm, to administer an important study, and we need your help.

We are looking for students who attended a college, university, or trade school within the past three years to participate in voluntary interviews. During the 60-minute interviews, participants will provide feedback on survey questions intended for use in the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22). The BPS study collects data on student persistence in, and completion of, postsecondary education programs; their transition to employment; demographic characteristics; and changes over time in their goals, marital status, income, and debt, among other indicators.

Participants will have the ability to choose whether they would like to do the interview in-person at the EurekaFacts office in Rockville, MD or complete it remotely from a quiet location of their choosing where they have access to either a desktop or laptop computer and high-speed internet. Upon successful completion of the interview, all participants will be provided with a $50 gift card from a major credit card company as a ‘thank you’ for their time and effort.

By participating in this study, you will improve the BPS:20/22 survey, so that it can be easily understood and answered by thousands of other current and recent college students across the country!

[<IF E-MAILED> If you are interested in participating, please click on the following link and answer a few screening questions: 

<INSERT LINK TO INTAKE/SCREENING FORM>]

For more information about the study, please see our <INSERT LINK TO flyer> and <INSERT LINK TO FAQ page>. Please call <CONTACT PHONE> or e-mail <CONTACT E-MAIL> with any questions regarding this project.

We look forward to speaking with you soon!

Sincerely,

<CONTACT NAME>

<JOB TITLE>

EurekaFacts, LLC <CONTACT PHONE>

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Recruitment E-mail/Letter to Organizations

IF SENT AS E-MAIL–Subject: Paid Interview Study Opportunity

<DATE>

Dear <NAME>,

EurekaFacts, a research company based in Rockville, MD, is administering an important study on behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES). [<IF SCHOOL OR EDUCATION ORGANIZATION> Because of your role in the educational community <IF OTHER ORGANIZATION> Because your organization plays an active role in the community], we need your help.]

We are looking for current or recent students who first attended a college, university, or trade school at some point since January 1, 2017 to participate in voluntary interviews. During the 60-minute interviews, participants will provide feedback on survey questions intended for use in the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22). The BPS study collects data on student persistence in, and completion of, postsecondary education programs; their transition to employment; demographic characteristics; and changes over time in their goals, marital status, income, and debt, among other indicators.

Participants will have the ability to choose whether they would like to do the interview in-person at the EurekaFacts office in Rockville, MD or complete it remotely from a quiet location of their choosing where they have access to either a desktop or laptop computer and high-speed internet. While interviews will be audio and video recorded, NCES is required to follow strict procedures to protect personal information in the collection, reporting, and publication of data.⁰

Upon successful completion of the interview, all participants will be provided with a $50 gift card from a major credit card company as a ‘thank you’ for their time and effort. Participants who complete an interview at EurekaFacts will be provided with a physical gift card while those who complete the interview remotely will receive an e-gift card via e-mail.

You can play a key role in making sure that a variety of voices are heard to improve the survey used in BPS:20/22, so that it can be easily understood and answered by thousands of current and recent college students across the country.

We would like to ask for your assistance in letting your [<IF SCHOOL OR EDUCATIONAL ORGANIZATION> students and/or members] [<IF OTHER ORGANIZATION> members and/or clients] know how they can be involved in this important study.] Here is a link to a <INSERT LINK TO flyer> with a brief description of our study in the hope that you can post it for everyone to see. We have also included a link to our <INSERT LINK TO FAQ page> with additional information about the study.

We appreciate your consideration of this request and invite you to please call <CONTACT PHONE> or e-mail <CONTACT E-MAIL> so we can answer any questions you may have.

We look forward to speaking with you soon!

Sincerely,

<CONTACT NAME>

<JOB TITLE>

EurekaFacts, LLC <CONTACT PHONE>

Recruitment Phone Script for Potential Participants/Respondents

(Script when calling potential participants from a compiled list)

Good morning/afternoon/evening. Is this the <LAST NAME FROM CALL LIST> household? My name is <CALLER NAME> and I am calling from EurekaFacts, a Maryland research company. Our company is administering an important study on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education and we need your help.

We are recruiting individuals to participate in voluntary interviews where they will provide feedback on survey questions used in the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22). If you qualify and participate in the study, you will receive a $50 gift card from a major credit card company as a ’thank you.’

I’d like to tell you a little bit more about the study, but first, if now is a good time, can I ask you a few questions to determine your eligibility? (YES or NO and call back later)

[IF YES, ADMINISTER ELIGIBILITY SCREENING QUESTIONS.]

Thank you. During the interview, you will be asked to answer survey questions about your postsecondary education experiences. Then your interviewer will ask follow-up questions to understand your experience completing the questions such as how you perceived the difficulty of the questions, your confidence in your answers, and your opinion of the layout and usability of the survey in terms of the device you take it on.

Your participation and opinions will help us improve the survey, so that it can be easily understood and answered by thousands of other current and recent college students across the country.

The entire interview will last no longer than 60 minutes and participants may choose between completing the session at the EurekaFacts office in Rockville, MD or participating remotely from a quiet location of their choosing where they have access to either a desktop or laptop computer and high-speed internet. Upon successful completion of the interview, participants who participate at our office will receive a physical $50 gift card from a major credit card company while those who participate remotely will receive a $50 e-gift card from a major credit card company over e-mail.

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).]

Are you interested in participating in this study?

If NO: Thank you for your time. Have a good <day/evening>.

If YES: Great, thank you very much. To ensure that we interview a broad mix of individuals, I have a few brief screening questions to ask you. It should only take about 5 minutes of your time.

[RESUME ADMINISTERING ELIGIBILITY SCREENING QUESTIONS (see Attachment II)]

General Social Media Post

[CONTENT OF POST]

Paid Online Study Opportunity – Seeking students who attended a college, university, or a trade school within the past three years. Tap here to get started:<INSERT LINK TO SCREENER>

[CONTENT OF SHARED IMAGE & LINK]

[Image will be linked to study’s eligibility screener hosted on the EurekaFacts website]

Specific Social Media Post

[CONTENT OF POST]

Paid Online Study Opportunity – Seeking students who attended college, university, or a trade school for the first time at some point since January 1, 2017. Tap here to get started:<INSERT LINK TO SCREENER>

[CONTENT OF SHARED IMAGE & LINK]

[Image will be linked to study’s eligibility screener hosted on the EurekaFacts website]

General Flyer

OMB Control No.: 1850-0803

2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22)

BPS:20/22 is a nationally representative study that follows students’ paths through postsecondary education. We need participants for interviews where they will test, evaluate, and provide feedback on select survey questions from BPS:20/22 to improve the survey. We are looking for students who attended a college, university, or trade school within the past three years.

During the interview you will complete survey questions and provide feedback on the survey questions and your overall experience.

You will receive a $50 gift card from a major credit card company at the end as a thank you.

Your feedback will improve the survey, so that it can be easily understood and answered by thousands of current and recent college, university, and trade school students across the country.

The information you provide will be used only for survey development purposes. You will be assigned a study ID number and your responses will not be linked to your name.

Please contact us today if you are interested in participating.

For more information or to register, please call us at <NUMBER>, or go to the website below:

<LINK>

Specific Flyer

OMB Control No.: 1850-0803

2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22)

BPS:20/22 is a nationally representative study that follows students’ paths through postsecondary education. We need participants for interviews where they will test, evaluate, and provide feedback on select survey questions from BPS:20/22 to improve the survey. We are looking for current or recent students who first attended a college, university, or trade school at some point since January 1, 2017.

During the interview you will complete survey questions and provide feedback on the survey questions and your overall experience.

You will receive a $50 gift card from a major credit card company at the end as a thank you.

Your feedback will improve the survey, so that it can be easily understood and answered by thousands of current and recent college, university, and trade school students across the country.

The information you provide will be used only for survey development purposes. You will be assigned a study ID number and your responses will not be linked to your name.

Please contact us today if you are interested in participating.

For more information or to register, please call us at <NUMBER>, or go to the website below:

<LINK>

OMB Control No.:1850-0803 185008080803XXXX

Frequently Asked Questions (FAQs)

1. What is this study about? 

On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES), EurekaFacts, a research company based in Rockville, MD, is looking for students who attended a college, university, or trade school within the past three year to participate in cognitive interviews, to provide feedback on a set of survey questions from the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22). Your feedback about the survey will help NCES improve the survey, so that it can be easily understood and answered by thousands of current and recent college students across the country.

2. Where will the interview take place?

Participants can choose whether they would like to come to the EurekaFacts office at 51 Monroe Street, PE 10, Rockville, MD 20850 to do the interview or complete it remotely. Participants electing for a remote session must have access to either a desktop or a laptop computer as well as a high-speed internet connection and webcam. If you would like to participate remotely and don’t have your own webcam, EurekaFacts can ship one to you along with a pre-paid shipping label for you to return it to us. All webcams will have to be returned to receive the incentive. For those who elect to come in for the interview, our office is conveniently located across the pedestrian bridge connected to the Red Line Rockville Metro Station. There is also ample parking in our building.

3. What will happen during the interview?

During the interview, you will be asked to answer survey questions about your postsecondary education experiences. Some topics may include your enrollment experiences since January 1, 2017, social and academic activities during college, and any employment information since beginning college. After answering certain survey questions, your interviewer will ask follow-up questions to get your opinion on the survey. These questions ask you to evaluate the difficulty of the survey question, your confidence in your answers, and your understanding of the survey question.

4. What kind of device can I take the survey on?

Participants who elect to come into our office for the interview may use either a smartphone, tablet, or laptop to complete the survey. If you would like to participate at our office, we can also provide a device for you to use.

Participants who complete the interview remotely must have access to either a laptop or desktop computer in addition to a high-speed internet connection and a webcam. We can provide participants who want to participate remotely with a webcam if they do not have one; the webcam will need to be returned to receive the incentive.

5. How long will the whole session last?

The interview will not exceed 60 minutes.

6. Will information be kept confidential?

You will be assigned a unique identifier (ID) and your name will not be linked to your answers. All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Audio and video of the session will be recorded only for analysis, and all personally identifying information will be destroyed at the close of the study.

7. Is there an incentive for participating?

Yes. If you participate at our office, you will receive a physical $50 gift card from a major credit card company as soon as your interview is over. If you participate remotely and do not borrow a webcam, you will receive your $50 gift card electronically via e-mail 5-7 business days after your interview; this e-gift card will also be from a major credit card company. If you participate remotely and borrow a webcam, your e-gift card will arrive 5-7 days after we receive the webcam back from you.

8. What are the possible risks of being in the study?

There are no known risks to participating in this voluntary study.

9. Who can I contact with questions or for further information?

If you have any questions about the study, you can call <NAME>, the <TITLE> at <CONTACT PHONE>. If you have questions about your rights as a study participant, you can call the EurekaFacts at <PHONE NUMBER> (a toll-free number).

10. Can participation end early?

Participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to stop answering the survey questions at any time.

Confirmation E-mail for In-person Participants

Subject: Confirmation for NCES Interview Study

<DATE>

Dear <NAME>:

Thank you for agreeing to participate in this voluntary interview to provide feedback on survey questions for the National Center for Education Statistics (NCES). The session will last 60 minutes.

This is confirmation that you are scheduled at the following location, date, and time:

LOCATION: EUREKAFACTS, LLC

51 Monroe Street, PE 10

Rockville, MD 20850

DATE: <Day of Week>, <Month> <Day>, <Year>

TIME: XX:XX AM/PM Eastern

GIFT CARD: You will receive a $50 gift card from a major credit card company at the conclusion of the session for your time and effort.

PLEASE READ THE FOLLOWING INSTRUCTIONS:

You will come to the EurekaFacts office at 51 Monroe Street, PE 10, Rockville, MD 20850. Our office is conveniently located across the pedestrian bridge connected to the Red Line Rockville Metro Station. There is also ample validated parking in our building.

Please bring the following:

• Your valid picture ID – this is required for participation.

• Either a smartphone, tablet or laptop. If this is not possible, we will provide you with a device to use.

• Signed consent form (attached); we will have copies available, if needed.

• If you use eyeglasses for reading or watching TV, you should wear them during the study session.

We look forward to speaking with you. If you have any questions or concerns, please contact:

<CONTACT NAME>

EurekaFacts, LLC

<CONTACT INFORMATION >

ATTACHED: Consent Form

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Confirmation E-mail for Remote Participants

Subject: Confirmation for NCES Interview Study

<DATE>

Dear <NAME>:

Thank you for agreeing to participate in this voluntary interview to provide feedback on survey questions for the National Center for Education Statistics (NCES). The session will last 60 minutes.

This is confirmation that you are scheduled at the following location, date, and time:

LOCATION: The interview will take place as an online meeting. In the coming days, your interviewer will reach out directly by sending you an e-mail invitation to the meeting. At your scheduled interview time, please be in a private location with strong internet connection, a webcam, and a computer or laptop to use to complete the survey.

DATE: <Day of Week>, <Month> <Day>, <Year>

TIME: XX:XX AM/PM <TIMEZONE>

GIFT CARD: All participants will receive a $50 e-gift card as a thank you for their time.

PLEASE READ THE FOLLOWING INSTRUCTIONS:

Please use these directions below for conducting your remote interview: 

• Before the interview time, open the attached consent form, type your name and the date, save it. Please reply all to this e-mail and attach the signed consent form. This will have to be done in order to participate in the interview.

• Please be in a quiet, private location with a strong internet connection. You will need to join the meeting from either a computer or laptop and make sure your webcam is set up. 

We look forward to speaking with you. If you have any questions or concerns, please contact:

<CONTACT NAME>

EurekaFacts, LLC

<CONTACT INFORMATION >

ATTACHED: Consent Form

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

In-person Participant Confirmation Phone Script

(Parenthetical text is provided for reference and does not need to be read aloud)

Good morning/afternoon/evening. May I please speak to <NAME> about an interview session at EurekaFacts to test and provide feedback for survey questions for the National Center for Education Statistics (NCES)?

Good morning/afternoon/evening. My name is <NAME> and I’m calling from EurekaFacts about your scheduled interview session that we are carrying out on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. We are pleased that you have agreed to participate in this study and help us improve our survey. This is a reminder that we have scheduled your session for <TIME><TIMEZONE> on <DATE>.

The session will take place at our office at 51 Monroe Street, PE10, Rockville, MD 20850 and will last no more than 60 minutes.

During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since January 1, 2017, social and academic activities during college, and any employment information since beginning college. Your interviewer will ask open-ended questions about your overall survey experience and also about your understanding and confidence when answering specific survey questions. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

As soon as you complete the session, you will receive a $50 gift card from a major credit card company as a thank you for participating.

Do you have any questions at this time?

<Answer any questions.>

Again, your session is scheduled at <TIME><TIMEZONE> on <DATE>. You will come to the EurekaFacts office at 51 Monroe Street, PE 10, Rockville, MD 20850. Our office is conveniently located across the pedestrian bridge connected to the Red Line Rockville Metro Station. There is also ample validated parking in our building. Please arrive 15 minutes before your scheduled time to check in. If you have any questions before then, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this very important study.

Remote Participant Confirmation Phone Script

(Parenthetical text is provided for reference and does not need to be read aloud)

Good morning/afternoon/evening. May I please speak to <NAME> about an interview session at EurekaFacts to test and provide feedback for survey questions for the National Center for Education Statistics (NCES)?

Good morning/afternoon/evening. My name is <NAME> and I’m calling from EurekaFacts about your scheduled interview session that we are carrying out on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education. We are pleased that you have agreed to participate in this study and help us improve our survey. This is a reminder that we have scheduled your session for <TIME><TIMEZONE> on <DATE>.

The session will take place virtually as an online meeting.

During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since January 1, 2017, social and academic activities during college, and any employment information since beginning college. Your interviewer will ask open-ended questions about your overall survey experience and also about your understanding and confidence when answering specific survey questions. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

You will receive a $50 gift card as a thank you for your participation.

Do you have any questions at this time?

<Answer any questions.>

Again, your session is scheduled at <TIME><TIMEZONE> on <DATE>. At the time of your interview please be in a quiet location where you have access to high-speed internet, a webcam, and either a laptop or desktop computer.

If you have any questions before then, please call <PHONE NUMBER>.

Thank you again for agreeing to be part of this very important study.

Thank You E-mail/Letter to Participants/Respondents

IF SENT AS E-MAIL - Subject: Thank you - NCES Survey Interview Study

<DATE>

Dear <Name>,

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, EurekaFacts would like to thank you for participating in your interview to provide feedback for the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22).

Your participation provided valuable feedback about the survey questions. Your open, candid, and insightful comments were very helpful. Your input, combined with input from other respondents like you, is incredibly useful and informative and is helping us develop the best possible survey questions for NCES.

Again, thank you for your assistance.

Sincerely,

<CONTACT NAME>

<TITLE>

EurekaFacts, LLC

<CONTACT INFORMATION >

Attachment II. Eligibility Screening Questions

All initial eligibility screening will be conducted by telephone or through a self-administered online recruitment screener. The recruitment screener, a 3-5-minute online survey, will be used to determine whether individuals qualify to participate in the study to test a subset of BPS:20/22 survey items. Potential respondents who contact EurekaFacts via telephone or e-mail in response to the recruitment advertisements will have the choice to complete eligibility screening on the telephone with EurekaFacts or receive an e-mail containing the online recruitment screener link.

Respondents who click the screener link provided in the MTurk platform advertisement, targeted social media ads, and e-mails will go directly to the online recruitment screener. The responses to the online recruitment screener questions will determine their eligibility for the project.

If the recruitment screener determines that potential respondents are eligible, respondents will receive a phone call to schedule the cognitive interview at a time that is convenient for the respondent. Following the phone call, respondents will also receive a confirmation e-mail and/or phone call with the date, time, and location information for the session (see Attachment I for cognitive interview confirmation e-mail and phone script). For respondents participating remotely, the confirmation e-mail will also include a consent form that respondents must sign and return to EurekaFacts prior to the scheduled interview in order to participate.

The introduction to the eligibility screener for the telephone agents when screening is conducted over the telephone can be found in “Recruitment Phone Script” in Attachment I. The script in the beginning of the screener is the text displayed on the online version. Likewise, the language for the second to last question has two versions, dependent on whether the survey would be viewed by the participant or read to them by a EurekaFacts agent.

E-mail with Self-Screener to Interested Participants who Call EurekaFacts

IF SENT AS E-MAIL - Subject: Paid Interview Study Opportunity

<DATE>

Hello <NAME>,

Thank you for your interest in our study. EurekaFacts is carrying out this study among students who attended a college, university, or trade school within the past three years and would greatly value your opinions.

Eligible respondents who participate in a 60-minute interview will receive a $50 gift card as a thank you for their feedback. If you are interested, please click on the link below to take a brief survey to determine your eligibility.

<Link to online eligibility screener>

Thank you,

<CONTACT NAME>

<TITLE>

EurekaFacts, LLC

<CONTACT INFORMATION >

NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Eligibility Screener (Self-Administered and Over the Telephone)

See “Recruitment Phone Script” in Attachment I for language EurekaFacts agents use when speaking to participants. For the self-administered screener, the introduction and question about interest in participating are presented on the first page. Clicking “Yes” brings the participant to the eligibility questions on the next screen. The PRA statement is added as a footnote, visible beneath the “Next” button.

Introduction to self-administered screener:

Thank you for your interest. EurekaFacts is carrying out an important study on behalf of the National Center for Education Statistics (NCES), which is part of the U.S. Department of Education, and we need your help. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

We are recruiting individuals to participate in a voluntary interview to provide feedback about survey questions related to postsecondary education experiences. During the interview, you will be asked to answer survey questions about your postsecondary enrollment experiences since January 1, 2017. Some topics may include your enrollment experiences since January 1, 2017, social and academic activities during college, and any employment information since beginning college. Additionally, you will be asked to evaluate the difficulty of the questions, your confidence in your answers, and your understanding of the questions.

The interview will take about 60 minutes to complete and can be scheduled at your convenience. Some of the interviews will be conducted at the EurekaFacts office in Rockville, MD, while others will be conducted remotely via a video online conferencing program, such as Skype for Business. All of the information provided by participants may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

If you qualify and participate in an online interview, you will receive a $50 e-gift card from a major credit card company for as a thank you within 10 business days following the interview. If you qualify and participate in the interview at the EurekaFacts office, you will receive a $50 gift card from a major credit card company at the end of the session as a thank you.

No matter the method – your participation and opinions will help us improve the questionnaire, so the questions are clear and can be easily understood and answered by thousands of other current and recent college students across the country.

Are you interested in participating in this study?

• Yes  CONTINUE

• No  THANK AND TERMINATE

Great, thank you very much. To ensure that we interview a broad mix of individuals, please answer the following brief screening questions. It should take only about 5 minutes of your time⁰

NEXT

Click next to advance to the next screen

1. Were you attending high school or completing any high school requirements at any time between January 1, 2017 and today?

   • Yes

   • No

2. What is your age?

• Under 18  THANK AND TERMINATE (Ineligible)

• 18-24

• 25-29

• 30-34

• 35-39

• 40-49

• 50-59

• 60 or older

3. In which month and year did you first attend any college, university, or trade school after completing your high school requirements?

- Month

- Year

**If Q3 = (January 2017 – June 2020)  CONTINUE, else THANK AND TERMINATE (Ineligible)

4. What is the name of the college, university, or trade school you first attended in [Q3 MONTH] [Q3 YEAR]?

   [NPSAS INSTITUTION]

5. Which of the following were you working on at [NPSAS INSTITUTION] when you first attended in the [{if Q3 between Jan 2017 and June 2017} 2016-2017 academic year (July 1, 2016 – June 30, 2017) {else if Q3 between July 2017 and June 2018} 2017-2018 academic year (July 1, 2017 – June 30, 2018) {else if Q3 between July 2018 and June 2019} 2018-2019 academic year (July 1, 2018 – June 30, 2019) {else} 2019-2020 academic year (July 1, 2019 – June 30, 2020)]?

• Associate’s degree (usually a 2-year degree)

• Bachelor’s degree (usually a 4-year degree)

• Undergraduate certificate or diploma (usually less than 2 years), including those leading to a certification or license (example: cosmetology)

• Undergraduate level classes

• Other  THANK AND TERMINATE (Ineligible)

6. Are you currently attending [NPSAS INSTITUTION]?

(Answer "No" if you are on a break between semesters, terms, or quarters, such as on summer break, and not currently attending classes.)

• Yes  Q9

• No  Q7

7. Did you complete all the requirements for the [NPSAS DEGREE] you began in the [Q3 MONTH] [Q3 YEAR] academic year at [NPSAS INSTITUTION]?

• Yes

• No

8. Are you currently attending any college, university, or trade school?

(Answer "No" if you are on a break between semesters, terms, or quarters, such as on summer break, and not currently attending classes.)

• Yes

• No

9. Other than the [NPSAS DEGREE] you [{if Q6 = yes} are {else} were] working on at [NPSAS INSTITUTION], have you earned any degrees or certificates since high school?

(Only include degrees or certificates earned through a college, university, or trade school.)

• Yes  THANK AND TERMINATE (Ineligible)

• No  CONTINUE

10. Are you of either Hispanic or Latino origin?

• Yes

• No

• Prefer not to Answer

11. What is your race? (Select all that apply)

• American Indian or Alaska Native

• Asian

• Black or African American

• Native Hawaiian or Other Pacific Islander

• White

• Prefer not to answer

12. What is your total annual income based on the following ranges?

• Less than $20,000

• $20,000 to $49,999

• $50,000 to $99,999

• $100,000 or more

• Prefer not to answer

13. What is your sex?

• Male

• Female

• Prefer not to answer

14. To participate in the interview and receive your gift card, we must receive a signed consent form. If you participate remotely via online conferencing, you will have to read, complete, and return the consent form before the scheduled interview. If you are coming to our Rockville, MD office to participate in the interview, you will have to sign the consent form in our office before the start of the session. Do you understand that you must have a completed and signed consent form to participate?

    • Yes

    • No  THANK AND TERMINATE (Ineligible)

15. Eligible respondents will have the opportunity to decide whether they would like to come to our Rockville, MD office for the interview or complete the interview remotely from a location of their choosing. However, remote interviews do have special technological requirements. In some cases, we may be able to provide eligible respondents with missing technology. Are you interested in finding out if you have the necessary technology for a remote interview or are you certain you would come to the Rockville, MD office if an interview were scheduled?

• Yes, I would like to check my technology  Cont. to Q16

• No, I’m certain I would do my interview in-person Skip to Q21

16. To participate in a remote interview, you would need access to either a desktop or laptop computer. Would you be able to access a desktop or laptop computer to complete a remote interview?

• Yes  Skip to Q18

• No  Cont. to Q17

17. Unfortunately, either a laptop or desktop computer is required for a remote interview. However, we can accept a wider range of technology for in-person interviews at our office. Is it at all possible for you to come to our Rockville, MD office for an in-person interview?

• Yes  Skip to Q21

• No  THANK AND TERMINATE (Ineligible)

18. To participate in a remote interview, you would also need access to a webcam. Would you be able to access a webcam to complete a remote interview?

• Yes  Cont. to Q19

• No  Cont. to Q19

19. Finally, to participate in a remote interview, you would also need access to a microphone. A microphone could be built-in to your laptop, desktop, or webcam or it could be a separate device attached to your laptop or desktop by a USB cord. Would you be able to access a microphone for a remote interview?

• Yes

• No

If respondent answers “Yes” to Q18 AND Q19, skip to Q21. If respondent answers “No” to Q18 OR Q19, ask Q20:

20. Unfortunately, you are missing a key piece of technology, but we may be able to help. Once we receive your signed consent form, we would be able to send you a webcam with a built-in microphone to use during your interview at no cost to you. However, you would need to return it to us in a pre-paid return box before we could begin processing your incentive payment. Do you understand and agree that the webcam will need to be returned in order for us to process your $50 incentive?

• Yes  Cont. to Q21

• No THANK AND TERMINATE (Ineligible)

21. [IF SELF-ADMINISTERED]
    Thank you for your interest. If you qualify, EurekaFacts will reach out to you to schedule your interview session. To make sure that happens at an appropriate time, can you please confirm your time zone for us?
    [IF TELEPHONE SCREENING]
    Thank you for your interest. If you qualify, we will follow up with you to schedule the interview. To make sure we do that at an appropriate time, can you please confirm your time zone for me?

• Eastern Standard Time (EST)

• Central Standard Time (CST)

• Mountain Standard Time (MST)

• Pacific Standard Time (PST)

• Alaska Standard Time (AKST)

• Hawaii Standard Time (HST)

22. And lastly, could you please provide us with your preferred contact information? (All required except “Alternate phone number.”)]

RECORD FIRST NAME: _____________________________________

RECORD LAST NAME: _____________________________________

RECORD E-MAIL ADDRESS: _____________________________________

BEST PHONE NUMBER: _____________________________________

ALTERNATE PHONE NUMBER: _____________________________________

[Termination Language: Thank you for your expressed interest. Based on the requirements of this study, we are not able to include you in the study at this time. <IF TELEPHONE> Have a good <day/evening.>]

Figure 1. Cognitive Interview Survey Login Page - Screenshot

This image is provided as an example. The survey login page is still being developed and the final image may include aesthetic changes that do not impact the content of the page.

Attachment III. Consent to Participate in Cognitive Interview

Consent Form for Participants

CONSENT FORM

FOR PARTICIPATION IN COGNITIVE INTERVIEW

Thank you for your interest in participating in the the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22) cognitive interviews. This consent form explains the study so please read it carefully and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them later as they occur to you.

1. What is this study about? 

On behalf of the U.S. Department of Education’s National Center for Education Statistics (NCES), EurekaFacts, a research company based in Rockville, MD, is looking for students who attended a college, university, or trade school within the past three years to participate in cognitive interviews, to provide feedback on a set of survey questions from the 2020/22 Beginning Postsecondary Student Longitudinal Study (BPS:20/22). Your feedback about the survey will help NCES improve the survey, so that it can be easily understood and answered by thousands of current and recent college students across the country.

2. Where will the interview take place?

Participants can choose whether they would like to come to the EurekaFacts office at 51 Monroe Street, PE 10, Rockville, MD 20850 to do the interview or complete it remotely. Participants electing for a remote session must have access to either a desktop or a laptop computer as well as a high-speed internet connection and webcam. If you would like to participate remotely and don’t have your own webcam, EurekaFacts can ship one to you along with a pre-paid shipping label for you to return it to us. All webcams will have to be returned to receive the incentive. For those who elect to come in for the interview, our office is conveniently located across the pedestrian bridge connected to the Red Line Rockville Metro Station. There is also ample parking in our building.

3. What will happen during the interview?

During the interview, you will be asked to answer survey questions about your postsecondary education experiences. Some topics may include your enrollment experiences since January 1, 2017, social and academic activities during college, and any employment information since beginning college. After answering certain survey questions, your interviewer will ask follow-up questions to get your opinion on the survey. These questions ask you to evaluate the difficulty of the survey question, your confidence in your answers, and your understanding of the survey question.

4. What kind of device can I take the survey on?

Participants who elect to come into our office for the interview may use either a smartphone, tablet, or laptop to complete the survey. If you would like to participate at our office, we can also provide a device for you to use.

Participants who complete the interview remotely must have access to either a laptop or desktop computer in addition to a high-speed internet connection and a webcam. We can provide participants who want to participate remotely with a webcam if they do not have one; the webcam will need to be returned to receive the incentive.

5. How long will the whole session last?

The interview will not exceed 60 minutes.

6. Will information be kept confidential?

You will be assigned a unique identifier (ID) and your name will not be linked to your answers. All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Audio and video of the session will be recorded only for analysis, and all personally identifying information will be destroyed at the close of the study.

7. Is there an incentive for participating?

Yes. If you participate at our office, you will receive a physical $50 gift card from a major credit card company as soon as your interview is over. If you participate remotely and do not borrow a webcam, you will receive your $50 gift card electronically via e-mail 5-7 business days after your interview; this e-gift card will also be from a major credit card company. If you participate remotely and also borrow a webcam, your e-gift card will arrive 5-7 days after we receive the webcam back from you.

8. What are the possible risks of being in the study?

There are no known risks to participating in this voluntary study.

9. Who can I contact with questions or for further information?

If you have any questions about the study, you can call <NAME>, the <TITLE> at <CONTACT PHONE>. If you have questions about your rights as a study participant, you can call the EurekaFacts at <PHONE NUMBER> (a toll-free number).

10. Can participation end early?

Participation in this study is completely voluntary. You have the right to refuse to answer particular questions. You may elect to stop answering the survey questions at any time.

____ Yes, I agree to participate in this study

____ No, I do not want to participate in this study

By signing or typing your name below, you agree to participate in this cognitive interview and understand that your responses will be recorded. You agree to receive your participation incentive in the form of a $50 gift card and according to the stipulations in section 7 of this consent form.

Your written or typed signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

We thank you for your voluntary participation and ask you to sign the form below.

Print Name: _____________________________________________

Signature line:____________________________________________

Date:________________________

To be completed by EurekaFacts staff:

Respondent Payment Card Transaction Number: _______________________________________________________

Attachment IV. Interview Protocol

Interview Protocol Guide

Note to Interviewer: The following script does not have to be read verbatim. Interviewers should be thoroughly familiar with the content to the extent that you present the cognitive interview process to the participant in a conversational style throughout the interview. You should develop a friendly rapport by projecting a warm and reassuring tone toward the participant.

After answering questions and giving further explanation, begin the interview with the first question.

The duration of each cognitive interview will be up to 60 minutes for participants.

Section 1: Welcome

Hello, my name is ______.

Thank you for being available today to help us.

I have some questions that participants from a variety of schools and degree programs across the country will be asked to answer in an upcoming study. These questions are part of a nationally representative study, conducted by the National Center for Education Statistics, that follows students’ paths through postsecondary education.

The purpose of this interview is to essentially “test the survey.” Our goals are to make sure that these questions are applicable to a wide variety of educational and employment circumstances, and to make the questions clearer so that others can easily understand and answer them. That is why we need your feedback. Your candid responses and feedback today will help us improve the survey for others and, ultimately, help to ensure that the results of the study are accurate and valid.

There are no right or wrong answers. We will work together on this task for a total of 60 minutes. If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This interview will be audio recorded and may be video recorded so that analysts can review the recordings later. Recording the interview also allows me to focus all my attention on our discussion, since what you have to say is very important. Do I have your permission to record this interview?

**START RECORDING**

Section 2: Informed Consent (See Attachment III)

In addition to recording the interview, there may be observers listening in so that they can see firsthand what it is like for you to answer the survey questions. Nevertheless, all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Furthermore, your name will not be included in any report. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

Your participation in this interview is very important because it will help us improve an education study with topics related to respondents like you. Once we start, the interview will last about an hour. When we are done, you will receive a $50 <e-gift card or gift card> from a major credit card company <if remote, via e-mail within 5-7 business days> as a thank you for your participation.

Do you have any questions?

Section 3: Cognitive and usability Interviewing

Let me explain a little about what you can expect during this interview. As I said before, there are no right or wrong answers because this is not a test. I am interested in your experiences, thoughts, and interpretations of the survey questions. I will be asking you for details about your interpretations of the survey questions, such as how you understand what is being asked or if anything was confusing to you. Like I said before, your honest feedback is important, so please feel free to share your thoughts and ask any questions that come to your mind.

FOR REMOTE INTERVIEWS, PROVIDE INSTRUCTIONS ABOUT TECHNOLOGY (E.G. SCREEN SHARING, SCREEN CONTROL) AND GENERAL INTERVIEW PROCESS.

FOR IN-PERSON INTERVIEWS, PROVIDE INFORMATION ABOUT GENERAL INTERVIEW PROCESS.

ASK THE PARTICIPANT TO BEGIN THE ONLINE SURVEY.

CONDUCT INTERVIEW. ASK A SUBSET OF GENERAL PROBES (EXAMPLES PROVIDED BELOW) AT QUESTIONS WHERE THE RESPONDENT HAS GIVEN SOME INDICATION OF DIFFICULTY WITH THE QUESTIONING BASED ON ANSWERS/BEHAVIORS, AS THE INTERVIEW PROGRESSES.

EXAMPLES OF GENERAL PROBES:

• What did you think the question was asking?

• Please explain, in your own words, what the question is asking.

• How did you go about answering this question?

• Please explain how you came up with your answer.

• What were you thinking of?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer the question? If difficult, what could we do to make it easier to answer?

• Probe for examples.

• Did you find any words or part of the question confusing? If so, what could we do to make it less confusing?

• How did you feel about answering the questions?

• What does <this word> mean to you?

• Can you tell me more about that?

• You seemed to hesitate/take a moment before giving/entering your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term/question sound ok to you, or would you prefer something different?

EXAMPLES OF USABILITY PROBES:

• How did you find the overall experience of completing the survey?

• What was your overall impression of the features and functions of the survey?

• Did the features and tools function the way you expected? If not, why?

• If you could change anything about the way the survey worked, the features or functions, what would it be and why?

• What did you like best about the survey and why?

• What did you like least about the survey and why?

• What did you think about the layout of the survey pages?

• Was there anything in the survey that made it particularly easy or difficult to complete?

• Were specific questions more difficult to read than others? What would make it easier to answer these questions?

• Are there any specific changes or revisions that you think could be made to the survey to make it easier to complete by computer, phone, and tablet?

Section 4: Debriefing Procedures

• Ask for any last observations/comments/concerns.

• Stop recording.

• Thank the respondent for helping to test the survey questions.

• For remote interviews, obtain details needed for e-gift cards and end interview.

• For in person interviews, end interview and provide $50 gift card.

0 All of the information provided may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151) . NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543).

0According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated at 5 minutes per response, including the time to review instructions, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or comments or concerns about the contents or the status of your individual submission of this questionnaire, please write directly to: BPS:20/22, National Center for Education Statistics, Potomac Center Plaza, 550 12th Street SW, Washington, DC 20202.

1