Various Demographic Area Pretesting Activities

Protocol for 2015 National Survey of College Graduates

Participant ID #: |_N__|_S__|_C__|_G__|___| ___|___|___|

Interview Date: |__|__| / |__|__| / |__|__|__|__| (mm/dd/yyyy)

Interviewer initials: |__|__|

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Section 1: Interviewer: Read/ Paraphrase the following text:

Greeting: Hello. My name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study.

What: In order to help us improve our surveys, we turn to people like you to find out if our questions make sense and are fairly easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with people and see how it works for them. So you will be helping us test a questionnaire from one of our surveys.

How: [FOR PAPER TESTING: SHOW RESPONDENT NSCG FORM] This is a questionnaire called the National Survey of College Graduates. It this were the real survey, you would receive this questionnaire in the mail. In a couple of minutes I am going to ask you to fill out this survey just as you would if you had received the questionnaire at your home but with one major difference.

[FOR CATI TESTING]: I want you to answer the questions exactly the way you would if an interviewer had come to your home for an official interview, but with one major difference.

[FOR WEB TESTING; see next page]: I want you to answer the questions exactly the way you would if you got on your computer at home, but with one major difference.

Think Aloud: I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: How many windows are in your home?

Interruptions: [FOR PAPER TESTING]: I am going stop you at a couple of points in the questionnaire and ask you some questions about your answers, or about the questions themselves . Remember, there are no right or wrong answers, because only you know what you are thinking.

[FOR CATI AND WEB TESTING]: From time to time I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.

Do you have any questions before we begin?

Confidentiality: Our session today is completely confidential. Any names you provide will never be used in our reports. Your participation in this study is completely voluntary and you can decline to answer any particular question.

Recording: So I don’t have to rely on my memory later on, I’d like to tape record this interview. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes. Is that ok with you?

** HAVE PARTICIPANT SIGN CONSENT FORM**

**Begin: **TURN ON TAPE RECORDER**

OK, let’s begin. Please remember to think aloud as you answer the questions.

INTERVIEWER INSTRUCTIONS: [FOR PAPER TESTING:] AS THE RESPONDENT FILLS OUT THE FORM, RECORD OBSERVATIONS ASSOCIATED WITH EACH QUESTION. AFTER THE RESPONDENT HAS COMPLETED EACH SECTION, ASK THE PROBES ASSOCIATED WITH EACH QUESTION AND SECTION.

[FOR CATI AND WEB TESTING:] NO NEED TO TAKE NOTES, BUT AS YOU ASK THE QUESTIONS, RECORD ANSWERS WHEN REQUESTED IN THE CHECK BOXES SO YOU WILL NOT NEED TO SCROLL BACK THROUGH THE QUESTIONNAIRE.

FOR WEB TESTING ONLY:

Today, we will be working on a few projects together. There are three parts to our session:

1: First, we’ll do a word recall task.

2: Second you will help us evaluate the design of the online National Survey of College Graduates by completing it online. I’ll ask you some satisfaction questions and debriefing questions about the survey as well.

3: Third you will work on a set of questions.

Then we’ll wrap up and I or my colleague will escort you back out.

[Eye tracking]

In addition to the survey that we are about to work on, and thinking aloud that we just practiced, we are also going to record where you look on the screen. In a moment, we will do a simple task that will allow the computer to find your eyes. I am going to have you position yourself in front of the screen so that you can see your nose in the reflection at the bottom of the monitor. To calibrate your eyes, please follow the dot across the screen with your eyes.

[Calibrate the participants’ eyes]

Ok great. I now have a demographic questionnaire that I’d like you to fill out.

[Hand the demographic questionnaire to participant and wait in the room while they fill it out and return it to you]

Now I would like us to work on the word recall task. This is a short exercise that gives us information about memory.

[Proctor the memory task—this is in testing docs folder on M drive: 2013Commitment Research Study]

Complete survey and follow protocol

A1.

Probe: What were you doing at this time?

A2.

If yes, Probe: What does it mean to “look for work” in this question?

Probe: What time period were you thinking of when you answered this question?

A3.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

Probe: Was it easy or difficult to select a response?

A4.

A5.

A6.

A7.

Probe: How did you come up with your answer?

Probe: Was it easy or difficult to select a code?

Probe: Was there an answer you thought about but did not select

A8.

A9.

If more than one job, Probe: How did you decide which job to choose as your principal employer?

A10.

A11.

Probe: How did you come up with this estimate?

A12.

A13.

□ If yes to SELF-EMPLOYED or BUSINESS OWNER,

Probe: Tell me more about your principal employer. Do you own your own business or do you work for someone who owns their own business?

A14.

A15.

A16.

A17.

A18.

Probe: How did you come up with your answer?

Probe: Was it easy or difficult to select a code?

Probe: Was there an answer you thought about but did not select?

A19.

□ Did the respondent choose the option for “some other field, specify”?

CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

Probe: Which field were you thinking about when you gave your answer?

Probe: What does the term “technical expertise” mean to you in this question? Would the term “knowledge and skills” mean the same thing to you?

A20.

A21.

Probe: What does the phrase “related to your highest degree” mean to you in this question?

A22.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

Probe: What do the terms pay and promotion mean to you in this question? Do they mean the same thing or are they different?

A23.

A24.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

Probe: How did you come up with your answer?

Probe: Was there an answer you thought about but did not select?

A25.

A26.

A27.

A28.

Probe: Was there something else that contributed to your job satisfaction that was not listed here?

A29.

A30.

Probe: Can you tell me how you came up with your answer?

A31.

A32.

□ Did R have difficulty writing in the number of hours per week?

If yes, Probe: I noticed that you hesitated at this question. Can you tell me what you were thinking?

A33.

Probe: Would your answer have changed if the number of hours per week in the instruction was 30 instead of 35?

A34.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

A35.

A36.

A37.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

A38.

Probe: Was it easy or difficult to determine your earned income?

A39.

If yes, Probe: Can you tell me a bit more about your certification or license?

Probe: Can you give me an example of a professional license? Can you give me an example of an industry license?

A40.

□ Did R seem to read or have trouble with the response categories?

If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?

A41.

A42.

Probe: Can you tell me a bit more about your answer?

A43.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

A44.

A45.

□ Did the respondent choose the option for “required or expected by employer”?

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

A46.

Probe: Can you tell me a bit more about your answer?

A47.

If yes, Probe: Can you tell me more about your answer?

Probe: If you had to pay out of pocket for a license but were later reimbursed for your expenses, how do you think you would have answered this question?

A48.

Probe: In your own words, what is this question asking?

Probe: If the question had asked about your “current occupation,” would your answer have changed? Why? What if it asked about your “current line of work”?

A49.

□ R’s response is NOT consistent with the response option 4 in A45

If response is not consistent, Probe: I noticed that you selected the option for (required by an employer in A45 but not in A49/ required by an employer in A49 but not in A45). Can you tell me about what you were thinking?

□ Did R have difficulty translating responses into yes/no categories?

If yes, Probe: I noticed that you hesitated here. Can you tell me about what you were thinking?

[FOR PAPER TESTING: STOP AND ASK R ALL SECTION A PROBES]

B1.

B2.

Probe: What were you thinking about when you were answering this question?

B3.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

C1.

C2.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

C3.

C4.

C5.

C6.

[FOR PAPER TESTING]: STOP AND ASK R ALL SECTIONS B AND C PROBES

D1.

D2.

□ Did R go to high school in a foreign country?

D3.

D4.

□ Did R seem to read or have trouble with the response categories?

If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?

D5.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

D6.

D7.

D8.

D9.

If D2 = foreign country, Probe: Can you tell me more about that?

D10.

Probe: Can you tell me more about that?

D11.

Probe once per R for part c: □ Did the respondent choose the option for “other, specify”?

CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

Probe for part d: What were you thinking about when you were answering this question?

Probe for part d: Do you think minors should be included here?

D12.

Probe: Can you tell me more about this?

Probe: What education were you thinking about in this question?

Probe: Did you include room and board in your response, or just tuition?

D13.

Probe: How did you come up with your answer?

Probe: How difficult was it for you to answer this question?

If $1-10,000 or higher for one degree or more, Probe: Did you include interest in your estimate(s)?

D14.

If yes, probe: Cana you tell me a bit more about your educational certificate?

Probe: What does “educational certificate” mean to you in this question?

Probe: Can you give me an example of an educational certificate?

D15.

D16.

D17.

D18.

□ Did the respondent choose the option for “required or expected by employer”?

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

D19.

□ Did R seem to have trouble with the response categories?

If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?

If applicable, probe about university employer/graduate assistant situation

If not asked after section A, Probe: If you had to pay out of pocket for a certificate but were later reimbursed for your expenses, how do you think you would have answered this question?

D20.

Probe: What does this question mean to you?

Probe: If the question had asked about your “current occupation,” would your answer have changed? Why? What if it asked about your “current line of work”?

D21.

□ Did R seem to read or have trouble with the response categories?

If yes, Probe: I noticed that you hesitated at this question. Can you tell me about your answer?

□ R’s response is NOT consistent with the response option 4 in D18

If response is not consistent, Probe: I noticed that you selected the option for (required by an employer in D18 but not in D21/ required by an employer in D21 but not in D18). Can you tell me about what you were thinking?

D22.

D23.

D24.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

D25.

D26.

D27.

D28.

[FOR PAPER TESTING]: STOP AND ASK R ALL SECTION D PROBES

E1.

Probe: What does it mean to live in a “marriage-like relationship”?

E2.

Probe: Would your answer change if the question asked whether your spouse or partner was
“working for pay or profit”?

Probe: What does “full-time” work mean to you?

If necessary, Probe: How many hours do you think full-time work is?

E3.

□ Did the respondent choose the option for “other, specify”? CATI response: ________________

If yes, Probe: Do you think that your response could have fit within any of the provided categories?

If not asked after section A, Probe: What does the term “technical expertise” mean to you in this question? Would the term “knowledge and skills” mean the same thing to you?

E4.

E5.

E6.

E7.

E8.

E9.

E10.

E11.

E12.

E13.

Probe: What does this question mean to you?

Probe about respondent’s history of entrance and exit to U.S.

E14.

E15.

E16.

E17.

E18.

E19.

E20.

E21.

E22.

E23.

E24.

Probe: Can you tell me more about that?

E25.

E26.

E27.

E28.

Probe: How do you feel about providing this information?

Probe: Would you give names for these people?

How about phone numbers?

How about addresses?

How about email addresses?

E29.

Probe: How did choose your answer?

[FOR PAPER TESTING: STOP AND ASK R ALL SECTION E PROBES]

DEBRIEFING:

[FOR PAPER TESTING]: Hand respondent ACS employment question sheet. I have one other question for you to answer. It’s similar to one you answered earlier. Can you please answer this question?

[NOTE TO INTERVIEWER:] CHECK RESPONSE TO A13. IF ANSWERS ARE INCONSISTENT, PROBE.

[FOR CATI AND WEB TESTING]: WORDING OF PROBES WILL BE DETERMINED WHEN SCREEN SHOTS ARE AVAILABLE.

Overall, would you say the survey questions were easy or difficult to respond to? Which ones? Why?

Do you think there are questions some people would find difficult? Sensitive? Which ones?

Did you think any of the questions were asking the same thing?

Did you notice the italicized instructions in the survey?

Did you notice that some of the questions asked about different time periods? Did that cause you any confusion?

Do you have anything else you would like to tell us that you haven’t had a chance to mention yet?

Usability: [After survey is completed – Have R complete overall evaluation]

Ok great. I now have a questionnaire where you can evaluate the web survey.

[After final evaluation is complete-- Now proctor the research component]

I just need to step out and get something.

[Exit the room to retrieve and open the randomly selected envelope.

If envelope says “C” go back into participant’s testing room and say:]

Now it’s time for our final set of questions. Please answer these questions, I’ll be in the other room.

[Open the link FQ Questionnaire—located at (M:\USABLE\2013 Commitment Research Study\Questionnaire Links) Then leave the room.]

Are you…

39