Cognitive Testing of Job Corps' new Student Experience Assessment (SEA) Survey

Student Experience Assessment

Cognitive Testing Directions for Interviewer

Before you begin:

1. Read/explain to the student the “Purpose” and “Confirm participation” from the “Instructions for Students” document. At any point, if the student indicates they do not want to participate, say “thank you” and follow the instructions we’ve been given by the center for where they go when they’re finished with the interview. If at any point the student refuses to answer questions or seems upset ask them if they want to continue and stop if they do not.

2. Place a copy of the survey in front of the student along with a pencil. The student can follow along while we ask the questions and provide the potential answers. The student may NOT keep the survey. Tell them not to write their name on it. Keep track if the student makes any notes on the survey, circles or underlines any words, etc. as this will be used for our analysis.

3. Read the student the “Ground Rules” and “What we need you to do” from the “Instructions for Students” document and answer any questions they have.

4. Listen for and record information on two levels of understanding:

First, as written, does the student understand the question/text/words for both the survey questions and the answer choices? For example: “I don’t know what this word means.” Or “Are you asking about me or my friends?”

Second, is the question, and the way it is written, appropriate – is it relevant, appropriate to the student’s life or center? For example, “This never happens on center.”

5. Record all notes for each survey question on/in _____________________________. Be specific to what the student is saying – capture specific language if possible, do not paraphrase what the student has said.

6. Pay attention to body language: If the student is shaking their head “no” while choosing an answer that means “yes”, you’ll need to follow-up. Or, if the student seems reluctant to answer a question or seems uncomfortable, be aware and follow-up where appropriate.

7. Should the student divulge information to you of a reportable nature, such as reporting they have been sexually assaulted on center, please see the “Reportable Events” document for further instruction.

For each question survey question:

1. Read the number and the question to the student. Read the possible answers to the student.

2. If the student is able to answer or chooses an answer choice without comment, ask the FIRST Follow-up Question below. If the student cannot answer, takes too long, or says “I don’t know”, begins asking questions, or offers alternative answer choices, start with the SECOND Follow-up Question below.

• FIRST Follow-up Question: Was that question easy or hard to answer?

  • If “easy” check the Status box as OK and move on to the possible answers for that question. HOWEVER,

    • If the student hesitates, or it seems like they took a long time to understand or answer, move on to SECOND Follow-up Question.

• If the student asks any questions about the question itself or the possible answers, move on to SECOND Follow-up Question.

• Ask about the answer choices: “Was it easy or hard to choose an answer choice?”

  • If “easy” check the Status box as OK and move on to the possible answers to that question.

    • If the student hesitates, or it seems like they took a long time to understand or answer, move on to SECOND Follow-up Question.

• If the student asks any questions about the question itself or the possible answers, move on to SECOND Follow-up Question.

• SECOND Follow-up Question: “Why was it hard to answer?”

  • LISTEN CAREFULLY. Take notes on exactly what the student says, such as “I don’t know what this word means”, “I’m not sure if you’re asking about me or my friends”, “Are you asking me about last month or my whole time here”…DO NOT PARAPHRASE what the student has said.

• If the student said that they do not understand a word or multiple words in the question, circle the words on your survey testing sheet.

• Additional Follow-up Questions Scenarios:

  • The student answers the survey question but the response seems odd or they add commentary about the question or the answer choices. Examples: “Well that happens all the time but I’ll choose 1-2 times.” -- “That question is just a joke!” -- “We don’t have that one at our center.” -- “The better answer would be (something we’re not offering as a choice).” OR the student is shaking their head “no” while choosing an answer that would mean “yes”.

    • Ask:

      • What made you say that?

      • Why did you respond that way?

      • What word would you use?

      • What does that mean to you?

      • Can you repeat the question in your own words?

      • Why would that be a better answer choice?

3