Attachment E
NTEWS Interview Protocol_Licenses and Certifications
NTEWS Round 1 Cognitive Interview Protocol
License and/or Certifications
NOTE: Use protocol for participants who indicated they did not have a vocational certificate but did have a license, a certification, or both.
Research Questions:
Do the credential questions accurately capture each type of credential?
Are respondents able to understand and answer the credential questions accurately as they are worded?
Are respondents able to understand and answer all other questions accurately as they are worded?
Are respondents able to follow the complex skip patterns in the questionnaire as if it was a paper-and-pencil instrument?
How do respondents react to the ordering of the questions and the flow of the questionnaire?
Materials:
Answers to recruitment credential screening questionnaire to verify educational attainment prior to beginning the interview
Protocol
Video-conferencing set-up
General Probing:
This protocol contains both retrospective and concurrent probes for the questions that have been identified as important to probe or that have been identified as potentially problematic prior to testing. Some questions not previously identified may prove to be problematic for respondents during testing. In these instances, the interviewer will employ general probes to understand the problem and follow-up probes that are specific to the difficulty the interviewer has observed.
General Probes:
In your own words, what do you think this question is asking?
If I were not here, how would you answer this question?
You mentioned _______. Can you tell me more about that?
How did you come up with your answer?
What were you including in that answer? What were you excluding?
EMPLOYMENT STATUS (Q1-Q5)
General Observations and Probing:
Note any problems with any questions/response options/response.
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondents follow the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
CURRENT EMPLOYMENT (Q6-Q22)
General Observations and Probing:
Note any problems with any questions/response options/response.
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondents follow the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
Question-Specific Observations Q6 – Q12:
Note how respondents with more than one job answer these questions:
Note for later questions if respondents maintain focus on main job
Note if respondents have problems with the weeks and hours worked.
Question-Specific Observations Q13 – Q22:
Note if respondents need to return to Q12 to remember their answer.
Note if respondent with more than one job seem to be switching their frame of reference.
Note how the self-employed answer these questions
Q6 – Q12:
If R has more than one job (Q6 > 1)
You indicated you have ____ jobs. Tell me about those jobs. (What kind jobs, how many hours at each job, etc.)
Q7. 35 Hours/week
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any response potential errors for Rs with more than one part-time job.)
(Probe any other potential response errors or difficulties)
Q8. Work Every Week
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or confusion about the instructions.)
(Probe any other potential response errors or difficulties)
Q10. Hours Worked:
In your own words, what do you think this question was asking?
How did you come up with your answer? (Find out how they added up the hours)
What were you thinking of when you were answering this question? (Make sure R is not reporting the clock hours they worked)
(Probe any potential response errors or difficulties)
Q12. Main Job:
In your own words, what do you think this question was asking?
How did you come up with your answer? (Find out how they determined what job they worked the most hours and how many hours they work at each job)
In your own words, what does ‘main job” mean?
(If R appears to have made an error in choosing the job worked the most hours) Did you notice this instruction (point to instruction)?
(If no) What changes, if any, could we make to the question to make it clearer?
(Probe any other potential response errors or difficulties)
All other respondents (Q6=1)
Q7. 35 Hours/week
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties for Rs with part-time job/seasonal jobs.)
(Probe any other potential response errors or difficulties)
Q8. Work Every Week
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties for Rs with part-time job/seasonal jobs.)
(Probe any potential response errors or confusion about the instructions.)
(Probe any other potential response errors or difficulties)
Q10. Hours Worked:
In your own words, what do you think this question was asking?
How did you come up with your answer? (Find out how they added up the hours)
What were you thinking of when you were answering this question? (Make sure R is not reporting the clock hours they worked)
(Probe any potential response errors or difficulties)
Q12. Main Job:
In your own words, what do you think this question was asking?
In your own words, what does “main job” mean?
(Probe any potential response errors or difficulties)
Q13 – Q22:
If R is self-employed (Q16 = Yes)
Q17. Type of Self-employment
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
Q21. 2-digit NAICS
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
All other respondents (Q16 = No)
Q19. Type of Employer
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
Q21. 2-digit NAICS
In your own words, what did you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
LICENSES FOR WORK (Q23-Q36)
General Observations and Probing:
Note any problems with any questions/response options/response
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondents follow the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
Question-Specific Observations Q23 – Q36:
Note how respondents respond to probes about licenses. This information may be needed in the case of response errors/confusion between licenses and certifications.
Note if respondents need to return to Q12 or Q26 to remember their answers.
Note if respondents with more than one license seem to be switching their frame of reference.
Q23 – Q36:
If R reports work-related license (Q23 = Yes AND Q25 = Yes)
You indicated that you have a license. Can you tell me about that license?
(Have R give a brief description, even if some of what they say is redundant with some of the questions)
Q23. Have License
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does the term “currently active” mean to you?
(Probe if R seems to have included a license that is not currently active based on the screener and their description above).
(If there are obvious response errors, such as business licenses or driver’s licenses) Did you notice these examples? Did you notice this instruction (point to instruction)?
What changes, if any, could we make to the question to make it clearer?
(Probe any other potential errors based on what R reported during screening and the description of their license(s) above.)
Q24. License Count
In your own words, what do you think this question was asking?
How did you come up with your answer?
(Probe any potential errors based on what R reported during screening and the description of their license(s) above.)
(If R seems confused by the instruction) In your own words, what does this instruction mean to you?
What changes, if any, could we make to make it clearer?
(Probe any other potential response errors or difficulties)
Q25. Work-related
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does “work-related reasons” mean to you?
(Probe any potential response errors or difficulties)
Q27. Subject Area
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does “primary subject area” mean to you?
What does “field of study” mean to you?
(Probe any potential response errors or difficulties)
Q29 Legally Required
In your own words, what do you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
Q30. Who Issued
In your own words, what do you think this question was asking?
How did you come up with your answer?
(If not obvious from R’s description above) Can you tell me the name of the (agency/professional assoc/business) that issued it?
(Probe any potential response errors or difficulties)
Q34. Use License
In your own word, what did you think this question was asking?
You answered _____ . Tell me more about that. (Find out how R uses the license.)
I’d like to know about why you decided to get this license. Can you tell me about that?
(If not obvious) Did you get this license for this job?
(if yes) Did you already have this license before you got this job or did you get it after you got the job?
(if no) Did you get it for a previous job? Some other reason?
(Probe any potential response errors or difficulties)
Q35. Financial Support
In your own words, what did you think this question was asking?
How did you come up with your answer?
(If R indicated employer support) You indicated that had employer support or reimbursement. Can you tell me about that?
(If not obvious) How much support did your employer provide? Did they pay for some of the training? All of the training?
(If not obvious) Did your employer provide anything other than financial support? (Find out if employer gave support through time to complete the training during work hours or any other non-monetary support)?
(If R indicated loans, grants, scholarships) You indicated that you received support from loans, grants, or scholarships. Can you tell me about that? (Find out what kind of support and from what source)
(If R indicated some other support) You indicated that you had some other support. Can you tell me about that? (Find out what the support is and if there were any response errors)
(Probe any other potential response errors or difficulties)
If R reports non-work-related license (Q23 = Yes AND Q25 = No)
You indicated that you have a license. Can you tell me about that license?
(Have R give a brief description, even if some of what they say is redundant with some of the questions)
Q23. Have License
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does the term “currently active” mean to you?
(Probe if R seems to have included a license that is not currently active based on the screener and their description above).
(If there are obvious response errors, such as business licenses or driver’s licenses) Did you notice these examples? Did you notice this instruction?
What changes, if any, could we make to the question to make it clearer?
(Probe any other potential errors based on what R reported during screening and the description of their license(s) above.)
Q24. License Count
In your own words, what do you think this question was asking?
How did you come up with your answer?
(Probe any potential errors based on what R reported during screening and the description of their license(s) above.)
(If R seems confused by the instruction) In your own words, what does this instruction mean to you?
What changes, if any, could we make to make it clearer?
(Probe any other potential response errors or difficulties)
Q25. Work-related
In your own words, what do you think this question was asking?
What does “work-related reasons” mean to you?
You answered _____ . Tell me more about that. (Find out if/how R uses the license.)
I’d like to know about why you decided to get this license. Can you tell me about that?
(Probe to make sure that this license was not for work-related reason.)
(If not obvious) Did you get this license for a previous job? Some other reason?
What kind of support, if any did you have to pay for this license?
(If not obvious/work-related) Did a previous employer help pay for some of the costs?
(If yes) How much of the cost did the employers pay?
(If not obvious) Did your employer provide anything other than financial support? (Find out if employer gave support through time to complete the training during work hours or any other non-monetary support)?
(Probe any other potential response errors or difficulties)
All other respondents (Q23 = No)
Q23 Have License
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does the term “currently active” mean to you?
Have you had any licenses in the past that are no longer currently active?
(Probe any potential response errors to find out why R did not report license that is currently active?)
(Probe any other potential response errors or difficulties)
CERTIFICATIONS FOR WORK (Q37–Q50)
General Observations and Probing:
Note any problems with any questions/response options/response
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondents follow the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
Question-Specific Observations Q37 – Q50:
Note how respondents respond to probes about certifications.
Refer back to respondents’ responses to the license questions and note any potential problems or potential response errors.
Note if respondents need to return to Q12 or Q40 to remember their answers.
Note if respondents with more than one certification seem to be switching their frame of reference.
Q37 – Q50:
If R reports work-related certification (Q37 = Yes AND Q40 = Yes)
You indicated that you have a certification. Can you tell me about that certification?
(Have R give a brief description, even if some of what they say is redundant with some of the questions)
Q37. Have Certification
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does the term “currently active” mean to you?
(Probe if R seems to have included a certification that is not currently active based on the screener and their description above).
(If there are obvious response errors) Did you notice these examples? Did you notice this instruction?
What changes, if any, could we make to the question to make it clearer?
(Probe any other potential errors based on what R reported during screening, the description of their certification(s) above, and how they answered the license questions)
Q38. Certification Count
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any potential errors based on what R reported during screening and the description of their certification(s) above.)
(If R seems confused by the instruction) In your own words, what does this instruction mean to you?
What changes, if any, could we make to make it clearer?
(Probe any other potential response errors or difficulties)
Q39. Work-related
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does “work-related reasons” mean to you?
(Probe any potential response errors or difficulties)
Q41. Subject Area
In your own words, what do you think this question was asking?
How did you come up with your answer?
What does “primary subject area” mean to you?
What does “field of study” mean to you?
(Probe any potential response errors or difficulties)
Q43. Legally Required
In your own words, what do you think this question was asking?
How did you come up with your answer?
(Probe any potential response errors or difficulties)
Q44. Who Issued
In your own words, what do you think this question was asking?
How did you come up with your answer?
(If not obvious from Rs description above) Can you tell me the name of the (agency/professional assoc/business) that issued it?
(Probe any potential response errors or difficulties)
Q48. Use Certification
In your own word, what did you think this question was asking?
You answered _____. Tell me more about that. (Find out how R uses the certification.)
I’d like to know about why you decided to get this certification. Can you tell me about that?
(If not obvious) Did you get this certification for this job?
(if yes) Did you already have this certification before you got this job or did you get it after you got the job?
(if no) Did you get it for a previous job? Some other reason?
(Probe any other potential response errors or difficulties)
Q49 Financial Support
In your own words, what did you think this question was asking?
How did you come up with your answer?
(If R indicated employer support) You indicated that had employer support or reimbursement. Can you tell me about that?
(If not obvious) How much support did your employer provide? Did they pay for some of the training? All of the training?
(If not obvious) Did your employer provide anything other than financial support? (Find out if employer gave support through time to complete the training during work hours or any other non-monetary support)?
(If R indicated loans, grants, scholarships) You indicated that you received support from loans, grants, or scholarships. Can you tell me about that? (Find out what kind of support and from what source)
(If R indicated some other support) You indicated that you had some other support. Can you tell me about that? (Find out what the support is and if there were any response errors)
(Probe any other potential response errors or difficulties)
If R reports non-work-related certification (Q37 = Yes AND Q39 = No)
You indicated that you have a certification. Can you tell me about that certification?
(Have R give a brief description, even if some of what they say is redundant with some of the questions)
Q37. Have Certification
In your own words, what do you think this question was asking?
How did you come up with your answers?
(If there are obvious response errors) Did you notice these examples? Did you notice this instruction?
What changes, if any, could we make to the question to make it clearer?
(Probe any other potential errors based on what R reported during screening and the description of their certification(s) above.)
Q38. Certification Count
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any potential errors based on what R reported during screening and the description of their certification(s) above.)
(If Rs seem confused by the instruction) In your own words, what does this instruction mean to you?
What changes, if any, could we make to make it clearer?
(Probe any other potential response errors or difficulties)
Q39. Work-related
In your own words, what do you think this question was asking?
What does “work-related reasons” mean to you?
You answered _____ . Tell me more about that. (Find out if/how R uses the certification.)
I’d like to know about why you decided to get this certification. Can you tell me about that?
(If not obvious) Did you get this certification for a previous job? Some other reason?
(Probe to make sure that this certification was not for work-related reason.)
What kind of support, if any did you have to pay for this certification?
(If not obvious/work related) Did an employer help pay for some of the costs?
(If yes) How much of the cost did the employers pay?
(If not obvious) Did your employer provide anything other than financial support? (Find out if employer gave support through time to complete the training during work hours or any other non-monetary support)?
(Probe any other potential response errors or difficulties)
All other respondents (Q37 = No)
Q37. Have Certification
In your own words, what do you think this question was asking?
How did you come up with your answers?
Have you had any certifications in the past that are no longer currently active?
(Probe any potential response errors to find out why R did not report certification)
(Probe any other potential response errors or difficulties)
WORK EXPERIENCE PROGRAMS (Q51–Q59)
General Observations and Probing:
Note any problems with any questions/response options/response
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondent follows the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
Question-Specific Observations Q51 – Q59:
Note if respondents need to return to Q12 to remember their answers.
Note if respondents seem to be switching their frame of reference.
Make special note of the complex skip patterns for the response options in these questions.
Q51 – Q59:
If R reports completing a WEP (Q51 = Yes)
You indicated that you completed a WEP. Can you tell me about that?
(Probe to find more about what kind of WEP it was, if it was part of college/school, etc.)
Q51. Complete WEP
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does the term “work experience program” mean to you?
(Probe any potential errors or difficulties)
Q53. Related to Main Job
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does “related to your main job” mean to you?
Can you tell me what kind of work you were doing for this WEP? (Find out if the work they were doing is similar to their main job.)
(Probe any potential errors or difficulties)
Q54. Use Skill and Knowledge
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any potential errors or difficulties)
Q56. WEP Wage
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does “wage’ mean to you?
Can you tell me a little bit about how you were paid? Was it hourly? Was it a salary? Something else?
(If R has any difficulty selecting an answer, probe that difficulty. Find out if it is because more than one option applies, if it because the response option is not there, they don’t know the answer, or some other reason).
(Probe any other potential errors or difficulties)
Q59. Formal Training – Mentor
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does “formal training” mean to you?
(If no) Can you give some examples of formal training from a coworker or mentor?
Did you work closely with other coworkers or supervisors during your WEP? Can you tell me about that? (Find out if they did receive mentoring but did not report it.)
(Probe any other potential errors or difficulties)
Why R Got Credential
I’d like to know about why you decided to get this certification. Can you tell me about that?
(If not obvious) Did you get this certification for a previous job? Some other reason?
All other Rs (Q51 = No)
Q51. Complete WEP
In your own words, what do you think this question was asking?
How did you come up with your answers?
What does the term “work experience program” mean to you?
Can you think of some examples of the types of work experience programs people may participate in?
(Probe any potential errors or difficulties)
LANGUAGE PROFICIENCY QUESTIONS (Q60-Q62)
General Observations and Probing:
Note any problems with any questions/response options/response
Note if respondents seem to read the examples. Are they helpful or problematic?
Note whether respondents follow the skip pattern(s)
In general, probe these questions retrospectively, unless the respondent is stuck.
For questions without question-specific probes, probe only if respondents appear to have problems with these questions or the skip patterns.
Use general probes and any necessary follow-ups as needed.
Question-Specific Observations Q60 – Q62:
Note if respondents need to return to Q12 to remember their answers.
Note if respondents have any problems with the response options
All Rs
Q60. Native Speaker
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any other potential errors or difficulties)
Q61. English for Job
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any difficulties in understanding the response options)
(Probe any difficulties in selecting a response option)
(Probe any other potential errors or difficulties)
Q62. Spanish for Job
In your own words, what do you think this question was asking?
How did you come up with your answers?
(Probe any difficulties in understanding the response options)
(Probe any difficulties in selecting a response option)
(Probe any other potential errors or difficulties)
WRAP UP
Overall, what did you think about this survey?
Are there any question that you think might be missing? Any that we don’t need?
Other than everything we have already talked about, is there any other feedback you would like to give us?
Wrap up the interview and Thank the respondent for their time.
NTEWS Interview Protocol_Licenses and Certifications
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Beck, Jennifer L |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |