Appendix B - NAEP 2015 - ECLS-K 2011 Cog Lab Report

Appendix B - NAEP 2015 - ECLS-K 2011 Cog Lab Report.pdf

National Assessment of Education Progress (NAEP) 2014-2016 System Clearance

Appendix B - NAEP 2015 - ECLS-K 2011 Cog Lab Report

OMB: 1850-0790

Document [pdf]

Download: pdf | pdf

Appendix B

Cognitive Interview Reports
For
NAEP-ECLS-K:2011 Study

NAEP Item Development (ID)
Survey Questionnaires
Cognitive Interview Report

2015 NAEP-ECLS Special Study Student
Questionnaires
Grades 4 and 8

Deliverable in response to ID Task 9.2
Submitted: July 11, 2014

2015 NAEP-ECLS Special Study Cognitive Interview Report1
TABLE OF CONTENTS

EXECUTIVE SUMMARY ..................................................................................................................................................3
STUDY RATIONALE ........................................................................................................................................................9
METHODOLOGY ............................................................................................................................................................9
DATA COLLECTION PROCESS .......................................................................................................................................10
RESULTS FROM THE COGNITIVE INTERVIEWS ............................................................................................................12
SUMMARY ...................................................................................................................................................................45
APPENDICES (A-H) .......................................................................................................................................................46
APPENDIX I (PROTOCOL) .............................................................................................................................................56

This document was prepared by ETS in collaboration with CRP, Inc. Contact Debby Almonte (dalmonte@ets.org) and Jonas Bertling
(jbertling@ets.org), NAEP Survey Questionnaires Team, ETS.

EXECUTIVE SUMMARY
In 2015, one goal of the National Assessment of Educational Progress (NAEP) is to develop better measures of SocioEconomic Status including measurement of parent occupation. A special study will focus on linking the NAEP and ECLS-K
assessment in 2015 to validate 4th grade student responses to SES-related questions by comparing to parent responses
from ECLS-K. As part of NAEP’s item development process for the grade 4 student questionnaire for this special study,
newly developed survey questions were pretested (i.e., cognitive interviews were conducted) with a small number of
students before being administered in the 2015 Special Study. The main objectives of this cognitive interview study
were:
1. Identify whether the proposed items can be administered to 4th grade students or whether major issues with
item and answer choice wording warrant revisions pre data collection;
2. Identify whether any of the job titles used in the multiple choice items need to be simplified or revised;
3. Identify which of three alternative versions of a newly created multiple choice item measuring parental
occupation is most suitable for inclusion in the 2015 questionnaire. This included exploring whether students
preferred a particular answer choice layout for the job title item (e.g., grouping job titles by prestige, grouping
job titles by category such as “medical worker”, or listing job titles alphabetically).
Main Findings and Recommendations
In total nine cognitive interviews were conducted with 3rd and 4th grade students (see Table 2 for demographic
information).2
Regarding the first objective, we conclude that the majority of students found all 12 items (see Table 1) very easy or easy
to understand. Results do not indicate that any item needs to be dropped or substantially revised before taking it to the
2015 administration. At the same time, several smaller issues were identified that should be addressed to further
improve the items before the 2015 administration. Main findings and item recommendations are discussed below.
Equivalent mother and father items are discussed jointly.
Table 1. Overview of Recommendations
Item
Sequence
1

Item Wording

Item answer choices

Recommendation

How far in school did your
mother go?

Keep.

How far in school did your
father go?

3
4

Does your mother work?
What does your mother do?
(for example, She’s a lawyer;
She teaches 1st grade; She
cooks in a restaurant; She
takes care of the home and
family)
What does your mother do
for a living?

She did not finish high school. / She graduated
from high school. / She had some education after
high school. / She graduated from college. / I
don’t know.
He did not finish high school. / He graduated from
high school. / He had some education after high
school. / He graduated from college. / I don’t
know.
Yes / No / I don’t know.
N/A – Open ended item.

Accountant / Artist / Athlete / Carpenter / Cashier
/ Cleaner / Clerk / College professor /

Revise stem (see text
below for details);

5 (version
a), 6
2

Keep.

Keep.
Revise stem (see text
below for details).

Please note, one participant mentioned not knowing her father therefore they only provided responses for the mother items.

(version
b), 7
(version c)

8
9

10
(version
a), 11
(version
b), 12
(version c)

Does your father work?
What does your father do?
(for example, He’s a lawyer;
He teaches 1st grade; He
cooks in a restaurant; He
takes care of the home and
family)
What does your father do for
a living?

Construction worker / Cook / Customer service /
Day care worker / Dental hygienist / Dentist /
Designer / Doctor / Education administrator /
Electrician / Engineer / Executive / Factory worker
/ Farmer / Firefighter / Grounds worker / Health
aide / Helper / Housekeeper / Janitor / Lawyer /
Librarian / Manager / Marketing / Mechanic /
Medical technician / Military / Nurse / Office
support / Painter / Pharmacist / Police /
Receptionist / Religious worker / Repair
technician / Salesperson / Scientist / Secretary /
Security guard / Server / Social worker / Software
developer / Stocker / Supervisor / Surgeon /
Surveyor / Taxi driver / Teacher / Therapist /
Transportation / Truck driver / Veterinarian /
Writer (See pages 23, 27, and 35 to reference
answer choice layout)
Yes / No / I don’t know.
N/A – Open ended item.

administer version c.

Accountant / Artist / Athlete / Carpenter / Cashier
/ Cleaner / Clerk / College professor /
Construction worker / Cook / Customer service /
Day care worker / Dental hygienist / Dentist /
Designer / Doctor / Education administrator /
Electrician / Engineer / Executive / Factory worker
/ Farmer / Firefighter / Grounds worker / Health
aide / Helper / Housekeeper / Janitor / Lawyer /
Librarian / Manager / Marketing / Mechanic /
Medical technician / Military / Nurse / Office
support / Painter / Pharmacist / Police /
Receptionist / Religious worker / Repair
technician / Salesperson / Scientist / Secretary /
Security guard / Server / Social worker / Software
developer / Stocker / Supervisor / Surgeon /
Surveyor / Taxi driver / Teacher / Therapist /
Transportation / Truck driver / Veterinarian /
Writer (See pages 23, 27, and 35 to reference
answer choice layout)

Revise stem (see text
below for details);
administer version c.

Keep.
Revise stem (see text
below for details).

Items 1 and 2




Eight out of nine (8 out of 9) participants were able to describe the intent of these items.
All 9 participants were able to indicate their mother and/or father’s educational achievement.
Seven out of nine (7 out of 9) participants found these items very easy or easy to understand. Only 2 participants
found this item hard to understand.

Items 1 and 2- RECOMMENDATION:
No revisions suggested; keep item as is.
Items 3 and 8



All 9 participants were able to describe the intent of these items, and found these items very easy or easy to
understand.
All 9 participants were able to indicate whether their mother/father worked, and were able to specify whether
their mother/father worked full time or part time.

Items 3 and 8- RECOMMENDATION:
No revisions suggested; keep item as is.
Items 4 and 9





Only 3 out 9 participants were able to describe the intent of these items using their own words; the other
participants repeated the question.
When probed about their decision making process, participants stated they decided what to write by thinking
about what their mother/father does. One out of nine (1 out of 9) participants provided a job title as their
response when they were asked “what does your mother do?” (e.g., “she is a writer). No participants provided a
job title as their response when they were asked the equivalent father item. The other participants either
described what their mother/father did at home and/or work (e.g., she cooks, cleans, and calls me to put my
clothes away, “he does stuff that is nice for our family,” “she does interviews and works on the computer”), or
mentioned the mother’s/father’s job location or company name (e.g., “he works at ATF,” “she works at a nail
salon”).
Despite inconsistencies as to how participants responded to this item, 8 out of 9 participants found these items
very easy or easy to understand.

Items 4 and 9 RECOMMENDATION:
Suggest revising the sequence and wording of this item so that it follows the multiple choice item and reads:
What does your [mother/father] do at work?
The intent of this question would be ask the participant to describe what their mother/father does at work, and to
validate the response the participant provided for the multiple choice item. Slightly changing the intent of this question
seems necessary given that participants did not understand that this item and the multiple-choice item were asking the
“same thing.”

Items 5, 6, 7, 10, 11, and 12















When asked describe the intent of the time using their own words, 5 out of 9 participants repeated the
question, and 1 participant mentioned that the item is “the exact same thing as the one before” (i.e., the openended question). Two out of nine (2 out of 9) participants used job titles in the answer choices when describing
the intent of this item.
When asked about their decision making process, 2 out 9 participants mentioned thinking about what their
mother/father does at work. One participant mentioned thinking about what their father does at work and at
home (e.g., engineer and cooking), and other participant mentioned thinking about their father in general (e.g.,
he’s a helper, he cooks, he’s health, her works a lot and he does a lot of stuff.).
Six out of nine (6 out of 9) participants either selected more than one response or inquired as to whether they
could select more than one response.
No participant preferred Version A of this item (i.e., group job titles by prestige; see Figure 1). Five out of nine (5
out of 9) participants preferred Version B of this item (i.e., group job titles by category), and 4 out of 9
participants preferred Version C of this item (i.e., list job titles alphabetically).
While half of the participants preferred Version B of the item, several issues with the category names (i.e.,
medical worker, office worker, equipment worker, education worker, specialized worker, helps people) were
identified during the cognitive interviews.
o Only 5 out of 9 participants understood the officer worker and specialized worker categories.
o Four out of five (4 out of 5) participants understood the equipment worker category when the item
referenced their mother, but no participant (0 out of 4) understood this category when the item
referenced their father.
When participants were asked whether they could “see a pattern that would explain why the options are
ordered like this” in Version B only 3 participants mentioned that that the job titles where in categories. The
other participants either did not respond to the probe or provided vague responses (e.g., “list is by subtitles,”
“I see a pattern where jobs are supposed to be”).
When participants were asked whether they could “see a pattern that would explain why the options are
ordered like this” in Version C 7 out of 9 participants mentioned that the job titles were in “ABC order” or
“alphabetical;” two participants did not provide a response.
As shown in Figure 2, there are a few job titles that students indicated having trouble with. Job titles where 5
or more participants indicated having difficultly where flagged as problematic. The following job titles were
flagged as problematic:
o Carpenter, Education administrator, Mechanic, Office support, Software developer, Marketing, Surveyor,
Secretary, Stocker, Clerk, and Social worker.
While the data suggests that these job titles should be deleted we suggest keeping all job tiles. The rationale
for this recommendation is that students would recognize these job titles if their parent does in fact have that
job.

Items 5-7 and 10-12 RECOMMENDATION:
Consider revising the items so that it reads “What does your [mother/father] do for work.” Additionally, consider adding
inline directions consistent with NAEP practices to specify whether a participant can choose more than one answer.
While approximately an equal number of participants preferred Version B (N=5) and Version C (N=4), we suggest
administering Version C given that many participants experienced difficultly understanding 3 out of the 6 grouping
categories in Version B.

Number of Participants

6
5
4
3
2
1
0
Version A

Version B

Version C

Version Preference

Figure 1. Number of participants who preferred version a, b, and c of the item “What does your [mother/father] do for a
living?”

Number of participants who did not experience difficulty with the job title

Number of participants who experienced difficulty with the job title

Social worker
Clerk
Stocker
Secretary
Surveyor
Marketing
Software developer
Office support
Mechanic
Education administrator
Carpenter
Salesperson
Religious worker
Dental hygienist
Therapist
Supervisor
Server
Pharmacist
Medical technician
Health aide
Factory worker
Executive
Engineer
Customer service
Veterinarian
Transportation
Military
Lawyer
Accountant
Surgeon
Repair technician
Painter
Nurse
Housekeeper
Firefighter
Farmer
Electrician
Construction worker
College professor
Cashier
Writer
Truck driver
Teacher
Taxi driver
Security guard
Scientist
Receptionist
Police
Manager
Librarian
Janitor
Helper
Grounds worker
Doctor
Designer
Dentist
Day care worker
Cook
Cleaner
Athlete
Artist
0

Figure 2. Number participants who experienced difficulty with the job titles in the item “What does your [mother/father]
do for a living?”

STUDY RATIONALE
The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at
grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, U.S. history, civics, geography,
economics, and the arts. NAEP occasionally administers special studies, such as the NIES (National Indian Education
Study), National Validity Studies (NVS) (Computer Familiarity and Accessibility Study), and ECLS (Early Childhood
Longitudinal Study). NAEP is administered by the National Center for Educational Statistics (NCES), part of the Institute
for Education Sciences (IES), in the U.S. Department of Education. NAEP’s primary purpose is to assess student
achievement in the various subject areas and to also collect background questionnaire (i.e., non-cognitive) data to
provide context for the reporting and interpretation of assessment results).
In addition to assessing student achievement, one goal in 2015 is to link NAEP and the ECLS-K: 2011 in order to provide
additional contextual information. In line with these efforts, a new questionnaire namely, the NAEP-ECLS survey
questionnaire, was created to validate student responses to SES-related questions by comparing to parent responses
from NAEP-ECLS. As part of NAEP’s item development process for the NAEP-ECLS survey questionnaires, a portion of
survey questions are pretested on a small number of students before they are administered in an operational
assessment. These pretest activities include cognitive interviews. Educational Testing Service (ETS), the NAEP survey
questionnaires item developer, is the lead contractor for these pretesting activities. An overview of the survey
questionnaires is presented below. In addition to assessing subject-area achievement, NAEP collects background
questionnaire data to provide context for the reporting and interpretation of assessment results. NAEP questionnaires
serve to fulfill reporting requirements of federal legislation and to provide a context for reporting student performance.3

METHODOLOGY
Methodology used in this study was the administration of cognitive interviews. In cognitive interviews (often referred to
as a cognitive laboratory study or cog lab), an interviewer administers a structured protocol in a one-on-one interview
using two methods: think-aloud interviewing and verbal probing techniques. With think-aloud interviewing, respondents
are explicitly instructed to "think aloud" (i.e., describe what they are thinking) as they determine their answers to
questions or tasks. The respondent reads each question, and then the interviewer records the cognitive processes that
the respondent describes in arriving at an answer to the question. With verbal probing techniques, the interviewer asks
probing questions, as necessary, to clarify points that are not evident from the “think-aloud” process. These probes
might include, for example, asking the respondent to rephrase the question in his or her own words or assess whether
the response categories for multiple-choice questions are relevant. A verbal probing technique was used in the current
NAEP-ECLS cognitive interview study. Given that cognitive interview studies are largely observational, the data collected
were mainly verbal reports in response to probes, in addition to volunteered comments. In this study each student
received a different sequence of items, as outlined in Appendix A.
Sampling and Recruitment Plan
Cognitive interviews are used to study the cognitive processes that respondents use to answer questions in order to
identify problems and limitations with the questions. There is so far no consensus in educational methodology and
practice regarding the minimum or optimal sample size necessary to provide valid results for survey question
development.4 Nonetheless, a sample size of five to fifteen individuals has become the standard for cognitive interviews
when developing items for the general population. Although a sample size of five per cell will likely facilitate the
identification of major problems with an item, nine students were interviewed in the current study in order to record
differing additional responses.

Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).
Almond, P. J., Cameto, R., Johnstone, C. J., Laitusis, C., Lazarus, S., Nagle, K., Parker, C. E., Roach, A. T., & Sato, E. (2009). White
paper: Cognitive interview methods in reading test design and development for alternate assessments based on modified academic
achievement standards (AA-MAS). Dover, NH: Measured Progress and Menlo Park, CA: SRI International.

The questionnaire data presented in this report were elicited from third grade students entering the fourth grade and
fourth grade students entering the fifth grade. Students were recruited from the Metropolitan Washington, D.C. area.
Every effort was made to engage students from a range of demographic groups, that is a mix of gender, race/ethnicity,
socioeconomic background, and urban and suburban students. Interviews were conducted at the corporate offices of
CRP, Incorporated (CRP) in Silver Spring, Maryland and at Shugoll Research, in Alexandria, Virginia to obtain a diverse
sample of students. The frequency of gender, grade, race/ethnicity, locale, and socioeconomic status, of the
participants, is outlined in Table 2. CRP, a subcontractor to ETS, is a minority- and woman-owned professional services
and management consulting firm that provides technical consulting services primarily in the fields of education and
behavioral health.
Table 2. Frequency of gender, grade, race/ethnicity, locale, and socioeconomic status.
Variable
Frequency
Female
5
Male
4
rd
3 Grade
8
4th Grade
1
African American
2
Asian American
3
Caucasian
3
Hispanic/Latino
1
Urban
7
Suburban
2
Rural
0
Low SES
5
High SES
4

CRP staff contacted the parents or legal guardians of students and key informants within the community to recruit
participants for the survey questionnaire interviews (see Appendices B, C, and D). The parents or legal guardians of the
participating students completed consent forms at the time of the interview (see Appendix E). For participants selected
to take part in the study, CRP confirmed the interview date, time, and location with the parents or legal guardians of the
participating students (see Appendix F). All interviews were scheduled for 60 minutes which included welcoming the
students at the beginning of the interview and debriefing them at the end of the session. Audio recordings of the
cognitive interview sessions were captured for the researchers of the NAEP-ECLS study to review and summarize
findings. A thank you letter was sent to the parents or legal guardians (see Appendix G) after the students completed the
cognitive interviews.
DATA COLLECTION PROCESS
Cognitive Interviews
The cognitive interviews were conducted as follows. Participants were first welcomed, introduced to the interviewer and
the observer(s) if in-room observers were present, (i.e., observers were composed of CRP and NCES staff). Participants
were told that they were recruited to help answer questions about how people answer survey items. Parents/legal
guardians were seated in a different room so that student responses would not be influenced by their presence.
Participants were reassured that their participation was voluntary and that their answers may be used only for research
purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law
[Education Sciences Reform Act of 2002, 20 U.S.C §9573].
The interviewer explained to participants that ETS was seeking their feedback on the survey questions, and that their
input was valuable. Participants were then asked to read the NAEP-ECLS survey items, indicate a response for each
survey item, and answer the probes associated with each item. The amount of verbal probing done after each item
10

varied, depending upon the type of items and the goals of the study. The protocol, which contains the welcome script,
the specific survey items included, and the generic and item-specific probes are contained in Appendix I. A digital audio
recording captured students’ verbal responses to the interview. Interviewers also recorded notes, including behaviors
(e.g., the participant appeared confused) and if extra time was needed during a particular part of the task). It should be
noted, interviewers were instructed to make the probes “their own” that is, they could slightly rephrase, if necessary,
but keep the overall meaning of the probe.

Assurance of Confidentiality
Participants were instructed that their participation was voluntary and that their answers will be used only for research
purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law
[Education Sciences Reform Act of 2002, 20 U.S.C §9573]. Written consent was obtained from parents/legal guardians
before interviews were conducted. Participants were assigned a unique student identifier (ID), which was created solely
for data file management and used to keep all participant materials together. The participant ID was not linked to the
participant name in any way or form. The consent forms, which included the participant’s name, were separated from
the participant interview files, secured for the duration of the study, and were destroyed after the final report was
released.
Compensation
A monetary incentive aimed at ensuring participation and motivation on behalf of the participants was provided. For his
or her time and effort, each participating student received as compensation after the cognitive interview, a $25 gift card
(e.g., Visa gift card). In addition, a gift card for $25 was given to the parent/legal guardian to remunerate him or her for
the time involved and to help offset the travel/transportation costs of taking the participating student to and from the
cognitive interview site.

RESULTS FROM THE COGNITIVE INTERVIEWS
Item 1 and Item 2 (Items not mandatory)
1) How far in school did your mother go?
a) She did not finish high school.
b) She graduated from high school.
c) She had some education after high school.
d) She graduated from college.
e) I don’t know.
2) How far in school did your father go?
a) He did not finish high school.
b) He graduated from high school.
c) He had some education after high school.
d) He graduated from college.
e) I don’t know.

All nine (9) participants were able to indicate the educational achievements of their mother; eight (8) of the nine (9)
participants were able to indicate the educational achievements of their father. Verbatim responses follow.3
Can you tell me, in your own words, what the question is asking?
Seven (7) of the participants were able to rephrase the question for the mother; and seven (7) of the participants were
able to rephrase the question for the father. Verbatim responses for the probes for Item 1 and Item 2 follow. One (1)
participant was unable to rephrase the question for the mother and father items.
MOTHER (item 1):
 001: Where did she go, where did she graduate.
 002: How much school my mother finished.
 003: How long did it take my mom to graduate.
 004: What grade did my mom go to in high
school.
 005: No response from participant.
 006: How smart is your mom.
 007: How far my mother went in school, and if
she finished high school or went to college.
 008: How far did my mother go in school.
 009: How far in school did my mom graduate.

FATHER (item 2)
 001: (student indicated she does not know her
father). Question not asked.
 002: How much school my parents finished.
 003: How long did it take my dad to graduate.
 004: How far did my dad go in what grade in
school.
 005: How far did my father go in school.
 006: Is your dad smart or stupid.
 007: How far my father went, like if he passed
college.
 008: How far in school did my father go.
 009: How far in school did my dad graduate.

Notations of explanation are enclosed in parentheses ( ).

Why did you select this choice? How did you know what answer to select?
All of the participants who answered these items were able to explain how they chose the answer. Verbatim
responses follow.

FATHER (item 2):
 001: (Student indicated she does not know her
father). Question not asked.
 002: No response from participant..
 003: He told me he graduated
 004: He finished high school, but did not
complete college.
 005: He did graduate from high school
 006: Because he showed us his yearbooks from
college and we saw his degree.
 007: Because he told me.
 008: Because he graduated from college. I know
because he told me.
 009: Because he only went to college.

MOTHER (item 1):
 001: No response from participant.
 002: No response from participant.
 003: I saw pictures of her graduate.
 004: She passed college.
 005: No response from participant.
 006: No response from participant.
 007: Because I saw her graduate.
 008: Because she graduated from high school.
 009: She did not just go to college, she went to
the university.

Did you find any part of the question confusing? Were there any words that you didn’t know?
Five (5) of the students indicated this question was not confusing for the mother items and six (6) for the father
items. None of the participants indicated that they did not know any words.
MOTHER (item 1):
 001: No response from participant.
 002: No response from participant.
 003: No
 004: No
 005: No response from participant.
 006: No
 007: No
 008: No
 009: Yes. Mother went to a University and not
college. There should be another choice for
university.

FATHER (item 2):
 001: (Student indicated she does not know her
father). Question not asked.
 002: No
 003: No
 004: Yes. He only had a little, not some. It didn't
make sense at first, had to read it a couple of
times.
 005: Yes. Did not recognize answer choice "c".
 006: No
 007: No
 008: No
 009: No. (Student differentiated between
mother going further than "college" and father
only having gone to college.)

Notations of explanation are enclosed in parentheses ( ).

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
Five (5) of the students who answered the mother question and seven (7) of the students who answered the
father question found the answer to be easy to very easy to answer, as indicated in Tables 3 and 4, respectively.
One (1) student indicated that the choice was moderately easy for the mother question. Verbatim responses
follow the tables.3
Table 3. Number of participants who found Item 1 (mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
3
Easy
2
Moderate
1
Hard
0
Very hard
0
No response
3
Total Number of Responses
9
Table 4. Number of participants who found Item 2 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
3
Easy
4
Hard
1
Very hard
0
No Response
1
Total Number of Responses
9

MOTHER (item 1):
 001: No response from participant.
 002: No response from participant.
 003: Easy
 004: Easy
 005: No response from participant.
 006: Easy
 007: Very easy
 008: Very easy
 009: Moderate

FATHER (item 2):
 001: (Student indicated she does not know her
father). Question not asked.
 002: Depends on who you are. For some it's
easy very easy. They were provided with a
detailed description. Hard if they have a vague
idea. Very hard it they don't know.
 003: Easy
 004: Hard
 005: Very Easy
 006: Easy, but for younger kids like my sister,
who is 8, she won't be able to answer this. She
doesn't know what a college degree is and
doesn't understand it like that. So it might be
hard for other kids to answer the question.
Some parents if they didn't graduate might not
want their kids to know.
 007: Very easy
 008: Very easy
 009: Easy

Notations of explanation are enclosed in parentheses ( ).

Item 3 and Item 8
3) Does your mother work?
a) Yes
b) No
c) I don’t know

8) Does your father work?
a) Yes
b) No
c) I don’t know

Can you tell me, in your own words, what the question is asking?
Six (6) of the participants repeated the item identically for the mother question. Only three (3) were able to
rephrase it in their own words. For the father item, four (4) of the participants repeated the original item as
asked. Only four (4) were able to rephrase it in their own words. Verbatim responses follow.3
FATHER (item 8)
 001: (Student indicated she does not know her
father). Question not asked.
 002: Does my father work.
 003: What does my dad do for a living.
 004: If my father works at a job.
 005: Does my father work.
 006: Is your family doing well. Work means that
he has a job.
 007: Does my father work at a job
 008: Umm...does my father work. He works as a
NIH person.
 009: Does my dad work.

MOTHER (item 3):
 001: Does your mother work. Yes. To me it
means when my mom goes whenever when I
wake up in the morning like I want to see my
mom and say good-bye but I can't but I don't see
her because she's gone she always wake up like
7 o'clock 6 everything but I want to see her
when she's off break but she off break at work
not at her house she used to do that but she's
not anymore that's why I want to see my mother
when she get off work she get off like 6 o'clock.
 002: Does my mother work.
 003: It means to me that does she work hard,
where does she works and how does she work.
 004: Is my mom going to work to make money.
 005: Does my mother work.
 006: What does my mom do for a living.
 007: Does my mother work.
 008: Does my mother work.
 009: Does my mother work.

Notations of explanation are enclosed in parentheses ( ).

Why did you select this choice? How did you know what answer to select?
Eight (8) of the participants were able to justify their responses for both the mother and father questions. One (1)
participant was not asked questions pertaining to her father. Verbatim responses follow.3
MOTHER (item 3):
 001: No response from participant.
 002: My mom does work. If not sure, there is no
option, must pick one or the other.
 003: Because I have been to her job a lot of
times.
 004: I know my mom works.
 005: Because she does work.
 006: Because she works.
 007: Because I saw her work before and she tells
me that she works.
 008: Does my mother work.
 009: She does work.

FATHER (item 8):
 001: (Student indicated she does not know her
father). Question not asked.
 002: He does work.
 003: No response from participant.
 004: Because he works and he works out.
 005: Because he works a lot and he tries his best.
 006: Because he has a job.
 007: I saw my dad at work before and he told me
that he works.
 008: He works as a NIH person.
 009: Because he does work.

Did you find any part of the question confusing? Were there any words that you didn’t know?
No students indicated that this item was confusing. However, one (1) student questioned the use of this probe
(006-Father item), indicating this was a “bad” question. Verbatim responses follow.
MOTHER (item 3):
 001: No response from participant.
 002: No
 003: No
 004: No
 005: No
 006: No
 007: No
 008: No
 009: No

FATHER (item 8):
 001: (Student indicated she does not know her
father). Question not asked.
 002: No
 003: No
 004: No
 005: No
 006: This is a bad question. Why do they need
to know if my father works? Because they are
judging you. What else would they be doing?
That's kind of a personal question.
 007: No
 008: No
 009: No

Notations of explanation are enclosed in parentheses ( ).

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
All but one (1) student indicated easy or very easy for both mother and father items as shown in Tables 5 and 6,
respectively. Verbatim responses follow these tables.3
Table 5. Number of participants who found Item 3 (mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
7
Easy
2
Hard
0
Very hard
0
Total Number of Responses
9
Table 6. Number of participants who found Item 8 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
5
Easy
3
Hard
0
Very hard
0
No response
1
Total Number of Responses
9

MOTHER (item 3):
 001: Easy
 002: Very easy. Because I don’t know was
provided.
 003: Very easy. Because I have been to her job a
lot of times
 004: Very easy
 005: Very easy
 006: Very easy
 007: Easy
 008: Very easy
 009: Very easy

FATHER (item 8):
 001: (Student indicated she does not know her
father). Question not asked.
 002: Very Easy
 003: Very easy
 004: Very easy
 005: Very easy
 006: That was easy, but it might be different for
kids who don't see their father that much. My
friend doesn't see his father for a year. It would
be hard for him to answer and it might make him
feel bad.
 007: Very easy
 008: Easy
 009: Easy

Notations of explanation are enclosed in parentheses ( ).

Do you know if your mother is working full time or part time?
Seven (7) of the participants indicated that their mother worked full time, one (1) indicated part time and full
time, and one (1) indicated part time. Verbatim responses follow.3










001: Part time and full time
002: Full time
003: Full time
004: Part time
005: Full time
006: Full time
007: Full time
008: Full time
009: Full time

Do you know if your father is working full time or part time?
Four (4) of the participants indicated full time, two (2) part time, and three (3) participants did not respond to this
probe.










001: (Student indicated she does not know her father). Question not asked.
002: Part time
003: Full time
004: No response from participant.
005: Part time
006: Full time
007: Full time
008: No response from participant.
009: Full time

Notations of explanation are enclosed in parentheses ( ).

Item 4 and Item 7
4) What does your mother do? (for example, She’s a lawyer; She teaches 1st grade; She cooks in a restaurant;
She takes care of the home and family)
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

7) What does your father do? (for example, He’s a lawyer; He teaches 1st grade; He cooks in a restaurant; He
takes care of the home and family)
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

Can you tell me, in your own words, what the question is asking?
For the mother item, six (6) participants repeated the original item. From the other responses it was unclear
whether the three (3) other students were able to rephrase the question in their own words. For the father item,
two (2) participants responded with the original question, five (5) were able to explain the question in their own
words, and two (2) participants did not respond to this question. Verbatim responses follow.3
FATHER (item 7):
 001: (Student indicated she does not know her
father). Question not asked.
 002: No response from participant.
 003: Student indicated that examples were
distracting to the question.
 004: What does my father do.
 005: Explain your own lawyers…that what your
dad does.
 006: What does he do for a living.
 007: What job does my father do.
 008: Because he works at NIH...he helps doctors
to help people.
 009: What does he do for a living.

MOTHER (item 4):
 001: Ask me that my mother cooks cleans my
clothes are dirty when messed up I spill
something on my clothes that I just got. What
does she do. [re-reads]
 002: Umm, what momma's work is. What she
does in it.
 003: I want to write it down. It means…hmm I
don’t really know what it means…yes [nods head
yes]…yes.
 004: What does my mother do.
 005: What my mom does.
 006: What does my mom do.
 007: What does my mother do.
 008: Umm what does my mother do.
 009: What does your mother do.

Notations of explanation are enclosed in parentheses ( ).

How did you decide what to write down? How did you decide how much to write?
The eight (8) participants who answered the mother and father items gave a vague description of what they
decided to write down because they knew what their mother and father did for a living. One (1) participant for
both the mother and father items did not respond to the question. Verbatim responses follow.3
MOTHER (item 4):
 001: She do cooks and do the clothes and I help
with the clothes. She cooks cleans do the
laundry says my name put clothes in right pile.
 002: Based on how much I thought I could fit in
the box. Keeping my answer simple enough.
 003: My mother does interviews and works on
the computer.
 004: My mother works at a nail salon. I looked at
the examples. I know my mom works at a nail
salon.
 005: No response from participant.
 006: Because that is what she does.
 007: Because she told me she does nails.
Because I understood the question.
 008: My mother does nails for other people.
Because she does nails for other people.
 009: She is a writer and does public relations. I
thought I could write both. As a public relations
this is an entrepreneur. She writes articles for
the magazine.

FATHER (item 7):
 001: (Student indicated that she does not know
her father). Question not asked .
 002: The lines.
 003: He works for metro. He told me where he
works and it has the logo on his shirt.
 004: He works at a place for building. Since my
dad works at a place that is for building, for a
shorter way to write.
 005: He does stuff that is not mean, he does
stuff that is nice for our family.
 006: It's not asking that much, I just wrote what
my father does.
 007: Because I understood the question.
 008: My father works at NIH. He helps doctors.
 009: He works at ATF. I wanted to explain what
category it belongs in.

Notations of explanation are enclosed in parentheses ( ).

Did you find any part of the question confusing? Were there any words that you didn’t know?
Two (2) participants who answered the mother item and one (1) participant who answered the father item, found
the question confusing. One (1) participant did not respond to the question in each category. Verbatim responses
follow.3
MOTHER (item 4):
 001: Yes it was confusing.
 002: Not really. Not too sure if the examples are
necessary. Because usually if you know what
your mother does, you know what your mother
does. If you don't, then you don't.
 003: Yes. Because it was it was different from
questions that I normally answer people
normally ask.
 004: No response from participant.
 005: No
 006: No
 007: No
 008: No
 009: No

FATHER (item 7):
 001: (Student indicated that she does not know
her father). Question not asked.
 002: No
 003: No. I understand better now. I know each
word goes with its own line. Like housekeepers
help clean, then a helper…I don’t know what a
stocker is.
 004: No
 005: Yes. Because it was very confusing to find
where is, like, I don't understand the words
because it talks about something.
 006: No
 007: No
 008: No
 009: No

What do you think about the examples in the question? Did the examples help you answer the question?
For the mother item, three (3) participants indicated the examples were not helpful. Four (4) participants did not
respond to the probe. For the father item, one (1) participant found the examples helpful, and three (3) of the
participants did not respond. Verbatim responses follow.
 001: (Student indicated that she does not know
MOTHER (item 4):
her father). Question not asked.
 001: umm it would be easy without the example.
 002: No response from participant.
 002: Examples are not necessary.
 003: They are still distracting.
 003: No. Would like to see the questions without
 004: Yes
the examples. The examples are distractions.
 005: Yes, they helped me because they told me
 004: Yes
what your dad does or take care of your family
 005: No response from participant.
or home.
 006: No response from participant.
 006: No response from participant.
 007: No response from participant.
 007: The examples are fine because these are
jobs that fathers do.
 008: Yes
 008: They said for example he's a lawyer, and he
 009: No response from participant.
teaches 1st grade.
 009: Thought the examples were the answer
choices.
3
Notations
of explanation
are enclosed in parentheses ( ).
FATHER
(item
7):

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
As indicated in Tables 7 and 8, seven (7) participants for the mother item found the item easy to very easy, one (1)
found it hard, and one (1) participant did not respond. For the father item, eight (8) participants found the item
easy to very easy to answer. One (1) participant did not respond. Verbatim responses follow.3

Table 7. Number of participants who found Item 4 (mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
4
Easy
4
Hard
0
Very hard
0
No response
1
Total Number of Responses
9
Table 8. Number of participants who found Item 7 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
4
Easy
4
Hard
0
Very hard
0
No response
1
Total Number of Responses
9

MOTHER (item 4):
 001: Easy without examples.
 002: Easy
 003: No response from participant.
 004: Easy
 005: Hard
 006: Easy
 007: Very easy
 008: Very easy
 009: Very easy

FATHER (item 7):
 001: (Student indicated that she does not know
her father). Question not asked.
 002: Easy
 003: Very easy
 004: Easy
 005: Easy
 006: Very easy
 007: Easy
 008: Very easy
 009: Very easy

Notations of explanation are enclosed in parentheses ( ).

Item 5 and Item 10
5) What does your mother do for a living?
Fill in the circle next to the list that most closely describes what she does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

(a) Doctor, dentist, surgeon, veterinarian
(b) College professor, librarian
(c) Teacher
(d) Engineer, surveyor, scientist, lawyer, social worker, religious worker
(e) Nurse, pharmacist, therapist, medical technician, dental hygienist
(f) Artist, athlete, designer, writer
(g) Executive, manager, supervisor, education administrator, accountant, software developer
(h) Office support, secretary, clerk, receptionist, customer service
(i) Mechanic, repair technician, farmer, carpenter, grounds worker, electrician, painter, miner
(j) Salesperson, marketing
(k) Server, cook, cashier, health aide, janitor, day care worker
(l) Police, firefighter, security guard
(m) Factory worker, transportation, truck driver, taxi driver
(n) Housekeeper, cleaner, helper, stocker, construction worker
(o) Military
(p) Other: ______________________

10) What does your father do for a living?
Fill in the circle next to the list that most closely describes what he does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Can you tell me, in your own words, what the question is asking?
This probe was only asked when the item was the first version. One (1) student who answered the mother item
was unable to rephrase, and three (3) participants were not able to rephrase for the father item. Only one (1)
participant did not respond to the question. Verbatim responses follow.3
MOTHER (item 5):
 001: (When asked if she understood the
question, student replied "yes".) They told me
that we was a veterinarian. She works at the VA I
go there sometimes I ask to go there sometimes
I ask to go there she says no she works at two
jobs actually at the VA and she works at the
Giant…the Giant and the pharmacy …she takes
care of the people in the bed. She works for the
government
 002: The exact same thing as the one before. It's
almost the same question. Basically the same
question, just in a list.
 003: What does my mom do for a living
 004: Like if she's an artist or an athlete or a
designer and a writer for like books.

FATHER (item 10):
 005: What does my dad does for a living.
 006: What does he do for a living.
 007: What job does my father do.
 009: What does he do for a living.

As you were reading through the answer choices, what were you thinking about? Did you read all the answers
before making your selection?
For the mother item, two (2) participants made decisions according to the tasks that are included as a part of the
job. Two (2) participants did not respond to the question. For the father item, four (4) of the participants were
able to explain how they made their choices. One (1) participant did not respond to the question. Verbatim
responses follow.
FATHER (item 10):
MOTHER (item 5):
 005: I was thinking about what he does. I picked
one of these because these are the things my
 001: No response from participant.
dad does. He cleans the house, he's a
 002: No response from participant.
housekeeper, he's a helper, he cooks, he's
 003: I was thinking why do they looked mixed
healthy, he works a lot and he does a lot of stuff.
up. I was looking for a veteran I didn’t know If I
 006: No response from participant.
had to circle the bubble…I circled that. I though
doctor would be with the "D's"…[inaudible]
 007: Chose multiple options, engineer and
umm the pattern is.. I think
cooking because that what is he does.
 008: Umm I was thinking do they have his
 004: At the hair salon, she paints designs. She
job…Chose "other" because the list did not have
gives options.
his work.
 009: I was thinking about the government.

Notations of explanation are enclosed in parentheses ( ).

Can you tell me, in your own words, how you understand the words “most closely” in the question?
This probe was only asked when the item was the first version. This probe was only asked once for the mother
and once for the father. Neither of the responses adequately answered the probe.
MOTHER (item 5):
 004: Stuff that is very close to a designer.

FATHER (item 10):
 005: It means look closely at what he does.

Why do you think the different options are presented in this order? Can you see a pattern that would explain
why the options are ordered like this?
For the mother item, three (3) participants found a pattern and one (1) did not. For the father item, three (3)
participants found a pattern, one (1) did not, and one (1) did not answer the question adequately. Verbatim
responses follow.3
FATHER (item 10):
MOTHER (item 5):
 005: I think that he's very like supportive.
 001: Yes they go together in a like say I'm a vet
 006: each option has an alphabetical choice.
in a hospital worked at and I worked at these
 007: They are in categories and groups.
things like I'm a veterinarian I take care of
 008: Like each one is in order…Like police,
people I would give medicine so they feel better,
firefighter, security guard go together.
give them water…yes uh I don't think so but I do
 009: Did not see a pattern.
know they give people medicines... the
veterinarian. She give people this she asks
people do um if they need any more water if
your throat is dry…no.
 002: Doctor, dentist, veterinarian all help people
or animals with their health.
 003: The pattern is each work goes with its own
line. A doctor is close to a surgeon is close to a
dentist, close to a veterinarian they are doctors
it keeps going in the same pattern. A librarian
teachers about books a professor about life
...school. I understand it better because now I
know each word goes with its own line...so each
word is close to the next word. like house keeper
you know it helps clean your house, a cleaner
the same thing it cleans things. A stocker I don't
know what that is and a construction worker
builds things puts things together.
 004: They came in another time. It goes in order.

Notations of explanation are enclosed in parentheses ( ).

Did you find any part of the question confusing? Were there any words that you didn’t know?
One (1) participant found the question confusing, and one (1) did not. For the father item, one (1) participant
found the question confusing, and one (1) did not .Verbatim responses follow.3
MOTHER (item 5):
 001: No response from participant.
 002: No
 003: Yes. (Question was initially confusing.
Participant then indicated that the question was
not confusing after the moderator continued to
probe their understanding of the job titles). Now
I understand better. I know each word goes with
its own line, like doctor, veterinarian, surgeon
are all alike.
 004: No response from participant.

FATHER (item 10):
 005: No response from participant.
 006: Not for me, but for my friends and kids in
general, the answer choices are all bunched up
together. It's confusing.
 007: No response from participant.
 008: No response from participant.
 009: No response from participant.

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
As indicated in Tables 9 and 10, three (3) of the participants found item 5 easy or very easy. One (1) participant
found item 5 of medium difficulty to answer. Four (4) participants found item 10 easy or very easy. One (1)
participant found item 10 hard. Verbatim responses follow the tables.
Table 9. Number of participants who found Item 5
(mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
1
Easy
2
Medium
1
Hard
0
Very hard
0
Total Number of
4
Responses

MOTHER (item 5):
 001: It would be very easy
 002: Medium; wasn't easy to look over (easy to
answer once he found the answer)
 003: Easy
 004: Easy

Table 10. Number of participants who found Item
10 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
1
Easy
3
Hard
1
Very hard
0
Total Number of
5
Responses

FATHER (item 10):
 005: Kind of easy
 006: This one is hard.
 007: Easy
 008: Easy
 009: Very Easy

Notations of explanation are enclosed in parentheses ( ).

Item 6 and Item 11
6) What does your mother do for a living?
Fill in the circle next to the job that most closely describes what she does.
Medical worker
O Doctor
O Dentist
O Surgeon
O Veterinarian
O Nurse
O Pharmacist
O Therapist
O Medical technician
O Dental hygienist
O Health aide
O Other Medical worker
(please specify):
__________

Office worker
O Executive
O Manager
O Supervisor
O Accountant
O Software developer
O Office support
O Secretary
O Clerk
O Receptionist
O Customer service
O Other Office worker
(please specify):
__________

Education worker
O College professor
O Librarian
O Teacher
O Education administrator
O Day care worker
O Other Education worker
(please specify):
__________

Specialized worker
O Engineer
O Surveyor
O Scientist
O Lawyer
O Social worker
O Religious worker
O Artist
O Athlete
O Designer
O Writer
O Salesperson
O Marketing
O Stocker
O Other Specialized worker
(please specify):
__________

Helps people
O Server
O Cook
O Cashier
O Janitor
O Police
O Firefighter
O Security guard
O Housekeeper
O Cleaner
O Helper
O Military
O Other Helps People
(please specify):
__________

11) What does your father do for a living?
Fill in the circle next to the job that most closely describes what he does.
Medical worker
Office worker
O Doctor
O Executive
O Dentist
O Manager
O Surgeon
O Supervisor
O Veterinarian
O Accountant
O Nurse
O Software developer
O Pharmacist
O Office support
O Therapist
O Secretary
O Medical technician
O Clerk
O Dental hygienist
O Receptionist
O Health aide
O Customer service
O Other Medical worker
O Other Office worker
(please specify):
(please specify):
__________
__________

Education worker
O College professor
O Librarian
O Teacher
O Education administrator
O Day care worker
O Other Education worker
(please specify):
__________

Equipment worker
O Mechanic
O Repair technician
O Farmer
O Carpenter
O Grounds worker
O Electrician
O Painter
O Factory worker
O Transportation
O Truck driver
O Taxi driver
O Construction worker
O Other Equipment
worker (please
specify):
__________
Helps people
O Server
O Cook
O Cashier
O Janitor
O Police
O Firefighter
O Security guard
O Housekeeper
O Cleaner
O Helper
O Military
O Other Helps People
(please specify):
__________

Can you tell me, in your own words, what the question is asking?
This probe was only asked when the item was the first version. For the mother item, four (4) of the participants
were able to rephrase the question. For the father item, six (6) of the participants were able to rephrase the item
in their own words. Verbatim responses follow.3

MOTHER (item 6):
 001: Yes she goes to school downtown far far
away I forgot what street it is…it means where
did she go where did she graduated did she
went. .she was 13 years old in I think she was in
8th grade I don't know... graduated high school
which that's where she graduated at... that
means she didn't finish …no...that means you are
getting a job...yes I don't know… oh yeah ..I cross
out d because who graduates from a church... I
was confused I don't know anything about my
mom, where she was at. St. Anthony's.
 002: About the job. What she gets money for.
Housekeeping is work. A different kind of work.
 007: What does my mom do for a living, like
what does she do most of the time.
 009: What does she do.

FATHER (item 11):
 001: Student indicated this version was easier,
however, questions about father were not
asked.
 002: What he does for a living; he does various
jobs at the restaurant.
 003: Where does he work and how does he
work.
 005: Its asking me what does my father does.
 006: What does he do. For a living is how he
makes money, basically the same as "for a job".
 007: What jobs does my father do at home or
anywhere.
 008: What is he doing. What does do in his life. I
think it is the same.

Notations of explanation are enclosed in parentheses ( ).

As you were reading through the answer choices, what were you thinking about? Did you read all the answers
before making your selection?
Two (2) of the participants who answered the mother question were able to describe what they were thinking
about as they made their choice. Four (4) of the participants who answered the father question were able to
describe what they were thinking about. Verbatim responses follow.3

MOTHER (item 6):
 001: No response from participant.
 002: No response from participant.
 003: Looked at the subtitles and looked for what
had military in it.
 004: It was kind of hard the way it’s worded.
Used headings to choose more than one.

FATHER (item 11):
 005: I would think about my dad. (Other
responses): I chose a painter because he paints
stuff. I chose cook because he cooks stuff. I put
housekeeper because… He's a cleaner, he's a
helper, he helps other people. Engineer because
he fix machines. He designs stuff. Day care
worker because he helps a little girl. (Multiple
options): He loves me.
 006: I chose two choices because they are the
same thing but in different categories. My
father works in an office but he's also a
specialized worker because he goes to people's
houses. So he's both.
 007: Multiple Options: Because these are things
that my father usually does, like engineering and
cleaning and helping others at work.
 008: No response from participant.
 009: Still a government. He has two things that
he does. Thinking ATF was more executive.
Made more sense.

Can you tell me, in your own words, how you understand the words “most closely” in the question?
This probe was only asked when the item was the first version. Only three (3) of the nine (9) participants were
asked this probe. Two (2) of the participants were able to determine what they thought “most closely” meant.
Verbatim responses follow.
FATHER (item 11):
 006: Most like.
 008: Like the closest one because he only does
one work.

MOTHER (item 6):
 002: It could be one or more than one. Because
you can't do more than one job for a living.
Because you can be a part time artist and sell
your work, and then you can also be a part time
musician. But you can't really make and sell work
as an artist and travel to concerts and things as a
musician.

Notations of explanation are enclosed in parentheses ( ).

Why do you think the different options are presented in this order? Can you see a pattern that would explain
why the options are ordered like this?
Two (2) of the participants responded that they saw a pattern for the mother item. Two (2) participants
responded that they saw a pattern for the father item. Verbatim responses follow.3
FATHER (item 11):
MOTHER (item 6):
 005: No response from participant..
 001: No response from participant.
 006: Because they are the top 6 categories.
 002: Executive is higher. Higher is listed first.
 007: The options over here, I don't think my
 003: List is by subtitles.
father does these things because I never saw
him do it, and he tells me what he does for a
 004: Because some are more important than
living. I see a pattern where the jobs are
others.
supposed to be.
 008: No [frowns]..no because he helps people
[inaudible]..yeah a little they helped can you
repeat the question.
 009: They are all in categories.

Did you find any part of the question confusing? Were there any words that you didn’t know?
Two (2) participants answered the mother question and one (1) participant answered the father question. The
three (3) participants who answered the probe found the question not to be confusing.
MOTHER (item 6):
 001: No response from participant..
 002: No
 003: No
 004: No response from participant..

FATHER (item 11):
 005: No response from participant.
 006: No response from participant.
 007: No response from participant.
 008: No response from participant.
 009: No

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
As indicated in Tables 11 and 12, three (3) participants found the item easy to very easy. One participant found it
hard. For item 11, four (4) participants found the item easy to very easy to answer. Verbatim responses follow.
Table 11. Number of participants who found Item 6 (mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
1
Easy
2
Hard
1
Very hard
0
Total Number of Responses
4

Notations of explanation are enclosed in parentheses ( ).

Table 12. Number of participants who found Item 11 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
1
Easy
3
Hard
0
Very hard
0
No response
1
Total Number of Responses
5

MOTHER (item 6):
 001: Very easy
 002: Easy
 003: Easy
 004: Hard

FATHER (item 11):
 005: Very easy
 006: No response from participant.
 007: Easy because this time I understood the
question because last time you explained it to
me
 008: Just easy…umm that these were there.
 009: Easy

How would you describe a “medical worker” / “office worker” / “equipment worker” / “education worker” /
“specialized worker” / person who “helps people” does at work?
The number of participants who understood the categories and the number of participants who did not
understand the categories for items 6 and 11, are outlined in Tables 13 and 14. Verbatim responses follow table
13. 3
Table 13. Number of participants who indicated understanding the six categories in Item 6 (mother).
Categories
Number of participants Number of participants
Comments*
who understood the
who did not understand
category
the category
Medical worker
5
0
Office worker
3
2
Students did not elaborate
Equipment worker
4
1
Student did not elaborate
Education worker
4
1
Student did not elaborate
Specialized worker
1
4
Students did not elaborate
Helps People
5
0
* The “Comments” column identifies what participants most commonly did not understand about the category, or
whether they suggested another term for the category.

Notations of explanation are enclosed in parentheses ( ).

Table 14. Number of participants who indicated understanding the six categories in Item 11 (father).
Categories
Number of participants Number of participants
Comments*
who understood the
who did not understand
category
the category
Medical worker
4
0
Office worker
2
2
Students did not elaborate
Equipment worker
0
4
Students did not elaborate
Education worker
3
1
Student did not elaborate
Specialized worker
2
2
Students did not elaborate
Helps People
4
0
Medical Worker
001: Yes, help people with heart attack, coughing
002: Helps people, helps
003: Doctor surgeon nurse anyone who helps when you are sick
004: Like a person that's for your health.
005: Does medical stuff.
006: Works with people's health
007: Helps you with your health when you are sick or have a fever.
008: Like a doctor stuff like that
009: People who help you in a hospital
Office Worker
001: When asked if she could describe an office worker: no
002: Works in the office and they help people by experimenting.
003: Someone who works in an office or computer
004: I don't know.
005: I think that it does when somebody is in the store, the office comes out.
006: Can do anything, really. Could even be a medical worker because they might work in an office.
007: Someone who works in the office on a computer. Someone who works for a long time.
008: Police
009: Works in an office
Equipment worker
001: When asked if she could describe an equipment worker: no
002: Someone who works with equipment, like a builder.
003: Gives you equipment like a computer or hard drive
004: Uses equipment for stuff.
005: Does a lot of work.
006: The people that come to your house and fix stuff. And taxis.
007: Someone that does a good deed or something else like painting.
008: Like they do housework
009: People who work with a lot of equipment
Education worker
001: Yes, like a teacher
002: Someone who works in a school, like a principal. Someone who teaches something.
003: Someone who educates…yes. I don’t know about that one
004: Helps you have education.
33

005: I don't know.
006: Someone that teaches.
007: Someone who helps others with their work or takes care of them.
008: They teach
009: Works in a school or help people in private school
Specialized worker
001: (When asked if she knows what a specialized worker is): no
002: Umm, I don't really know. I mean I guess something you wouldn't find in a college. The kind of stuff you do in
the Olympics. If you do that for a living.
003: I don’t know
004: I don't know.
005: I don't know.
006: Ones that don't fit into any categories that are special, that make more money than the others.
007: A person that does stuff that people might want to do, people might not want to do. And they are special.
008: Umm they are special
009: They specialize in different things. Like doctors
Person who “helps people”
001: (When asked if she knows what a person who “helps people” does): yes
002: Lower than other office worker, like janitor, cashier, keeps things orderly, helps out with daily stuff
happening.
003: Umm a cook helps with cooking helps around the house
004: Cooks, gives them food and advice, keeps our place safe.
005: Somebody that helps somebody.
006: People that help us. They make us food.
007: Helps people with what they need to be helped with
008: They like to help people
009: They help people a lot, like firefighters help people if their house is burning.

Notations of explanation are enclosed in parentheses ( ).

Item 7 and Item 12
7) What does your mother do for a living?
Fill in the circle next to the job that most closely describes what she does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Accountant
Artist
Athlete
Carpenter
Cashier
Cleaner
Clerk
College professor
Construction worker
Cook
Customer service
Day care worker
Dental hygienist
Dentist
Designer
Doctor
Education administrator
Electrician
Engineer
Executive
Factory worker
Farmer
Firefighter
Grounds worker
Health aide
Helper
Housekeeper
Janitor
Lawyer
Librarian
Manager

o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Marketing
Mechanic
Medical technician
Military
Nurse
Office support
Painter
Pharmacist
Police
Receptionist
Religious worker
Repair technician
Salesperson
Scientist
Secretary
Security guard
Social worker
Software developer
Stocker
Supervisor
Surgeon
Surveyor
Taxi driver
Teacher
Therapist
Transportation
Truck driver
Veterinarian
Server
Writer
Other:__________________

12) What does your father do for a living?
Fill in the circle next to the job that most closely describes what he does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Can you tell me, in your own words, what the question is asking?
This probe was only asked when the item was the first version. Only four (4) of the participants were asked this
probe. Four (4) participants for the father item were able to rephrase the question in his/her own words.
Verbatim responses follow.3

MOTHER (item 7):
 003: What does my mom do for a living.
 005: What my mom does.
 008: What does my mother do for a living
because she because they didn’t have nails.

FATHER (item 12):
 004: Asking for detail what my father does.
 005: What my mom does.
 007: What does your father do for a living
 009: What does he do with his day. What does
he do to earn money.

As you were reading through the answer choices, what were you thinking about? Did you read all the answers
before making your selection?
One (1) participant could describe what he/she was thinking when answering the mother item. Three (3)
participants could describe what they were thinking when answering the father item. Verbatim responses follow:

MOTHER (item 7):
 001: No response from participant..
 002: No response from participant.
 003: Saw that it was alphabetical and just looked
at the list. Looked at the whole list.
 004: You design nails as art.
 005: No response from participant.
 006: No response from participant.
 008: No response from participant..








3
Notations
of explanation
are enclosed in parentheses ( ).
FATHER
(item
12):

004: Since I saw construction worker for my
mother, I did not want to choose it again. So I
wrote it.
005: For a living. (Multiple answers): I chose
writer, housekeeper, helper, cook, designer,
college professor, social worker, cleaner, artist,
nurse. I chose nurse because when I get hurt or
someone gets hurt, he helps.
006: No response from participant.
007: No response from participant.
008: Chose other because they did not have
what he did on the list.
009: This is not a popular thing. Did not think it
would be on the list.

Can you tell me, in your own words, how you understand the words “most closely” in the question?
This probe was only asked when the item was the first version. Only four (4) of the participants were asked this
probe. None of the participants were able to answer what “most closely” meant for the mother item. One (1)
participant provided a vague response for the father item. Verbatim responses follow.3

MOTHER (item 7):
 001: You have to listen closely…I don’t know. I
was reading each one [laughs] yes, yes, yes I
read the rest of them..[moans, fakes crying]
 003: What word was just about the same as her
job. Yes.

FATHER (item 12):
 007: What he does, sometimes engineering and
sometimes working from home. There might be
other choices, he cooks, he cleans a lot. And
other things that I don't know.
 009: Maybe the closest. But I couldn't find what.

Why do you think the different options are presented in this order? Can you see a pattern that would explain
why the options are ordered like this?
Three (3) participants were able to find a pattern for the mother item. Four (4) participants were able to find a
pattern for the father item. Verbatim responses follow.
MOTHER (item 7):
 001: No response from participant.
 002: Alphabetically.
 003: That it is alphabetical and that's it. Hard
because some of these words I don’t know.
 004: Yes, alphabetical.
 005: No response from participant..
 006: No response from participant.
 008: No response from participant.

FATHER (item 12):
 004: No response from participant.
 005: Unable to see a pattern.
 006: ABC order.
 007: ABC order, in a list.
 008: They are in A-B-C order
 009: Alphabetical order.

Notations of explanation are enclosed in parentheses ( ).

Did you find any part of the question confusing? Were there any words that you didn’t know?
One (1) participant found the item confusing for the mother item. None of the participants found the item
confusing for the father item. Verbatim responses follow.3
MOTHER (item 7):
 001: No response from participant.
 002: No
 003: Yes, did not know some of the words.
(Student confused “veteran” and “veterinarian.”
Students mother is a “veteran.”
 004: No
 005: No
 006: No
 008: No response from participant.

FATHER (item 12):
 004: No
 005: No (Student showed signs of boredom).
 006: No
 007: Not really.
 008: No
 009: No

Notations of explanation are enclosed in parentheses ( ).

Overall, how easy or hard was it to choose an answer from the list of choices? Would you say choosing an
answer was very easy, easy, hard, or very hard?
For the mother item, four (4) participants found the item easy, one (1) found it off medium difficulty and two (2)
found it hard. For the father item, three (3) participants found the item easy to very easy and two (2) participants
found the item to be hard. Verbatim responses follow tables 15 and 16.3

Table 15. Number of participants who found Item 7 (mother) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
0
Easy
4
Medium
1
Hard
2
Very hard
0
Total Number of Responses
7
Table 16. Number of participants who found Item 12 (father) very easy, easy, hard, or very hard.
Level of Difficulty
Frequency
Very easy
1
Easy
2
Hard
2
Very hard
0
Total Number of Responses
5
MOTHER (item 7):
 001: Easy
 002: Medium; wasn't easy to look over (easy to answer once he found the answer)
 003: Hard. Some of the words I don't really know.
 004: A little hard.
 005: Easy
 006: Easy
 008: It was easy
FATHER (item 12):
 004: Very easy
 005: Very easy.
 006: Easy.
 007: Hard because sometimes I got confused about what your father does for a living and what your father
does. What he does for a job might be more clear.
 008: It was easy. I think this to me is easier [inaudible].
 009: Hard. Missed the words "most closely".

Notations of explanation are enclosed in parentheses ( ).

Version preference
No participants preferred version A, five (5) participants preferred version B, and four (4) participants preferred
version C (Table 17).
Table 17. Number of participants who preferred “version 1,” “version 2”, and “version 3” of the “What does
your mother/father do for a living?” item.
Versions
Number of participants who
Comments*
preferred a given version
Version a (i.e., item with answer
0
options a through o)
Version b (i.e., item with answer
5
 Okay. The second one, the middle,
options classified by six categories)
easy, this was super-duper hard…this
one is easy…it had all the special
workers that I like and want to be…I
do know some words but I don't
know what they stand for.
 This list is easier. Pretty easy to look
for the job by the category.
 The list with the subtitles was easier.
Subtitle list is scrambled, but using
the headings made it easier
 I still think it is the one with the
categories because it is more clear
and not bunched up and you can read
it more clearly.
Version c (i.e., item with
alphabetized answer options)







The alphabetical list. Could read
options easier.
Question 5 was easier because it's
straight and you can just pick.
Participant prefers the ABC order
response options because that was
the easiest.
I think it was the same…these
two…umm I think this one makes
more sense (ABC)

* The “Comments” column identifies what participants most commonly did not understand about the version,
what they liked/disliked about a version, or whether they made suggestions on how to improve a version.

Job Titles
Two-thirds of the sample had problems defining the following job titles: carpenter, clerk, customer service, dental
hygienist, education administrator, engineer, executive, factory worker, health aide, marketing, medical
technician, pharmacist, religious worker, salesperson, secretary, social worker, software developer, stocker,
supervisor, surveyor, and therapist, as shown in Table 18.
Table 18. Number of participants who did/did not experience difficulty with the job titles in items 3 – 5 and 10 –
12. There were ten (10) job titles in which a majority of the students experienced difficulties in defining. These job
titles are in bold below.
Number of participants Number of participants
Comments*
who did not experience
who experienced
difficulty with the job
difficulty with the job
Job Title
title
title
Accountant
6
2
Artist
8
0
Athlete
8
0
Thought about carpet cleaner
Carpenter
4
5
Cashier
Cleaner
Clerk
College professor
Construction worker
Cook
Customer service
Day care worker
Dental hygienist
Dentist
Designer
Doctor
Education administrator
Electrician
Engineer

8
9
1
8
8
9
6
9
4
9
7
9
3
8
6

1
0
8
1
1
0
3
0
4
0
0
0
5
1
3

Executive
Factory worker
Farmer
Firefighter
Grounds worker

5
5
8
8
9

3
3
1
1
0

009: referred to “branch of
government”

Table 18. Number of participants who did/did not experience difficulty with the job titles in items 3 – 5 and 10 –
12. (continued)
Number of participants Number of participants
Comments*
who did not experience
who experienced
difficulty with the job
difficulty with the job
Job Title
title
title
Health aide
6
3
Helper
9
0
Housekeeper
8
1
Janitor
9
0
Lawyer
7
2
Librarian
9
0
Manager
8
0
Word was interchanged with
Marketing
3
6
“market”
Mechanic
4
5
Medical technician
5
3
Military
7
2
Nurse
7
1
Office support
4
5
Painter
8
1
Word was interchanged with
Pharmacist
5
3
“farmer”
Police
9
0
Word was interchanged with
Receptionist
1
0
“reception”
Religious worker
5
4
Repair technician
8
1
Salesperson
5
4
Scientist
9
0
Secretary
2
7
Security guard
9
0
Server
6
3
Social worker
1
8
Software developer
3
5
Word was interchanged with
“stalker”
Word was interchanged with
Stocker
2
7
“stockbroker”
Supervisor
5
3

SUMMARY
One goal in 2015 is to link NAEP and the ECLS-K: 2011 in order to provide additional contextual information. In line
with these efforts, a new questionnaire namely, the NAEP-ECLS survey questionnaire, was created to validate
student responses to SES-related questions by comparing to parent responses from NAEP-ECLS. As part of NAEP’s
item development process for the NAEP-ECLS survey questionnaires, a portion of survey questions are pretested
before they are administered in an operational assessment. In this study, these pretest activities included
cognitive interviews with nine students who were in the third grade entering the fourth or in the fourth grade
entering the fifth grade.
The following are the overall findings from the research:
1) In general, the students were aware of the educational achievement of their mother and father. Students
were shown pictures or diplomas, told of, or participated in their parents’ educational achievements.
2) There were students who were unable to rephrase questions in their own words. For example, this is a
result of distinguishing between what the parents “do” or “do for a living.”
3) There was little confusion with the items and the students indicated that the items were “easy” or “very
easy” to answer. Even when the students’ responses suggested otherwise, students selected the “easy” or
“very easy” response.
4) Of the job titles, there were ten (10) in which a majority of the students experienced difficulties in
defining: carpenter, clerk, education administrator, marketing, mechanic, secretary, social worker,
software developer, stocker, and surveyor. In all, some students, i.e., not more than three (3), had
problems defining the following job titles: customer service, dental hygienist, engineer, factory worker,
health aide, marketing, medical technician, pharmacist, religious worker, salesperson, secretary, social
worker, software developer, stocker, supervisor, surveyor, and therapist. Only the job title, dental
hygienist, was tied with four (4) students each experiencing and not experiencing difficulty.
5) Students were not able to distinguish the literal and actual activities of their parents. The distinctions
between what parents “do”, “for work”, or “for a living” were confused with day-to-day activities that the
students perceived that the parent perform in their everyday lives. Phrases such as “do” or “for a living”
were associated with duties performed around the home and not necessarily associated with the work
outside of the home that parents do to earn money. Also, students responded to the “how far” question
as it relates to physical distance and not as it relates to academic progress.
6) Items contained in the interview seem insensitive when asking about parents, i.e., mother’s and father’s
work, etc. In the only instance where a participant did not know her father, this information was only
made apparent almost at the end of the interview. At that point the interview was stopped and the
remaining two items were not asked. It should be noted that limiting the participants to those who have a
mother and father is not representative of the national population as many households today are headed
by grandparents, aunts and uncles, or other siblings. Single parents also head up many “modern families”.
7) In some instances, there were not verbal responses from the participants, but expressions such as
students sighed, yawned, rolled their eyes, slouched in the chair, or lowered voice. However, every effort
was made by the moderators to get verbal responses from student participants as the students’ interest
waned or the students seemed confused.

APPENDICES (A-H)

Appendix A: Sequence Protocol

ORDER
1
2
3
4
5
6
7
8[1]
9
10

Seq 1
Seq 2
Seq 3
Seq 4
Seq 5
#3
#3
#3
#3
#8
#4
#4
#4
#4
#9
#5
#6
#7
#5
#10
#6
#7
#5
#7
#11
#7
#5
#6
#6
#12
#8
#8
#8
#8
#3
#9
#9
#9
#9
#4
#10, #11, or #10, #11, or #10, #11, or #10, #11, or #5, #6, or
#12
#12
#12
#12
#7
#1
#2

#2
#1

#1
#2

#2
#1

#1
#2

Seq 6
#8
#9
#11
#10
#12
#3
#4
#5, #6, or
#7
#2
#1

Seq 7
Seq 8
Seq 9
#8
#8
#8
#9
#9
#9
#12
#11
#12
#11
#12
#10
#10
#10
#11
#3
#3
#3
#4
#4
#4
#5, #6, or #5, #6, or #5, #6, or #7
#7
#7
#1
#2

#2
#1

#1
#2

[1] This item is contingent upon which version of this item the student stated he/she preferred. The interviewer will have to be prepared to present
the appropriate item on the spot.
Only the first item of the sequence should include job title probes (i.e., probes 8-23, pages 10-15 of items and probes document).

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix B: Grade 3 and Grade 4 Recruitment Flyer

STUDENT INTERVIEWS
SCHEDULED TO TAKE PLACE JUNE 18-27, 2014

Tell us what you think!

The one-on-one interviews are designed to inform the National Assessment of
Educational Progress (NAEP) 2015 survey questionnaire development.
Students in grades 3 and 4 will be asked to give their opinions about questions for the
2015 Nation’s Report Card assessments.
Incentives will be provided to the student ($25 Visa gift card) and to his or her
parent/guardian for transporting the child to the interview ($25 Visa gift card).
Student responses will be audio recorded. No video recording will take place.
Student identities and responses will remain anonymous.
Location of the Interview: The Office Suite of CRP, Inc.
1110 Bonifant Street, Suite 400, Silver Spring, MD 20910

Interested?

Contact:
CRP, Incorporated
*****************
Telephone:
1-866-277-4772
******************
Ask for:
Dr. Sondra Gaines or
Ms. Lakeisha Scott

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix C: Parent/Guardian Letter/E-mail of Introduction

[Date]
Name
Address
Address
City, State, ZIP Code
Dear [Insert parent/guardian name]:
It is my pleasure to invite your child to participate in a cognitive interview designed to discuss technology issues
in education. The cognitive interview is scheduled to take place in June, 2014. Your child was selected to
participate because [HE/SHE] is in the [THIRD/FOURTH] grade. Your child will be participating with
approximately eight other [THIRD/FOURTH] grade students from the metropolitan Washington, D.C., area.
However, each student will be interviewed alone. Students will be asked to provide their opinions about the
questions that have been designed for 2015 NAEP assessments. The information your child provides will be
helpful in the development of future assessments for [HIS/HER] grade and class.
The cognitive interview is designed to obtain feedback from students in the third and fourth grades to inform
the National Assessment of Educational Progress (NAEP) of the validity of newly developed test questions. Your
child’s identification will not be disclosed and all information about and derived from this cognitive interview
will be kept confidential. Parents/legal guardians are encouraged to stay and may view the entire cognitive
interview. Your child’s participation totals 60 minutes; children will be allowed breaks as needed. Your child will
receive a $25 Visa gift card at the end of the cognitive interview session for participating. You, the parent, will
also receive a $25 gift card for transporting your child to the cognitive interview session.
[CRP, Inc., EMPLOYEE NAME] from CRP, Inc., will be contacting you to confirm receipt of this [LETTER/EMAIL],
answer any questions you may have, and discuss your interest in allowing your child to participate in the
cognitive interview. For questions, please call Dr. Roger W. Morrell, Task Lead, at 1-866-277-4772 or e-mail at
rmorrell@crpcorp.com. Thank you for your time and consideration. We look forward to your child’s
participation in this important work.

Sincerely,

Carolyn B. Rudd, EdD
President/CEO
CRP, Inc.

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix D: Parent/Guardian Phone Script
Good . May I please speak to ?
Good . My name is , and I am contacting you from CRP, Inc. We’re
a minority- and woman-owned professional services and management consulting firm located in Silver Spring,
Maryland, that provides technical consulting. I am calling to follow up on a that we sent on
regarding a study we have been asked to do on behalf of the United States Department of Education’s
National Center for Education Statistics, in collaboration with Educational Testing Service, to inform the survey
questionnaire development. The surveys are conducted through the National Assessment of Educational
Progress, also known as NAEP. Your student, [child’s name], has been invited to participate in the cognitive
interview.
As you know from our , we are looking to conduct cognitive interviews with third- and fourthgrade students to find out how to improve the NAEP surveys. The cognitive interviews will last about 60
minutes. The cognitive interview is recorded using a voice recorder, but all information is reported in combined
form, so individuals will not be identified. Participant responses will be used only for research purposes and will
not be disclosed or used, in identifiable form, for any other purpose except as required by the Education
Sciences Reform Act of 2002.
These cognitive interviews are an important stage of the NAEP survey development process. Participant
comments will help the National Center for Education Statistics improve the survey questionnaires. Student
participants and their parent or legal guardian will each be given a $25 Visa gift card for their time.
[If the parent/guardian response is no] Thank you for your time. Have a wonderful .
[If the parent/guardian response is yes] Great, thank you very much. We may need to reach you by telephone. Is
the number I called the best number to use?
Do you have any questions at this time?
If you do have any questions before your scheduled cognitive interview for , please call .
Thank you again for agreeing to be part of this very important research study.

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix E: Parent/Guardian Informed Consent
PARENTAL CONSENT DOCUMENT
FOR PARTICIPATION IN RESEARCH
Educational Testing Service (ETS)
Title of Research Study: NAEP Item Development
(ID) Survey Questionnaires: Exploratory Research Study Plan for Cognitive Interviews
Thank you for your interest in having your child participate in the research study described below. This consent
form explains the research study. Please read it carefully and feel free to ask questions about anything you do
not understand. If you do not have questions now, you may ask them if they occur to you later.
1. What is this study about?
On behalf of the National Center for Education Statistics (NCES), part of the United States Department of
Education, ETS is conducting cognitive interviews with students in grades 3 and 4 about new test questions
for the National Assessment of Educational Progress (NAEP). The questions we are pretesting focus on
socioeconomic status (SES)-related questions and will be part of a new computer-based test given to a
sample of students across the country. In these sessions we will show students early versions of SES-related
questions. ETS staff will audio record and take notes on how your child reacts to the questions in order to
revise and refine the test content. Questions will focus on your child’s level of interest and ability to
understand the questions. Your child will also be asked whether he or she has any suggestions on how we
might improve the questions.
2. Where will the cognitive interview sessions take place?
The sessions will take place at CRP, Inc., during an after-school program. Each session requires only one visit.
3. Will you keep the information private and confidential?
Your child will be assigned a unique student identifier (ID), and at no time will his or her name be linked to
any of his or her answers. Your child’s responses will be used only for research purposes and will not be
disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences
Reform Act of 2002, 20 U.S.C., §9573]. The sessions will be audio recorded, as long as you agree, and the
recordings will be used only by the researchers conducting the study.
4. How long will each cognitive interview session last, and when will the sessions take place?
Each session will last up to 60 minutes. Sessions will take place .

2015 NAEP-ECLS Special Study
Cognitive Interview Report
5. Will the results go to my child’s school?
Your child’s answers will not be disclosed to his or her school and will not be linked to any personally
identifiable information, such as the school’s name.
6. What are the possible risks of being in the study?
There are no known risks to participating in this research.
7. Whom can I contact with questions or for further information?
is conducting this research at ETS on behalf of NCES. If you have any questions, please
call at or by e-mail at .
8. Can participation end early?
Participation in this study is completely voluntary. Your child has the right to refuse to answer particular
questions. Your child may also elect to withdraw from this study at any time and can end the session early if
he or she wishes.
By signing below, you agree that your child, _________________________________, may participate in this
research study. Your child will receive a $25 Visa gift card at the conclusion of each session. If you provide
transportation to CRP Inc., for your child to participate in the cognitive interview session you the
parent/guardian will receive a $25 Visa gift card. In addition, you will receive a thank you letter/e-mail for taking
your time to bring your child to the testing site. Your signature below will indicate that your questions have been
answered satisfactorily and that you have read and understood the information provided above.
I am a U.S. citizen/U.S. permanent resident. I have not received payment from ETS for any other service
performed during the current calendar year 2014 and I am not an employee of ETS.
I am a U.S. citizen/U.S. permanent resident. I have received payment from ETS during the current
calendar year 2014 that will NOT meet or exceed $600 in total and I am not an employee of ETS.
I am a U.S. citizen/U.S. permanent resident. I have received payment from ETS during the current
calendar year 2014 that WILL meet or exceed $600 in total and I am not an employee of ETS.
I am a foreign national on a non-immigrant visa. I certify that I have not received payments exceeding
$100 in total, including this study, from ETS in the calendar year 2014.
Print Name: _____________________________________________
Signature:
______________________________________________________Date:___________________
E-mail address: _____________________________________________
--------------------------------------------------------------------------------------------------------------------To be completed by ETS staff:
Payment Card Transaction Number: __________________________ (for student)
Payment Card Transaction Number: __________________________ (for parent/guardian)

2015 NAEP-ECLS Special Study
Cognitive Interview Report

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix F: Parent/Guardian Confirmation of In-Person Interview Letter/E-mail

Dear ,
Thank you for agreeing to allow to participate in this voluntary research cognitive interview.
Educational Testing Service (ETS) and CRP have been asked to conduct this research on behalf of the United
States Department of Education’s National Center for Education Statistics (NCES) to inform the development of
the 2015 National Assessment of Educational Progress survey questions. All of the information you and your
child provide will be used only for research purposes and will not be disclosed or used, in identifiable form, for
any other purpose except as required by the Education Sciences Reform Act of 2002, 20 U.S.C., § 9573.
Your student has been scheduled to participate in a cognitive interview study at on . The
cognitive interview will last no more than 60 minutes. To thank your child for his or her time, he or she will
receive a $25 Visa gift card for participating. To thank you the parent/guardian for transporting your child to
CRP Inc., for the cognitive interview session you will receive a $25 Visa gift card.
The cognitive interview will take place at .

We have a consent form this confirmation. Please sign, date, and return
the form when you come to the cognitive interview session, granting consent for your student to participate in
the NAEP research. Without your written consent, your student will not be able to participate in this research
study.
We look forward to your help in improving the NAEP survey questionnaires.
If you have any questions, please contact .
Sincerely,

Carolyn B. Rudd, EdD
President/CEO
CRP, Inc.

2015 NAEP-ECLS Special Study
Cognitive Interview Report
Appendix G: Parent/Guardian Thank-You Letter/E-mail

Dear :
On behalf of the National Center for Education Statistics (NCES) of the United States Department of Education
and CRP, Inc., I would like to thank you for allowing your student to participate in our research interviews.
Educational Testing Service (ETS) and CRP were asked to conduct the study on behalf of the United States
Department of Education’s National Center for Education Statistics (NCES) to inform the development of the
2015 National Assessment of Educational Progress (NAEP) survey questionnaire. Your student’s input will help
us develop the best possible survey questionnaires for future NAEP assessments.
Your student’s input, combined with input from other students, is incredibly useful and informative for ongoing
survey design efforts.
Again, thank you for your assistance.
Sincerely,

Carolyn B. Rudd, EdD
President/CEO
CRP, Inc.

2015 NAEP-ECLS Special Study
Cognitive Interview

APPENDIX I (PROTOCOL)

2015 NAEP-ECLS Special Study
Cognitive Interview

I. Interviewer Welcome Script and Assent/Consent
The following script does not have to be read verbatim. You, as the interviewer, should be familiar enough with
the script to introduce the participant to the cognitive interview process in a conversational manner. Text written
in italics is suggested content for you to be thoroughly familiar with in advance. You should project a warm and
reassuring manner toward the participant in order to develop a friendly rapport and as such should use
conversational language throughout.
After answering questions and giving further explanation, begin the interview with the first item.
Welcome Script:
Hello, my name is .
Thank you for agreeing to participate in our study. Your help is very important to us. Let me tell you a bit
more about what we are going to do today.
I have some questions that students like you in many different schools across the whole country will be
asked about themselves and their families. The questions are part of a survey for the National Assessment
of Educational Progress (NAEP) research study.
Our goal is to make these questions better so that other students can easily understand them. That is why
we need your feedback about these questions.
Some of the questions may be difficult to understand or answer. We want you to help us find out what is
easy to understand and what is difficult to understand. There are no right or wrong answers. Your
feedback will help the National Center for Education Statistics (NCES), which is part of the United States
Department of Education, make these questions clearer for students like you.
This interview is being recorded so researchers can review the recordings later. No one else will see or hear
the recording. Participant responses are voluntary and will be used only for research purposes. They will
not be disclosed or used, in identifiable form, for any other purpose except as required by the Education
Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573.
Do you have any questions?

2015 NAEP-ECLS Special Study
Cognitive Interview
II. NAEP-ECLS ITEMS AND PROBES
After the participant is welcomed and the purpose of the study is explained, the cognitive interview starts.
Please ask the participant to read each question first. Then, start with the generic probes followed by the
specific probes for each item. In the tables below each question are suggested probes, written in italics, which
you should be thoroughly familiar with in advance.
Some probes are generic; that is, they are included for all items. Some probes are item specific; that is, they
apply to specific aspects of a given item. Note that some follow-up probes are conditional on previous student
responses.
All student responses should be captured. The notetaker should also record approximate participant response
times for reading and answering each item.
The maximum duration of each cognitive interview will be 60 minutes.
Note: The order in which the items are presented to the student will be randomized; accordingly, various
versions of questionnaire booklets for the actual cognitive interviews will be created.
Also please note that half of the participants will start with the occupation items (i.e., items 5, 6, 7, 10, 11, and
12) that pertain to one gender, namely mother (items 5, 6, and 7) or father (items 10, 11, and 12). Then the
interviewer will present one version of the occupation item for the other gender; the version that the student
found the easiest/clearest to answer will be presented. For example, if the student answers the mother
occupation items (items 5, 6, and 7), and indicates that item 6 was the easiest/clearest to answer, the
participant would be presented with the father equivalent of item 6, namely item 11. The participant would not
be asked to answer items 10 or 12.
The same suggested probes should be used for both sets of questions (i.e., mother and father occupation
questions). The item-specific probes asking for understanding of the job titles should be asked only once during
the cognitive interview.

2015 NAEP-ECLS Special Study
Cognitive Interview
STUDENT QUESTIONNAIRE – GRADE 4
[Grade 4 ECLS – Student Question #1]

1) How far in school did your mother go?
a) She did not finish high school.
b) She graduated from high school.
c) She had some education after high school.
d) She graduated from college.
e) I don’t know.

First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.
Suggested Probes:
No.
1
2
3

Probe
Can you tell me, in your own words, what the question is asking?
Why did you select this choice? How did you know what answer to select?
Did you find any part of the question confusing? Were there any words that you didn’t
know?
If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?

Overall, how easy or hard was it to choose an answer from the list of choices? Would you
say choosing an answer from the list of choices was very easy, easy, hard, or very hard?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #2]

2) How far in school did your father go?
a) He did not finish high school.
b) He graduated from high school.
c) He had some education after high school.
d) He graduated from college.
e) I don’t know.

First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.
Suggested Probes:
No.
1
2
3

Overall, how easy or hard was it to choose an answer from the list of choices? Would you
say choosing an answer from the list of choices was very easy, easy, hard, or very hard?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #3]

3) Does your mother work?
a) Yes
b) No
c) I don’t know

First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.
Suggested Probes:
No.
1

2
3

Probe
Can you tell me, in your own words, what the question is asking?
If the student does not describe or mention what “work” means, probe this follow up
question: What does “work” mean to you?
Why did you select this choice? How did you know what answer to select?
Did you find any part of the question confusing? Were there any words that you didn’t
know?
If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?

4
5

Overall, how easy or hard was it to choose an answer from the list of choices? Would you
say choosing an answer was very easy, easy, hard, or very hard?
If answer choice “yes”: Do you know if your mother is working full time or part time?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #4]
4) What does your mother do? (for example, She’s a lawyer; She teaches 1st grade; She cooks in a
restaurant; She takes care of the home and family)
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.
Suggested Probes:
No.
1
2
3

Probe
Can you tell me, in your own words, what the question is asking?
Please read your answer out loud. How did you decide what to write down? How did you
decide how much to write?
Did you find any part of the question confusing? Were there any words that you didn’t
know?
If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?

4
5

What do you think about the examples in the question? Did the examples help you
answer the question?
Overall, how easy or hard was it to answer the question? Would you say it was very easy,
easy, hard, or very hard?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #5]

5) What does your mother do for a living?
Fill in the circle next to the list that most closely describes what she does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

(a) Doctor, dentist, surgeon, veterinarian
(b) College professor, librarian
(c) Teacher
(d) Engineer, surveyor, scientist, lawyer, social worker, religious worker
(e) Nurse, pharmacist, therapist, medical technician, dental hygienist
(f) Artist, athlete, designer, writer
(g) Executive, manager, supervisor, education administrator, accountant, software
developer
(h) Office support, secretary, clerk, receptionist, customer service
(i) Mechanic, repair technician, farmer, carpenter, grounds worker, electrician, painter,
miner
(j) Salesperson, marketing
(k) Server, cook, cashier, health aide, janitor, day care worker
(l) Police, firefighter, security guard
(m) Factory worker, transportation, truck driver, taxi driver
(n) Housekeeper, cleaner, helper, stocker, construction worker
(o) Military
(p) Other: ______________________

First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.
Item-Specific Probes:
No.
1

Probe
Can you tell me, in your own words, what the question is asking?

As you were reading through the answer choices, what were you thinking about? Did you
read all the answers before making your selection?
If the student “other”: Why did you choose other? Did you check to see whether any of
the other options could apply as well?

3
4
5

If the student selects multiple answers: I see that you selected multiple options. Can you
explain why you chose more than one option?
Can you tell me, in your own words, how you understand the words “most closely” in the
question?
Why do you think the different options are presented in this order? Can you see a pattern
that would explain why the options are ordered like this?
Did you find any part of the question confusing?

2015 NAEP-ECLS Special Study
Cognitive Interview

6
7

If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?
Overall, how easy or hard was it to choose an answer from the list of choices? Would you
say choosing an answer from the list of choices was very easy, easy, hard, or very hard?
The following probe will be asked if this version of the item is administered second:
Compared with the previous question that you answered [show it again to student], which
question was easier to answer? Why?
The following probe will be asked if this version of the item is administered third:
Compared with the previous two questions that you answered [show them again to
student], which question was easier to answer? Why?
Before moving on to the next question let’s take a closer look at the response options for
this question to see how we could improve this question. We will start with option a.
Were there any words you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up question.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “doctor” does at work?
- How would you describe what a “dentist” does at work?
- How would you describe what a “surgeon” does at work?
- How would you describe what a “veterinarian” does at work?
Now let’s look at option b.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the next follow up question.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “college professor” does at work?
- How would you describe what a “librarian” does at work?
Now let’s look at option c.
64

2015 NAEP-ECLS Special Study
Cognitive Interview

- How would you describe what a “teacher” does at work?
Now let’s look at option d.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what an “engineer” does at work?
- How would you describe what a “surveyor” does at work?
- How would you describe what a “scientist” does at work?
- How would you describe what a “lawyer” does at work?
- How would you describe what a “social worker” does at work?
- How would you describe what a “religious worker” does at work?
Now let’s look at option e.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “nurse” does at work?
- How would you describe what a “pharmacist” does at work?
- How would you describe what a “therapist” does at work?
- How would you describe what a “medical technician” does at work?
- How would you describe what a “dental hygienist” does at work?
Now let’s look at option f.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.

2015 NAEP-ECLS Special Study
Cognitive Interview
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what an “artist” does at work?
- How would you describe what an “athlete” does at work?
- How would you describe what a “designer” does at work?
- How would you describe what a “writer” does at work?
Now let’s look at option g.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what an “executive” does at work?
- How would you describe what a “manager” does at work?
- How would you describe what a “supervisor” does at work?
- How would you describe what an “education administrator” does at work?
- How would you describe what an “accountant” does at work?
- How would you describe what a “software developer” does at work?
Now let’s look at option h.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]
-

How would you describe what an “office support” does at work?
How would you describe what a “secretary” does at work?
How would you describe what a “clerk” does at work?
How would you describe what a “receptionist” does at work?
66

2015 NAEP-ECLS Special Study
Cognitive Interview

- How would you describe what a “customer service” does at work?
Now let’s look at option i.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she know what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “mechanic” does at work?
- How would you describe what a “repair technician” does at work?
- How would you describe what a “farmer” does at work?
- How would you describe what a “carpenter” does at work?
- How would you describe what a “grounds worker” does at work?
- How would you describe what an “electrician” does at work?
- How would you describe what a “painter” does at work?
- How would you describe what a “miner” does at work?
Now let’s look at option j.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “salesperson” does at work?
How would you describe what a person in “marketing” does at work?
Now let’s look at option k.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
67

2015 NAEP-ECLS Special Study
Cognitive Interview
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “server” does at work?
- How would you describe what a “cook” does at work?
- How would you describe what a “cashier” does at work?
- How would you describe what a “health aide” does at work?
- How would you describe what a “janitor” does at work?
- How would you describe what a “day care worker” does at work?
Now let’s look at option l.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she know what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what “police” do at work?
- How would you describe what a “firefighter” does at work?
- How would you describe what a “security guard” does at work?
Now let’s look at option m.
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “factory worker” does at work?
- How would you describe what a person in “transportation” does at work?
- How would you describe what a “truck driver” does at work?
- How would you describe what a “taxi driver” does at work?
Now let’s look at option n.

2015 NAEP-ECLS Special Study
Cognitive Interview
Were there any words that you didn’t know? [If a student indicates that he or she did not
understand a word, please prompt the student as follows.] This word means .
If the student still does not know the word move on to the follow up questions.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?
After checking for understanding, ask the following follow-up questions. [Skip follow-up
questions about words the student does not know even after hearing the job definition.]

- How would you describe what a “housekeeper” does at work?
- How would you describe what a “cleaner” does at work?
- How would you describe what a “helper” does at work?
- How would you describe what a “stocker” does at work?
- How would you describe what a “construction worker” does at work?
Now let’s look at option o. How would you describe what a person in the “military” does at
work?
[If a student indicates that he or she did not understand this word, please prompt the
student as follows.] This word means .
If the student still does not know the word move on to the next probe.
If the student indicates that he or she knows what the word means after hearing the
definition ask the following question: Do you know this job by a different word or term?

Let’s now look at option p. What does the word “other” mean to you?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #6]
This question is similar to the previous one, but it looks a bit different. Please read the question and select your
answer. First, let the student read the question and provide an answer. Then, use the suggested probes below to
further probe the student’s understanding of the question.

2015 NAEP-ECLS Special Study
Cognitive Interview

6) What does your mother do for a living?
Fill in the circle next to the job that most closely describes what she does.
Medical worker
O Doctor
O Dentist
O Surgeon
O Veterinarian
O Nurse
O Pharmacist
O Therapist
O Medical technician
O Dental hygienist
O Health aide
O Other Medical worker
(please specify):
__________

Education worker
O College professor
O Librarian
O Teacher
O Education administrator
O Day care worker
O Other Education worker
(please specify):
__________

Helps people
O Server
O Cook
O Cashier
O Janitor
O Police
O Firefighter
O Security guard
O Housekeeper
O Cleaner
O Helper
O Military
O Other Helps People
(please specify):
__________

2015 NAEP-ECLS Special Study
Cognitive Interview
Item-Specific Probes:
No.
1
2

Probe
Can you tell me, in your own words, what the question is asking?
As you were reading through the answer choices, what were you thinking about? Did you
read all the answer choices before making your selection?
If the student selects any of the “other” categories: Why did you choose other? Did you
check to see whether any of the other options could apply as well?

If student selects multiple answers: I see that you selected multiple options. Can you
explain why you chose more than one option?
Did you find any part of the question confusing?
If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?

Before moving on to the next question, let’s take a closer look at the titles used for the six
groups of choices. We will start with the first one.
- How would you describe what a “medical worker” does at work?
- How would you describe what an “office worker” does at work?
- How would you describe what an “equipment worker” does at work?
- How would you describe what an “education worker” does at work?
- How would you describe what a “specialized worker” does at work?
- How would you describe what a person who “helps people” does at work?
When you answered the question, did these six titles help you to find your mother’s job or
did the titles make the question more complicated?
Overall, how easy or hard was it to choose an answer from that list of choices? Would
you say choosing an answer from that list of choices was very easy, easy, hard, or very
hard?
The following probe will be asked if this version of the item is administered second:
Compared with the previous question that you answered [show it again to student],
which question was easier to answer? Why?
The following probe will be asked if this version of the item is administered third:
Compared with the previous two questions that you answered [show them again to
student], which question was easier to answer? Why?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #7]
Here is another version of the question you just answered. Please read the question and select your answer.
First, let the student read the question and provide an answer. Then, use the suggested probes below to further
probe the student’s understanding of the question.

7) What does your mother do for a living?
Fill in the circle next to the job that most closely describes what she does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

Item-Specific Probes:

2015 NAEP-ECLS Special Study
Cognitive Interview
No.
1
2
3

Probe
As you were reading through the answer choices, what were you thinking about? Did you
read all the answers before making your selection?
Why do you think the different options are presented in this order? Can you see a pattern
that would explain why the options are ordered like this?
Did you find any part of the question confusing?
If yes: Which part did you find confusing? Why? What could we do to make the question
easier to understand?

4
5

Overall, how easy or hard was it to choose an answer from the list of choices? Would you
say choosing an answer from the list of choices was very easy, easy, hard, or very hard?
The following probe will be asked if this version of the item is administered second:
Compared with the previous question that you answered [show it again to student],
which question was easier to answer? Why?
The following probe will be asked if this version of the item is administered third:
Compared with the previous two questions that you answered [show them again to
student], which question was easier to answer? Why?

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #8]

8) Does your father work?
a) Yes
b) No
c) I don’t know

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #9]
9) What does your father do? (for example, He’s a lawyer; He teaches 1st grade; He cooks in a
restaurant; He takes care of the home and family)
____________________________________________________________________________
____________________________________________________________________________
____________________________________________________________________________

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #10]

10) What does your father do for a living?
Fill in the circle next to the list that most closely describes what he does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

(a) Doctor, dentist, surgeon, veterinarian
(b) College professor, librarian
(c) Teacher
(d) Engineer, surveyor, scientist, lawyer, social worker, religious worker
(e) Nurse, pharmacist, therapist, medical technician, dental hygienist
(f) Artist, athlete, designer, writer
(g) Executive, manager, supervisor, education administrator, accountant, software
developer
(h) Office support, secretary, clerk, receptionist, customer service
(i) Mechanic, repair technician, farmer, carpenter, grounds worker, electrician, painter,
miner
(j) Salesperson, marketing
(k) Server, cook, cashier, health aide, janitor, day care worker
(l) Police, firefighter, security guard
(m) Factory worker, transportation, truck driver, taxi driver
(n) Housekeeper, cleaner, helper, stocker, construction worker
(o) Military
(p) Other: ______________________

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #11]

11) What does your father do for a living?
Fill in the circle next to the job that most closely describes what he does.
Medical worker
O Doctor
O Dentist
O Surgeon
O Veterinarian
O Nurse
O Pharmacist
O Therapist
O Medical technician
O Dental hygienist
O Health aide
O Other Medical worker
(please specify):
__________

Education worker
O College professor
O Librarian
O Teacher
O Education administrator
O Day care worker
O Other Education worker
(please specify):
__________

Helps people
O Server
O Cook
O Cashier
O Janitor
O Police
O Firefighter
O Security guard
O Housekeeper
O Cleaner
O Helper
O Military
O Other Helps People
(please specify):
__________

2015 NAEP-ECLS Special Study
Cognitive Interview
[Grade 4 ECLS – Student Question #12]

12) What does your father do for a living?
Fill in the circle next to the job that most closely describes what he does.
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

2015 NAEP-ECLS Special Study
Cognitive Interview
III. Job Title Definitions
Below are a list of definitions, as defined in Merriam-Webster’s dictionary5 (unless otherwise indicated), that
pertain to the job titles referenced in items 5, 6, 7, 10, 11, and 12. These definitions are to be used by the
interviewer to help explain job title(s) that the participant is not familiar with.
The following definitions do not have to be read verbatim. You, as the interviewer, should be familiar enough
with the definitions below to describe them to the participant in a conversational manner.
1. Accountant- someone whose job is to keep the financial records of a business or person
2. Artist- a person who is skilled at drawing, painting, etc.
3. Athlete- a person who is trained in or good at sports, games, or exercises that require physical skill and
strength
4. Carpenter- a person whose job is to make or fix wooden objects or wooden parts of buildings
5. Cashier- a person whose job is to take in or give out money in a store, bank, etc.
6. Cleaner- a person whose job is to clean something
7. Clerk- a person whose job is to keep track of records and documents for a business or office
8. College professor- a teacher at a university, college, or sometimes a secondary school
9. Construction worker6- a worker skilled in building offices, homes, etc.
10. Cook- someone who prepares and cooks food for eating at a home, in a restaurant, etc.
11. Customer service7- a person who provides service to customers before, during, and after a purchase
12. Day care worker- a person who supervises and cares for children or disabled adults during the day
13. Dental hygienist- a licensed dental professional who cleans and examines teeth
14. Dentist- a person whose job is to care for people's teeth
15. Designer- a person who creates and often produces a new product, style, etc.
16. Doctor- a person who is trained and licensed to treat sick and injured people
17. Education administrator8- a person whose job is to manage a school
18. Electrician- a person who works on and repairs electrical equipment
19. Engineer- a person who has scientific training and who designs and builds complicated products,
machines, systems, or structures
20. Executive- a person who manages or directs other people in a company or organization
21. Factory worker9- a worker in a mill or factory
22. Farmer-a person who runs a farm
23. Firefighter- a member of a group of people who works to put out fires
24. Grounds worker10- a person who maintains gardens and lawns

Merriam-Webster online dictionary (2014). http://www.merriam-webster.com/.
“Construction worker” was not found in the Merriam-Webster online dictionary. A definition was adapted from
dictionary.com (http://dictionary.reference.com/browse/construction+worker).
7
“Customer service” was not found in the Merriam-Webster online dictionary. A definition was adapted from Wikipedia
(http://en.wikipedia.org/wiki/Customer_service).
8
“Education administrator” was not found in the Merriam-Webster online dictionary. The definition was adapted for this
job title.
9
“Factory worker” was not found in the Merriam-Webster online dictionary. A definition was adapted from dictionary.com
(http://dictionary.reference.com/browse/factory+worker).
10
“Grounds worker” was not found in the Merriam-Webster online dictionary. A definition was adapted from Ask.com
(http://www.ask.com/question/what-is-a-ground-worker).
6

2015 NAEP-ECLS Special Study
Cognitive Interview
25. Health aide- a trained and certified health-care worker who provides assistance to a patient in the
patient’s home with personal care (such as hygiene and exercise) and light household duties (such as
meal preparation) and who monitors the patient’s health condition
26. Helper11- a person who helps or gives assistance, support, etc.
27. Housekeeper- a person whose job is to manage the cooking, cleaning, etc., in a house.
28. Janitor- a person who cleans a building and makes minor repairs
29. Lawyer- a person whose job is to guide and assist people in matters relating to the law
30. Librarian- a person who is in charge of or works in a library
31. Manager- someone who is in charge of a business, department, etc.
32. Marketing- someone who is involved in making people aware of a company’s products, making sure that
the products are available to be bought, etc.
33. Mechanic- a person who repairs machines (such as car engines) and keeps them running properly
34. Medical technician12- a person who is trained to give medical care to patients (not a doctor or a nurse)
35. Military- members of the armed forces (such as the army, navy, marines, and air force)
36. Miner13- a person who digs to find and collect coal, gold, diamonds, etc.
37. Nurse- a person who is trained to care for sick or injured people and who usually works in a hospital or
doctor's office
38. Office support14- a person employed to aid an executive, as in a corporate department, by coordinating
such office services and procedures as the supervision, maintenance, and control of the flow of work
and programs, personnel, budgeting, records, etc., for the entire department
39. Painter- a person who applies paint to buildings and walls
40. Pharmacist- a person whose job is to prepare and sell the drugs and medicines that a doctor prescribes
for patients
41. Police- a person who makes sure that rules and regulations are being followed
42. Receptionist- a person whose job is to help the people who call or enter an office, hotel, etc.
43. Religious worker15-a person who conducts religious worship and duties, such as a nun, monk, priest,
minister, or pastor)
44. Repair technician- a person whose job is to repair things
45. Salesperson- a person whose job is to sell things
46. Scientist- a person who is trained in a science and whose job involves doing scientific research or solving
scientific problems
47. Secretary- a person whose job is to handle records, letters, etc., for another person in an office
48. Security guard16- a person employed by a bank, airport, office building, etc., to help keep buildings and
people safe
49. Social worker- someone who works for a government or private organization that helps people who
have financial or family problems
50. Software developer17- A person who researches, designs, creates, and tests computer software
11

“Helper” was not found in the Merriam-Webster online dictionary. A definition was adapted from dictionary.com
(http://dictionary.reference.com/browse/helper).
12
“Medical Technician” was not found in the Merriam-Webster online dictionary. A definition was adapted from
dictionary.com (http://dictionary.reference.com/browse/emergency+medical+technician).
13
“Miner” was not found in the Merriam-Webster online dictionary. The definition of “mine” was adapted for this job title.
14
“Office support” was not found in the Merriam-Webster online dictionary. The definition of “administrative assistant”
was adapted from dictionary.com (http://dictionary.reference.com/browse/administrative+assistant).
15
“Religious worker” was not found in the Merriam-Webster online dictionary. A definition was adapted from
USEmbassy.gov (http://swaziland.usembassy.gov/religious_workers.html).
16
“Security guard” was not found in the Merriam-Webster online dictionary. A definition from dictionary.com
(http://dictionary.reference.com/browse/security+guard) was used for this job title.

2015 NAEP-ECLS Special Study
Cognitive Interview
51. Stocker- a person whose job is to put products on a store’s shelves and display cases
52. Supervisor18- a person in charge of a someone or something, such as a business, government, or school
53. Surgeon- a doctor who performs operations that involve cutting into someone’s body in order to repair
or remove damaged or diseased parts
54. Surveyor- someone whose job is to measure and examine an area of land
55. Taxi driver19- a person whose job is to drive a taxi .
56. Teacher- a person whose job is to teach students about certain subjects
57. Therapist- a person who helps people deal with mental or emotional problems by talking about those
problems
58. Transportation- the act or process of moving people or things from one place to another
59. Truck driver20- a person whose job is to drive a large truck (for example, an eighteen-wheeler) to
transport goods and materials over land
60. Veterinarian- a person who is trained to give medical care and treatment to animals
61. Server - a person who serves food or drinks to people in a restaurant
62. Writer- a person whose work is to write books, poems, stories, etc.

“Software developer” was not found in the Merriam-Webster online dictionary. A definition was adapted from Wikipedia
(http://en.wikipedia.org/wiki/Software_developer).
18
“Supervisor” was adapted from the Merriam-Webster online dictionary.
19
“Taxi driver” was not found in the Merriam-Webster online dictionary. The definition was adapted from the definition
“driver.”
20
“Truck driver” was not found in the Merriam-Webster online dictionary. A definition was adapted from Wikipedia
(http://en.wikipedia.org/wiki/Truck_driver).

File Type	application/pdf
File Title	Microsoft Word - Appendix C.doc
Author	JOConnell
File Modified	2014-07-31
File Created	2014-07-31