Appendix F NAEP Main 2015 Wave 2 NIES CogLab Report

Appendix F

NIES Cognitive Interview Reports 

NAEP Item Development (ID)
Survey Questionnaires
Cognitive Interview Study Report and
Recommendations for Operational
2015 National Indian Education Study
Student, Teacher, and School Questionnaires
Grades 4 and 8

Deliverable in response to ID Task 3.2.3
Submitted: May 2, 2013

Listening.
Learning.
Leading.

Table of Contents
Cognitive Interviews Report for NIES CogLab ............................................................................................... 3
Sample........................................................................................................................................................... 3
Procedures .................................................................................................................................................... 3
Analytic Approach ......................................................................................................................................... 4
Survey Instrument: Student Grade 4/Sequence I ......................................................................................... 5
Survey Instrument: Student Grade 4/Sequence II ...................................................................................... 33
Survey Instrument: Student Grade 8/Sequence I ....................................................................................... 57
Survey Instrument: Student Grade 8/Sequence II ...................................................................................... 83
Survey Instrument: Teacher Grade 4/Sequence I ..................................................................................... 110
Survey Instrument: Teacher Grade 8/Sequence I ..................................................................................... 133
Survey Instrument: Administrator Grade 4/Sequence I ........................................................................... 165
Survey Instrument: Administrator Grade 8/Sequence I ........................................................................... 187

Note: This report was prepared by Kauffman & Associates, Inc. (submitted to ETS on April 28, 2014). For
each item, an ETS recommendation was added to the report based on an evaluation of the findings
summarized by Kauffman. ETS recommendations also consider overall student burden, in addition to
item performance in cognitive interviews. ETS recommendations are highlighted in yellow. For more
detailed information on suggested item wordings and overall item counts, please see RoD.

Prepared by Kauffman & Associates, Inc.

ETS CogLab Report: NIES|2015

Cognitive Interviews Report for NIES CogLab
Date of Report:

April 28, 2014

Interview Dates:

December 11, 2013 – March 18, 2014

Location of Interviews: Arizona, California, Idaho, Maryland, Minnesota, Mississippi, Nevada, New
Mexico, North Carolina, Oklahoma, Oregon, South Dakota, and Washington.

Sample
Kauffman & Associates, Inc. (KAI) recruited a total of 159 individuals for participation in cognitive
interviews during which participants completed the background questions for the 2015 National Indian
Education Study (NIES). The demographics of the participants were as follows:
Table 1: Sample Population Demographics

Respondent Category

Total

4th Grade Students

49

8th Grade Students

44

4th Grade Teachers

22

8th Grade Teachers

20

4th Grade Administrators

12

8th Grade Administrators

12

Procedures
As part of the cognitive interview recruitment process, KAI attended five conferences with
representation from across the United States from October to December 2013. KAI’s outreach strategy
included sending emails to conferences attendees who indicated an interest in participating in, or
knowing more about, the cognitive labs after OMB approval had been granted and working through
these initial contacts to assess potential participants.
In addition to leveraging contacts made during conferences, KAI also engaged in direct outreach to
directors of tribal education departments and Title VII Directors around the United States. Through
ETS Post-Cognitive Interview Item Recommendations

Page 3

ETS CogLab Report: NIES|2015
these contacts, KAI secured the interest of schools who were asked a series of eligibility questions using
the sampling guidelines agreed upon between the National Center for Education Statistics, KAI, and ETS.
Schools were additionally asked to have consent forms signed by parents or guardians prior to KAI
interviewers arriving at the school.
At the end of the sessions, KAI staff asked participants to reflect on their overall impressions of the
cognitive interview. KAI interviewers then thanked and remunerated the participant who was then
asked to sign a receipt for their incentive payment.
The cognitive interviews were scheduled for a duration of 60 minutes. The items for both 4th and 8th
grade student questionnaires were split into two sequences per grade level and administered to at least
20 students per sequence. All 4th grade teachers received the same sequence and all 8th grade teachers
received the same sequence. Further, all 4th grade administrators responded to the same sequence and
all 8th administrators responded to the same sequence.

Analytic Approach
This report presents findings directly associated with the 159 cognitive interviews conducted by KAI. The
analysis is based upon both the responses given par interview participants as well as from the
observances from KAI interview staff regarding the comfort and demeanor of participants throughout
the interview process.
The analysis of the data collected was qualitative in nature and therefore based on drawing out themes
from participant responses and counting frequency of response occurrence as well as reporting out on
anomalous data. Due to the nature of qualitative data sometimes response rates equal more than the
number of respondents, particularly if multiple themes were present in a single participant’s response.
Further, several probe questions in the interview instruments were conditional so not all respondents
answered; some respondents chose not to respond to certain questions; and there are inevitable
skipped questions at times where interviewers missed asking a question.

ETS Post-Cognitive Interview Item Recommendations

Page 4

ETS CogLab Report: NIES|2015

Survey Instrument: Student Grade 4/Sequence I
Student #1

New/VH039733

Who taught you most of what you know about American Indian or Alaska Native history?
A. No one has taught me about American Indian or Alaska Native history.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 1
Reading: Two respondents had some trouble reading the item. One misread “please specify” as “please
speak clearly,” and another had trouble with the words “who,” “representatives,” and “specify.”
Understanding: Two respondents seemed to have trouble with “Alaska”; one said they didn’t
understand what Alaska was, another said they didn’t understand what an Alaska Native was, and a
third kept referring to Africa though they seemed to be referring to Alaska. A fourth respondent seemed
confused by what “AI/AN history” meant.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Four students out of 23 respondents seemed to misunderstand the question based on answers ranging
from “I don’t know” to non sequitur responses to the question; the other 19 clearly restated the
question.
Probe 3: What does the phrase “American Indian or Alaska Native history” mean to you?
Can you give me examples of “American Indian or Alaska Native history” that you have learned?
There was confusion for the 22 respondents around Probe 3 with one respondent speaking to issues in
Africa (ostensibly meaning Alaska), and another asked what Alaska and history were. Answers that
seemed out of place were from two respondents who answered “I don’t know,” one who responded
“pretty much nothing,” and two others who made references to language as AI/AN history. The other 15
answers demonstrated a clear understanding of the question.
Probe 4: In answer choice B, what does the phrase “Family members” mean to you?
Of the 22 respondents for this question, most noted specific family members; several spoke to broader
relationships of interdependence, for instance saying that the tribe was their family.

ETS Post-Cognitive Interview Item Recommendations

Page 5

ETS CogLab Report: NIES|2015
Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you?
Can you give me examples of “Tribal representatives or elders”?
“Elders” was relatively easy for the 22 respondents to pinpoint and identify as people older than
themselves or with more knowledge. “Tribal representatives” presented a bit of a challenge for most
respondents who either ignored the question or said that they did not know what a tribal representative
was. Many respondents also conflated the terms of “tribal representative” and “elder,” perhaps due to
the fact that they were presented in one question together.
Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
The 10 responses to this question were inconsistent and questionable with regard to their validity. One
respondent felt that their father was a tribal representative as he worked at the college. Another noted
that their grandfather was very involved in powwows, though his position was never clarified. Of the six
respondents who replied in the affirmative to this question, only the two aforementioned respondents
gave specific reasons as to why they considered their family members tribal representatives. Two
indicated that they did not have family members who were tribal representatives.
Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
Only one respondent chose answer choice F and named their parents as the “someone else” who had
taught them most of what they knew about AI/AN history.
Probe 8: What does the phrase “most of what you know” mean to you?
Out of 23 respondents, one said “nothing,” one responded “I don’t know,” and another said, “I don’t
understand this question.” All other (21) responses were appropriate and valid.
Probe 9: Which answer choice would you select if multiple people had taught you about American
Indian or Alaska Native history? How would you decide which answer choice to select?
The validity of the answers proffered a lot of variety among the 18 respondents. Seven suggested they
would choose a different answer entirely. None appropriately responded to being asked how they
would decide the best answer in a hypothetical situation, though five gave explanations as to why they
chose their original answer.

Solutions to Improve Item
Respondent: NA
Interviewer: Interviewers consistently noted that hypothetical questions (such as Probe 9) posed a
problem for many respondents, and this was particularly true for the 4th grade interviews.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Respondents generally seemed to understand and be comfortable with answering the question.

ETS Post-Cognitive Interview Item Recommendations

Page 6

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 7

ETS CogLab Report: NIES|2015
Student #3

New/VH040030

Who taught you most of what you know about American Indian or Alaska Native arts and
crafts?
A. No one has taught me about American Indian or Alaska Native arts and crafts.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 3
Reading: One respondent misread “please specify” as “please speak clearly”; another skipped the word
“Native” in Alaska Native.
Understanding: Only one student said that they did not know what the question was asking.
Changing Answers: Of 23 respondents, two chose multiple answer choices for the item.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Only one student out of 22 respondents reported that they did not know what the item was trying to
ask.
Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you?
Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have
learned?
Twenty of the 23 students gave valid responses to the questions. One student said that they did not
know, another said that this question was the same as the last, and a third student referenced
storytelling—which could be considered an art form but might also indicate some conflagration with the
previous question about history.
Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
This probe seemed to be confusing for the five students who responded to this probe. Answers ranged
from simply stating one family member to mentioning several. One student recounted, “I don’t really
know because when my grandma still has to take me to my other family members too.”

ETS Post-Cognitive Interview Item Recommendations

Page 8

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of respondents seemed to understand the item question and answer it
appropriately.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 9

ETS CogLab Report: NIES|2015
Student #8

New/VH040093

How much do you know about American Indian or Alaska Native systems of counting?
A. Nothing
B. A little
C. Some
D. A lot

Item 8
Reading: NA
Understanding: Given the inconsistency of responses across the various probes, it would seem that
most respondents did not truly understand what this question was asking as there was no clear
explanation for what respondents interpreted “systems of counting” to mean.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Two out of 23 respondents interpreted the question as having to do with whether or not one could
count in an AI/AN language. Fourteen students essentially repeated the question, though one thought
that perhaps the question had to do with the invention of computers. Seven respondents essentially
reported that they did not know what the question was asking; one of the seven did not even choose an
answer for the item due to not understanding.
Probe 3: What does the phrase “American Indian or Alaska Native arts and crafts” mean to you?
Can you give me examples of “American Indian or Alaska Native arts and crafts” that you have
learned?
Eight students of 22 respondents indicated that they did not know what AI/AN “systems of counting”
meant. Four thought it had to do with whether or not one could count in a Native language. Three felt it
had to do with how AI/AN people kept track of their populations. And four students essentially repeated
the words of the question. Other responses varied in their guesses to the meaning.
Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Of the 13 respondents who answered, five students said they did not know, two thought it had to do
with language, one thought it had to do with populations, one referenced basket weaving, at least two
answers were confusing regarding what the respondent was trying to express, one respondent spoke
about technology 50 years ago, and one respondent talked about their teacher having them count on
their hands.

ETS Post-Cognitive Interview Item Recommendations

Page 10

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
All 20 respondents who provided answers found the choices clear. One student noted that although
they found the answers clear, A was the exception, saying “I don’t know that if anybody knows or not.”

Solutions to Improve Item
Respondent: NA
Interviewer: Through team discussions, this question came up often as one that was confusing both to
students as well as to interview staff. No one seemed to be clear about what AI/AN systems of counting
was referring to, something that should be clarified if this question is to be used in the 2015 NIES study.

Initial KAI Recommendation (Keep/Change/Discard)
Discard: If there is not a clear definition of what systems of counting entails, then the recommendation
is to drop this question.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS
recommends revising the phrase to "ways of counting".
This recommendation is aligned with the similar grade 4 item (VH040103).

ETS Post-Cognitive Interview Item Recommendations

Page 11

ETS CogLab Report: NIES|2015
Student #9

New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of
counting?
A. No one has taught me about American Indian or Alaska Native systems of counting.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 9
Reading: NA
Understanding: It is hard to say how well students truly understood the question as many of them
repeated the question. Some claimed they did not know what the question was trying to ask, some
thought it had to do with language, others thought it was referencing AI/AN history or culture, others
thought it referred to general teaching (not specific to a particular topic), and one thought it was the
same as the previous question.
Changing Answers: No one changed their answers, but one student chose two answers.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Two out of 23 responding students said that they did not know what the question was asking, 11
essentially parroted back the question to the interviewer, two students felt that it had to do with being
able to count in a Native language, four thought the question was referring to AI/AN culture or history,
three seemed to think that the question had to do with who had taught them the most (did not seem
specific to AI/ANs), and one student asked if this was a repeat of the previous question.
Probe 3: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]:
Do you have any family members who are also tribal representatives? If yes, do you
first think of them as your family members or as tribal representatives?
Only one student was eligible to answer this conditional probe and they named their grandmother and
uncle as tribal representatives but did not offer what their positions in the tribe were.
Probe 4: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?

ETS Post-Cognitive Interview Item Recommendations

Page 12

ETS CogLab Report: NIES|2015
Only two students were eligible to answer this conditional probe. The first said “the people that come
to our school and teach us about it. I have no idea what they’re called. Most of them are dancers.” The
other replied that the “someone else” they were thinking of was their grandma.

Solutions to Improve Item
Respondent: NA
Interviewer: Again, a specification about what is meant by “systems of counting” would be critical in
making the meaning of this question clearer.

Initial KAI Recommendation (Keep/Change/Discard)
Discard: Given the current confusion about the meaning of the phrase, it is the suggestion of the
interviewers to discard the question.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS
recommends revising the phrase to "ways of counting".
This recommendation is aligned with the similar grade 4 item (VH040093).

ETS Post-Cognitive Interview Item Recommendations

Page 13

ETS CogLab Report: NIES|2015
Student #10

New/VH040098

How much do you know about American Indian or Alaska Native symbols and designs used in
math?
A. Nothing
B. A little
C. Some
D. A lot

Item 10
Reading: One student misread “systems” as “single.”
Understanding: Six respondents were clearly confused about what the question was trying to ask and
either said that they did not know, or gave responses that did not make sense in the context of the
question.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Responses varied among the 23 respondents, one said they did not know, one said “no” and 21 others
repeated sections of the question but did not seem to grasp what the question was trying to elicit.
Probe 3: What does the phrase “American Indian or Alaska Native symbols and designs” mean to you?
Answers from the 23 respondents varied greatly to this question. Only one respondent mentioned math
in their response. Other answers ranged from punctuation to arts and crafts. Five students expressed
not knowing what the question was trying to elicit, and the rest gave answers that have very little
validity as they were widely disparate in their interpretation of what was meant by “symbols and
designs,” and as noted before, only one student tied their answer back to math.
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of “American Indian or Alaska Native symbols and designs” that
you know about?
Only three of the 13 respondents referenced math, one noting that they were unsure how symbols and
designs were used in math but did point out how they may be used in decoration or communication;
and the other giving the following answer: “American Indian, basically my whole class, we just know that
when you write the symbols in math that if you’re from somewhere else, it’s a new kid you don’t know
and they could be from anywhere. If they don’t know how to do their Indian math that there would be a
whole class and they would have to decide how would you write the math symbols down.” Six gave
examples referring to arts, crafts, and modes of communication, one thought the question had to do
with punctuation, one referenced animals, and four claimed that they did not know.

ETS Post-Cognitive Interview Item Recommendations

Page 14

ETS CogLab Report: NIES|2015
Probe 5: Do you think that answer choices A through D are clear? If not, can you give me examples of
other answer choices that would be more clear?
Eighteen of 19 respondents felt the answers were clear; one felt that they weren’t and gave the
following explanation: “No. If they make them like, the X for times, the plus for plus or add, and
subtract, a minus. A minus with two on top and on bottom of it for division. That's how we do it in
English.”

Solutions to Improve Item
Respondent: One respondent made the following suggestion: “I think they could’ve asked it more
clearer because it’s not all just used in math, it’s used in different stuff, so maybe they could have just
said, ‘How much do you know about American Indian or Alaska Native symbols and designs used in
anything?’ maybe.”
Interviewer: Perhaps the category of symbols and designs could be expanded beyond math as most
students seemed to think about symbols and designs as decoration or communication.

Initial KAI Recommendation (Keep/Change/Discard)
Change: Given how most students interpreted the question having to do with arts, crafts, or forms of
communication, perhaps the question could ask about that. The relevance to math made little sense to
respondents and if this is the only purpose of the question, it is the suggestion of the interviewer team
that it be removed from the study.

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS agrees with KAI that this item should be dropped as the specific relevance to math made
little sense to respondents.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 15

ETS CogLab Report: NIES|2015
Student #11

New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and
designs used in math?
A. No one has taught me about American Indian or Alaska Native symbols and designs
used in math.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 11
Reading: NA
Understanding: Three respondents were noted as having difficulty understanding the item due to their
responses; one seemed to be equating symbols and designs with punctuation, one thought the question
referred to history, another felt the question had to do with whether or not American Indians or Alaska
Natives had taught them more about symbols and designs in math.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
One student out of 22 respondents referenced using math in a Native language, one said they did not
know, and 17 essentially repeated the question in slight different terms. Three gave unusual answers
ranging from symbols and designs having to do with AI/AN history, or whether or not American Indians
or Alaska Natives had taught them more, and another student made a reference to gestures.
Probe 3: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Only one student responded to this question on this item and specified their mother as a tribal
representative in the following: “There's my mom. I don't know what she is. She likes to file papers and
she's getting people's license or something like that. She's always busy all the time. Sometimes I can
barely even see her.”
Probe 4: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
NA
ETS Post-Cognitive Interview Item Recommendations

Page 16

ETS CogLab Report: NIES|2015
Solutions to Improve Item
Respondent: NA
Interviewer: Again, knowing the purpose and/or what this question is trying to ask would be very
helpful.

Initial KAI Recommendation (Keep/Change/Discard)
Discard: Given how the question currently stands, the recommendation is to discard it.

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting
the phrase "symbols and designs", and the specific reference to math caused additional confusion.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 17

ETS CogLab Report: NIES|2015
Student #12

Revised/VH040119

During 4th grade, have you attended assemblies or presentations in your school given by
elders or leaders from an American Indian or Alaska Native tribe or group about traditions and
cultures (ways of life, customs)?
A. Yes, once or twice
B. Yes, three or more times
C. No

Item 12
Reading: Two respondents had trouble with the word “assemblies”; of these two, one had trouble with
the word “traditions,” and the other had difficulty with the word “elders.”
Understanding: Two respondents were reported by interviewers as having trouble understanding the
question due to either difficulty with words in the question.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Sixteen of 22 respondents repeated some variation of the item question, four respondents gave
somewhat confusing answers such as “Have you ever met them?” and “Have I done everything I was
told to do?” and an explanation about “stories from a long long time ago.” Additionally there was one “I
don’t know” and one non-response.
Probe 3: What time frame were you thinking about when you answered this question?
Very little agreement or consistency in the 20 respondents’ answers to this question. Answers rarely
overlapped and included responses such as, “one to two times a year,” “two to three days,”
“December,” “four years,” “all school year,” “all years in school,” “five months,” “5 minutes,” “a week,”
and “a semester.” The most confounding was “When I was a baby. Yeah. It’s so dark.”
Probe 4: What does the phrase “assemblies or presentations” mean to you? Can you give me
examples of “assemblies or presentations”?
Half (11) of the 22 respondents gave broad definitions of what programs, presentations, and assemblies
consisted of, and about six of the students cited specific examples of assemblies or presentations they
had been to as a way to explain what assemblies and presentations meant to them. Only two of the
respondents said that they did not know, and seemingly non sequitur answers were limited to three:
“I don't think that much, I don't know that much. [Interviewer prompts] About learning traditions and
cultures and the way we live in right now and how did they create it before the date of 2014 out of
1614.”
“Like programs. [Confusion over what we were trying to ask, mentioned singing in Chapel Hill and having
activities at Grandma's house.]”
ETS Post-Cognitive Interview Item Recommendations

Page 18

ETS CogLab Report: NIES|2015
“It means that it's kind of telling me a story.”
Probe 5: What does the phrase “elders or leaders from an American Indian or Alaska Native tribe or
group” mean to you? Can you give me examples of “elders or leaders”?
The 21 responses to this probe yielded the following kinds of answers. About 10 respondents gave
either specific examples or provided explanations about what they thought made an elder or leader,
four respondents mentioned specific family members, three spoke about people coming to give
presentations, three said that they did not know, and two gave answers difficult to qualify.
Probe 6: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up
question]: How many times have you attended these assemblies or presentations in your school?
Given the responses to Probe 4 it is difficult to say the precise time frame that respondents were
thinking about for this question. The 15 who provided responses broke into two broad categories. Eight
students mentioned attending assemblies or presentations three times or more, and four said that they
had attended either one or two times.
Probe 7: [Conditional probe: If the participant selects answer choice C, then ask the follow-up
questions]: Is the reason you have not attended assemblies or presentations during 4th grade because
they are not available? If not, please explain why you have not attended assemblies or presentations
during 4th grade.
Of the seven students responding, three said that assemblies were not available at their school or
would be occurring later on in their schooling, two respondents said they had been to assemblies but
there had not been any elders or leaders present, and two respondents gave non sequitur answers
indicating that they did not fully understand the question.
Probe 8: Do you think that answer choices A and B are clear? If not, can you give me examples of
other answer choices that would be more clear?
Fourteen of 17 respondents felt that the answer choices were clear, one said that they did not know,
and two said that the answer choices were not clear.

Solutions to Improve Item
Respondent: One respondent felt that there should be another answer choice to reflect whether or not
someone had attended more than three times. Another felt that more specific kinds of answer choices
would be better.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: There seemed to be a lot of variance in answers to the probes related to this item. Perhaps
bolding “4th grade” would help to ensure that students are answering for that time period. From
responses to Probe 4 there seemed to be some agreement that assemblies required people coming
together to see a presentation that someone else is giving, whereas “presentation” seemed to be
interpreted as something that students might do in a classroom setting. This may indicate that using the
term “presentation” may be confusing if the item seeks to parse out whether or not students are
ETS Post-Cognitive Interview Item Recommendations

Page 19

ETS CogLab Report: NIES|2015
attending assemblies (which would likely involve dancers, etc.) or presentations given by leaders/elders.
Another point is the use of “leaders/elders” as they may not be the ones leading presentations or
assemblies about AI/AN cultures (for instance, given the example of dancers coming to the schools).

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS agrees with KAI that this item should be dropped given the large degree of variance in
the probe responses. Also, there seems to be a cognitive overload with this item for a 4th grader.

ETS Post-Cognitive Interview Item Recommendations

Page 20

ETS CogLab Report: NIES|2015
Student #17

Revised/VH040212

When my teacher talks about American Indian or Alaska Native history or culture, I try to read
more about it.
A. This is not like me.
B. This is a little like me.
C. This is a lot like me.

Item 17
Reading: One student misread the word “culture.”
Understanding: One student seems to be interpreting AI and AN as two separate issues or questions
within the item.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Seventeen of 22 respondents repeated some version of the item question, two interpreted the question
as asking them what they knew, two simply mentioned AI/AN culture, and one gave the following
response, “You know what you're doing about yourself that you know is explaining about you.”
Probe 3: What does the phrase “when my teacher talks about” mean to you?
Sixteen of 22 respondents made reference to their teachers talking or teaching; within this group the
topics were either not specified or were seen as revolving around AI/ANs (five students mentioned this),
history (two students mentioned this), art (one student), or bullying (one student). Three students said
that they did not know.
Probe 4: Which answer choice would you select if your teacher had not talked about American Indian
or Alaska Native history or culture? Why would you select this answer choice?
Fifteen of 22 respondents felt that they would choose a different answer if their teacher had not talked
about AI/AN history or culture in class, many of them citing that if their teacher had not talked about it
then they probably would not know about the topic. Seven students felt that their answer would stay
the same.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Respondents overall seemed to understand what the item was asking.
ETS Post-Cognitive Interview Item Recommendations

Page 21

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 22

ETS CogLab Report: NIES|2015
Student #19

New/VH040276

I want to be one of the best students in my class.
A. This is not like me.
B. This is a little like me.
C. This is a lot like me.

Item 19
Reading: NA
Understanding: Respondents seemed to have a firm grasp on what this question was trying to ask.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eighteen out of 21 respondents provided responses that were in keeping with what the question was
asking with some variation on what “being the best” meant to them—from getting good grades to being
the best reader to being well-behaved. Three respondents interestingly gave responses with a negative
take on being one of the best in the class, referencing greed, showing off, and making others feel bad
about themselves.
Probe 3: What does the phrase “one of the best students” mean to you?
This probe elicited interesting responses from the 22 respondents who provided answers. Eight
referenced having good behavior, six students used the word “best” to talk about what being “one of
the best students” meant to them, six mentioned knowing more than others or being smart, five talked
about grades, three simply noted being “better” than others, two discussed popularity, one said it
meant being fast with schoolwork, and one said that they did not know.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Question seemed easily understood and was being interpreted as intended by the majority of
students.

ETS Recommendation (05/02/2014)
Keep this item.

ETS Post-Cognitive Interview Item Recommendations

Page 23

ETS CogLab Report: NIES|2015

This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 24

ETS CogLab Report: NIES|2015
Student #20

New/VH040379

I enjoy being challenged in my classes.
A. This is not like me.
B. This is a little like me.
C. This is a lot like me.

Item 20
Reading: NA
Understanding: All students seemed to understand the gist of the question, though two seemed to
interpret “challenge” with negative connotations.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Answers to this question seemed fairly consistent. Ten of 21 respondents essentially repeated the
question, two mentioned that “being challenged” meant answering hard questions, two noted that
challenge meant not doing easy things, two specifically mentioned math, one student pointed to doing
independent work, and one noted learning new things. Only two responses seemed a little out of the
norm; one student stated “It’s telling me they like challenges,” and the other referenced staring
contests.
Probe 3: What does the phrase “being challenged” mean to you? Can you give me examples of you
being challenged in your classes?
Thirteen of 22 respondents noted that “being challenged” to them had to do with being pushed to do
harder work, usually by their teachers. Three specifically mentioned math in their responses, two talked
about tests, two referenced competitions (one of these respondents was the one who spoke of staring
contests), and two talked about tests.

Solutions to Improve Item
Respondent: NA
Interviewer: One of the interviewers felt that there could be more emphasis on this question being
related to schoolwork—they had the interview with the student who used staring contests as an
example of challenges.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of respondents understood the question and interpreted as intended.

ETS Post-Cognitive Interview Item Recommendations

Page 25

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 26

ETS CogLab Report: NIES|2015
Student #22

Trend/VC759291

How often does someone else who lives in your community or is a friend of your family help
you with your schoolwork? For example, they might help you to study for a test, help you with
a school project, or go over your homework with you.
A. Never or hardly ever
B. Once or twice a month
C. Once or twice a week
D. Every day or almost every day

Item 22
Reading: NA
Understanding: Most respondents seemed to understand that the item wanted to know if they were
getting help on their schoolwork; the piece about it being “someone else who lives in your community
or is a friend of your family” was not necessarily interpreted as it should have been. Students seemed to
latch onto different words in the bolded part of the statement like friends, family, and community, but
did not necessarily tie them together.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Respondents generally understood that the question had to do with receiving help on one’s homework
with 12 out of 22 respondents referencing that piece of the question specifically. Nine mentioned family
providing help, five referred to friends helping, five mentioned community members in their responses,
one spoke of a tutor, and one referenced a friend of the family. Three answers were somewhat
confusing; one student mentioned their teacher providing examples, another seemed to think the
question had to do with them helping their friends as well as being helped by their own family, and one
student interpreted the question as having to do with how much time they spent on homework.
Probe 3: What does the phrase “someone else who lives in your community or is a friend of your
family” mean to you?
Nine of 22 respondents spoke about friends of the family as their contemporaries in age, rather than as
an adult who was a friend of the family. Additionally nine responses referenced family specifically from
mothers/fathers, brothers/sisters, aunts/uncles, grandparents, and cousins. At times the cousins were
noted also as friends. Of the seven responses directly mentioning the idea of friends of the family,
friends were often noted as of contemporary age with the respondent and cousins were also often
mentioned. Three students said that they did not know what was meant by “someone else who lives in
your community or is a friend of the family.”

ETS Post-Cognitive Interview Item Recommendations

Page 27

ETS CogLab Report: NIES|2015
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of how these people have helped you with your schoolwork?
Who is this person?
Eleven out of 20 responding students mentioned family members from parents to grandparents and
everyone in between, and six students spoke of friends helping them—most indicated that these friends
were of the same age, though some were slightly ambiguous. Two students mentioned tutors, and three
gave unclassifiable answers.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: The wording of the bolded section seems to be causing confusion for many respondents who
do not seem to be interpreting the question as intended. We suggest revising the question to ask
specifically about adults, friends, or tutors in the community, or asking whether or not family is involved.
Friend of the family/someone else who lives in your community seems to involve a nuanced
understanding that is not being exhibited by the majority of the respondents to this question.

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS recommends dropping this item as it seems to overlap with the previous three trend
items (VC759288, VC759289, VC75929). Having four sequential items about who is helping the student
with his/her schoolwork is excessive; therefore this item can be dropped to make room for a new
item(s). Asking about "someone else who lives in your community/friend of your family" seems to
involve a nuanced understanding that is not being exhibited by the majority of the respondents to this
question.
The similar grade 8 item VC996640 is also proposed to be dropped.

ETS Post-Cognitive Interview Item Recommendations

Page 28

ETS CogLab Report: NIES|2015
Student #23

New/VH040389

How often do you help other students from your school with their schoolwork? For example,
you might help other students to study for a test, help them with a school project, or go over
their homework with them.
A. Never or hardly ever
B. Once or twice a month
C. Once or twice a week
D. Every day or almost every day

Item 23
Reading: NA
Understanding: The grand majority of students seemed to understand what this item was asking.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eighteen out of 22 respondents interpreted the question correctly, one student specifically mentioned
math homework, and another mentioned helping their siblings with school work. Only one of the
respondents seemed to be a bit confused about the angle of the question and seemed to think it was
asking them about whether other students were helping them with course work.
Probe 3: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
questions]: Can you give me examples of how you have helped other students with their schoolwork?
Are the other students that you have helped with their schoolwork in 4th grade? If not, what grade
are the students that you have helped with their schoolwork in?
Ten of 19 respondents mentioned helping “friends” with homework, nine noted that those they were
helping were in the 4th grade, six respondents pointed to helping family members such as siblings and
cousins with homework, and five respondents pointed to working with students in other grades. Two
respondents gave C or D as answers, but when asked for examples, they responded “no.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 29

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep/Change: It seems that there may be some room for confusion on this item depending on what
information is being sought. If the question is simply to capture whether or not students are helping
others with classwork, it is fine as is. If the question specifically wants to know about “other students
from your school,” that may need to be highlighted. Generally, the question is fine as is with students
responding consistently about helping others.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 30

ETS CogLab Report: NIES|2015
Student #25

New/VH040394

Trying hard in school will help me live a good life when I grow up.
A. Strongly disagree
B. Disagree
C. Agree
D. Strongly agree

Item 25
Reading: NA
Understanding: The majority of respondents seemed to grasp the question being posed to them.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eleven of 21 responding students noted the importance of education and mentioned post-elementary
education in their explanation of what the question was asking, seven students essentially repeated the
question using slightly different words, one noted hard work, one mentioned money, and one said that
they did not know what the question was asking.
Probe 3: What does the phrase “trying hard in school” mean to you? Can you give me examples of
what “trying hard in school” would be?
Fifteen of 22 responding students noted that “trying hard in school” meant studying or practicing or
doing your best. Seven mentioned doing well on tests in their responses, four mentioned getting good
grades, three mentioned college, and one mentioned getting help when you need it.
Probe 4: What does the phrase “a good life” mean to you? Can you give me examples of what “a good
life” would look like?
Answers to this probe varied in terms of what themes presented themselves. For instance, nine out of
22 responding students mentioned jobs or “good jobs” when discussing what it meant to have a good
life, five mentioned having a family or being able to help family, four mentioned having money, four
noted having knowledge or knowing a lot, four talked about having a house, three mentioned good
health (ranging from eating well to being drug free), two talked about happiness in general terms
(though one was specific about that having to do with ponies in a mansion), one said they thought it had
to do with having good manners, and one said they did not know.
Probe 5: What does the phrase “when I grow up” mean to you? Can you give me examples of what
being “grown up” means to you?

ETS Post-Cognitive Interview Item Recommendations

Page 31

ETS CogLab Report: NIES|2015
Responses to this probe varied greatly in terms of what students saw as being “grown up.” Seven of 22
responding students mentioned having a job, five said they thought they would be grown up when they
were in their 20s, five referenced education (often referring to college), five said it would be when they
were independent (often referencing not needing parents any more), three said when they were bigger
or taller, three mentioned having a house, two said it would be in junior high, two said when they had a
family of their own, one mentioned being clean, one mentioned having a car, one spoke of good
penmanship, and one said they did not know. One of the respondents summed up being a grown up
with the following, “They go to work every day. They go to dinner. They go to bed. They wake up, and
then they go back to work. That’s it.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: All respondents seemed to understand the question as it was intended.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 32

ETS CogLab Report: NIES|2015

Survey Instrument: Student Grade 4/Sequence II
Student #2

New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions
(way of life, customs)?
A. No one has taught me about American Indian or Alaska Native traditions.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 2
Reading: One of the students who did segments of both sequence 1 and sequence 2 initially thought
they had already read this question.
Understanding: Only one student was reported as not understanding the item and having trouble with
the probes as well.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-six out of 30 responding students essentially repeated the question; four indicated that they did
not understand or did not know what the question was asking.
Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions”
mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you
have learned?
All of the 31 responding students asked this question gave a response, 10 of them mentioned traditions
in their answers ranging from smudging and tobacco use, and one student noted Christmas as a
tradition. Seven students talked about powwows or dancing, four mentioned crafts, three expressed
that they did not know what “Alaska Natives” meant, and other answers involved food, housing, history,
language, and hunting. Six students indicated that they did not know what the question was trying to
ask.
Probe 4: In answer choice B, what does the phrase “Family members” mean to you?
Of the 30 respondents to this question (one was inaudible), 18 students named different members of
their family (for instance, parents or siblings or grandparents). Seven students simply said “family” when

ETS Post-Cognitive Interview Item Recommendations

Page 33

ETS CogLab Report: NIES|2015
asked what “family members” meant to them, four students spoke about relationships of teaching and
learning when talking about what it meant to them, and two mentioned being loved or cheered up.
Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you?
Can you give me examples of “Tribal representatives or elders”?
The 30 responses to this question varied a good bit as it asks about both tribal representatives and
elders. Thirteen students named someone either specifically or in terms of who in their family they
considered to be an elder or tribal representative. These answers ranged from grandparents to teachers
to family friends. Nine students noted that tribal representatives or elders were “old people,” six
mentioned teachers—either specific teachers or just people who teach younger people about their
culture. Five students said that they were people who represented the tribe, four said they did not
know, two simply responded “no,” and two respondents had difficult-to-categorize answers—“that
means they help me get yeast and stuff and all that.”
Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
The validity of responses to this question vary greatly given that very few students seemed particularly
sure of what a tribal representative was. Of the 16 respondents who provided answers, eight indicated
that they did have a family member who was a tribal representative (often giving answers like “family
members”), four indicated they did not, two said maybe, and one said they did not know, and another
student gave a hard-to-characterize answer—one said “elders.”
Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: Separate out elders and tribal representatives in probes to better determine how students
understand each of these categories.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It would seem that the majority of the students asked this question understood what the question
wanted to know about who had taught them what they knew about AI/AN traditions.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 34

ETS CogLab Report: NIES|2015
Student #4

New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska
Native language?
A. My school does not offer courses that are taught in an American Indian or Alaska
Native language.
B. Never or hardly ever
C. Once or twice a month
D. Once or twice a week
E. Every day or almost every day

Item 4
Reading: No difficulties reading the item.
Understanding: A few of the respondents seemed to have some issues with fully understanding the item
without some redirection from interviewers. One student seemed very confused by the term “Alaska
Native,” two others seemed to have a hard time understanding the broader question and got bogged
down in the idea of specific vocabulary or attendance overall.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of 31 responding students mentioned language in their restatement of the question, five
students did not mention language but rather seemed to interpret the question as having to do with
AI/AN culture, five seemed to be confused by the question—one stating, “What time do I arrive at
school”—and one student simply noted that they do not learn anything about AI/ANs in their school as
they are one of five students of AI/AN heritage in the entire school.
Probe 3: What does the phrase “American Indian or Alaska Native language” mean to you? Can you
give me an example of “American Indian or Alaska Native language” that you can speak or know
about?
Nineteen of 31 students responding to this probe referenced themes having to do with how people
spoke in the past; some of these students even referenced language being spoken by elders in their
community. Eight students mentioned specific AI/AN languages, three talked about being taught or
learning other languages, two students said “no,” and one gave an answer having to do with their
grandmother that strayed from language and into horseback riding.

ETS Post-Cognitive Interview Item Recommendations

Page 35

ETS CogLab Report: NIES|2015

Probe 4: How many American Indian or Alaska Native languages do you speak?
Eleven of 31 responding students reported speaking at least one AI/AN language, nine said they spoke a
little of one or two languages, nine said they did not speak any AI/AN languages, one student said they
spoke two languages, and one said they did not know.
Probe 5: How would you answer this question if you speak more than one American Indian or Alaska
Native language?
This probe drew varied answers from the 27 respondents. Five students indicated that they would
change their answer to D. Five gave answers that were difficult to characterize from “Talk like America”
to “If I could speak more than one American Indian languages, I would go lots of places” to “I speak
Spanish, Chinese—I say hello in Chinese. I can speak Italian.” Four said they did not know, and four gave
the answer B, but at least two of these indicated they thought that meant “once or twice a week.” It is
hard to say if this is a misunderstanding or misreading on the part of the students or if perhaps B was
heard instead of D by the transcriptionist. Three students would change their answer to E, two would
keep their answer as is, and one would change their answer to A (interestingly this participant gave D for
their original answer and indicated that they do in fact speak Lakota).
Probe 6: What is the difference between answer choices A and B?
Fourteen out of 31 responding students gave responses that indicated that they had interpreted the
differences between A and B correctly. Six students gave responses that were difficult to characterize in
that they were unfinished thoughts, or seemed to confuse what the question was asking. For instance,
one student said, “One talks about the Alaska Native language and the other one is for attending class.”
Another said “That usually A, they usually don't teach you much about A. You are very like the entire
language and what you eat and B is probably like you are never really learned anything about your
culture,” and another student had the categories backwards and said “Because B you don’t learn it and
A you do.” Four students said they didn’t know the difference between the two answers and four other
students felt that the two answer were the same. Two students suggested that A referred to learning in
school and B could be outside of school.
Probe 7: If there were no courses offered in your school that are taught in an American Indian or
Alaska Native language, which answer choice would you select?
Thirteen of 31 responding students said they would choose A; two indicated they were choosing this
because their school did not currently have classes in an AI/AN language. Nine said they would choose B.
Four said they would choose D, one said they would choose C, one said they would choose E, and two
indicated they did not know. It seems clear that many of the students did not understand what this
question was trying to ask.

Solutions to Improve Item
Respondent: NA
Interviewer: Hypothetical questions seemed to confuse many students. One actually said “Again, that
seems like another confusing question. I would think that I would answer the same thing but I don't. I

ETS Post-Cognitive Interview Item Recommendations

Page 36

ETS CogLab Report: NIES|2015
just feel like I don't want my answer to be the same.” This led the interview team to question how often
this might have been happening with other probes.

Initial KAI Recommendation (Keep/Change/Discard)
Change: This question generally seemed to cause some confusion for respondents. The item seemed to
make sense to students, but subsequent probes revealed that there was some confusion around the
difference between answers A and B. Some of this may be attributable to the difference in wording of
the two questions, and the description for A is much more detailed than that of the other answer
choices. Our recommendation is to change to be similar to B through E and to highlight in the item the
fact that the question specifically is asking about classes at school (keeping in mind that many students
who may take classes may do so outside of school).

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about
classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer
choice “A”, ETS recommends changing the word “courses” to the word “classes” to be consistent with
the item stem.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 37

ETS CogLab Report: NIES|2015
Student #5

New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?
A. I cannot speak an American Indian or Alaska Native language.
B. I can speak a few words or phrases.
C. I can speak well.

Item 5
Reading: NA
Understanding: Some respondents seemed to think that the question had to do with teaching or
learning the language, and even of those who restated the question in a way that seemed to capture the
intent, there seemed to be a wide interpretation of what the item was actually trying to capture.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-one of the 31 respondents provided a response that was on track with what the question was
trying to ask, but the variations within these answers was interesting. Some students essentially
restated the question with some explanation, and others said things like “Can you speak American
Indian language.” Three students seemed to think that the question had to do with how much they learn
or “how you teach it.” Another three gave answers that were difficult to categorize, ranging from “It
means how do you talk when you try to talk like them” (ostensibly referring to AI/ANs) to “There’s three
different ones and then you pick which one you think you’re at” (likely referring to the fact that they are
being asked to choose between three different answer choices).
Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?
Twenty-five students out of 30 respondents gave answers indicating that they had correctly interpreted
the phrase “I can speak a few words or phrases” as intended. Three students said that they did not
know, and one students said, “It means you can speak a few words in English and stuff. Alaska.”
Probe 4: In answer choice C, what does the phrase “I can speak well” mean to you?
Of the 29 students who responded to this question, 26 seemed to understand the gist behind what “I
can speak well” was trying to elicit, though this occurred in different ways with some students
interpreting it as a generic speaking well or fluently whereas others were more nuanced in their
descriptions, drawing on breadth of vocabulary and pronunciation. Only one student indicated that they
did not know what the phrase meant.
Probe 5: Do you think that additional answer choices should be added between B and C?If yes, can
you give me examples of what those answer choices would be?

ETS Post-Cognitive Interview Item Recommendations

Page 38

ETS CogLab Report: NIES|2015
Sixteen out of 30 responding students said that they did not think any additional answer choices should
be added. One said that they did not know, and 12 said that they did think that there was a need for
more answer choices. Of these 12, the answers generally focused on different nuances that students felt
were missing, though one suggested both B and C should be the same, indicating that they did not truly
understand the probe. Additionally, some of the “yes” responses restated the options available in
different words, suggesting that they may have been confused by what the probe was trying to capture.
A few examples of the other “yes” answers are as follows:
“I think they could put a little more in it, like, ‘I can speak medium well,’ like in the middle of little and
well. And like, ’I can barely do some words.’"
“I can speak mostly a couple of words. I can speak them very well, but sometimes I forget how they
sound.”
“Yeah. If you could only speak few words you can’t speak a lot, but you can speak a few, but if you can
speak well you can speak a lot. You can speak a few, but well.”

Solutions to Improve Item
Respondent: Might consider some of the answers given by students who thought that other answer
choices were needed between B and C, though many of the examples do not seem to indicate a viable
additional choice.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It seems that generally speaking, the majority of students responding understood what the
question was trying to ask. If there were any change to be made, it might be to stay consistent regarding
phrasing in the different answer options.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 39

ETS CogLab Report: NIES|2015
Student #6

New/VH040067

How do you rate yourself in reading in an American Indian or Alaska Native language?
A. I cannot read in an American Indian or Alaska Native language.
B. I can read a few words or phrases.
C. I can read well.

Item 6
Reading: NA
Understanding: Some of the respondents seemed to have difficulty understanding the item based on
responses to Probe 1, and at least two respondents felt initially that this was the same question as Item
5.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-five of 29 respondents provided a rephrasing of the question that indicated that they
understood the meaning of the question. One said they did not know, one referenced English (indicating
that they may have been confused about the item or probe), and one asked if the question was referring
to one’s first language or a Native language.
Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?
Thirteen out of 19 respondents mentioned a specific language in their answers. In at least three of these
cases, the interviewer or data entry person noted what language the student was familiar with was
based on answers to other questions if the student did not mention it themselves. One student said they
did not know, and four gave difficult-to-categorize answers such as, “I can read not that much but I can
read some” and “I can probably read a sentence or maybe a couple words that's written like on a board
or on a book” and “I could tell you who I am”—this last one may be referencing being able to introduce
oneself in a Native language, but it is hard to tell.
Probe 4: In answer choice B, what does the phrase “I can read a few words or phrases” mean to you?
Twenty-two of 27 respondents (one was inaudible) responding felt that that answer choice B was
referring to knowing a little bit or at least some words in a Native language. Three said they did not
know, and two gave the following answers: “That you kind of see something, and that you’re doing good
on something, that something is really hard to do” and “B is saying that I can only read most of the
words in Choctaw or in Indian.”

ETS Post-Cognitive Interview Item Recommendations

Page 40

ETS CogLab Report: NIES|2015

Probe 5: In answer choice C, what does the phrase “I can read well” mean to you?
Of the 27 students who responded, 23 seemed to adequately understand what answer choice C was
referring to, three said they did not know, and one said, “That you see something really big because
sometimes it's really hard to read.”
Probe 6: Do you think that additional answer choices should be added between B and C? If yes, can
you give me examples of what those answer choices would be?
Fourteen of 22 respondents indicated that they thought the answer choices were fine as they were and
eight felt there was a need for additional answer choices. Of the remaining respondents, nine answers
varied greatly, some of which are included in the following:
“Like, if you can't, if you don't know if you can read American language, they should have another thing
that says you don't know if you can read it.”
“I can read half of the words I know, but I can read a little better than that. Like, I almost can read really
well.”
“I think that there should be, I could read a little words, like maybe like 10 or five words because phrases
and words don’t go together really well sometimes.”
“Probably, like it should go, ‘I can read a few phrases,’ and then between them, it should say, ‘I can read
two or three paragraphs,’ and then, ‘I can read well.’"

Solutions to Improve Item
Respondent: Some of the suggestions for adding additional answer choices could be considered.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Most of the students given this question seemed to understand the item as it was. One suggestion
may be to keep phrasing consistent across answer choices.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 41

ETS CogLab Report: NIES|2015
Student #7

New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?
A. No one has taught me about an American Indian or Alaska Native language.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): _____________________________

Item 7
Reading: NA
Understanding: Some respondents thought that this was a repeat question when they read it.
Changing Answers: Not within the question, but a respondent who previously had noted that they did
not know what a tribal representative was indicated during this item that they had family members who
were tribal representatives.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-four of 31 respondents provided answers consistent with what the item was asking. Two
students did not mention language but instead seemed to think the question was referring to AI/ANs in
general, and four gave answers that were difficult to characterize:
“If the teachers are doing good in the school and which they are, they're doing an excellent job I think.”
“B, C, and E. For the first one.”
“I think it’s the same as number one.”
“Someone who you know, like a friend or a teacher, or your mom or sister, for example they could all
teach me but one that teaches me the best and the most. They teach me important stuff and good stuff
about them.”
Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?
All 29 responses to this probe were too varied to categorize, with the exception of “I don’t know,” which
occurred five times. Some students thought the question had to do with whether or not they could
speak a language, who taught them, how well they spoke it, etc. There was very little validity with
regard to answers.

ETS Post-Cognitive Interview Item Recommendations

Page 42

ETS CogLab Report: NIES|2015
Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska
Native language”?
Nine out of 27 respondents indicated that they did not know, five mentioned specific languages, five
said that they could not give an example, three gave examples indicating they were thinking about
vocabulary, three indicated that they considered counting in an AI/AN language as an example, and two
suggested that being able to speak it was “part of an American Indian or Alaska Native language.” As
with Probe 3, the phrasing of this question leaves a lot up for interpretation and the meaning seems not
to be clear to respondents.
Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
There were nine respondents to this question. Eight indicated that they did have family members who
were also tribal representatives, and one said they did not. This question was skipped for several
respondents as the question had been asked and answered in an earlier probe.
Probe 6: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: Revise or remove Probes 3 and 4 as they seem to confuse students, and answers given
have questionable validity. Additionally, the question about tribal representatives seems in general to
draw unreliable responses from students who do not seem to be sure of what a tribal representative is
or what such positions consist of. We recommend either changing the probe to more deeply explore
what students define as a tribal representative or remove the probe.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of students seemed to easily understand what the question was asking. We
recommend keeping phrasing consistent throughout answer options.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 43

ETS CogLab Report: NIES|2015
Student #13

Trend/VC759216

During 4th grade, have you attended school field trips to museums, traditional villages, or
other places to learn about American Indian or Alaska Native people?
A. Yes
B. No, but other 4th-grade students did.
C. Not offered to 4th-grade students.

Item 13
Reading: NA
Understanding: Some students did not interpret the item only in terms of school activities but also
brought in times that they may have gone on field trips with non-school organizations or families. One
student interpreted the question as having to do with funding. Additionally, several students seemed to
have a hard time only answering for what they had done during the 4th grade.
Changing Answers: Two students changed their answers; one could not explain why they chose to
change the answer, and the other switched their answer from B to C after some consideration.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-seven of 30 respondents answered the question correctly in the sense that they appropriately
identified the main components of the item, though some bordered on essentially repeating the
question. Two students gave difficult-to-categorize answers such as, “Have you seen or they put up in
museums what it does that they think,” and, “If you knew a little bit more about American Indian or
Alaska Native people.” One student said they did not know and another seemed to think the question
had to do with funding for field trips.
Probe 3: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
question]: What types of places have you attended to learn about American Indian or Alaska Native
people?
Seven students out of 13 respondents noted that they had visited museums; nine mentioned field trips
to historic sites, powwows, hatcheries, or missions. There may be some issues regarding the validity of
these answers as one student noted that the field trips they had been on had not been “traditional” per
se, and there may be some question as to the time frame from which students were drawing upon to
answer the question.
Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: How do you know that other 4th-grade students attended, or that activities were not
offered to 4th-grade students?
Seven out of 17 students responding said that field trips were either not offered or that they had not
been told about them, three responded that all the 4th graders go on field trips together, one mentioned
ETS Post-Cognitive Interview Item Recommendations

Page 44

ETS CogLab Report: NIES|2015
that they needed a permission slip—indicating that perhaps they did not fully understand the question
as it was asked. There were four responses that were hard to make sense of as they were non sequitur
answers ranging from having to do with getting sick on Halloween, an Ojibwe reading group, and not
wanting to go on a field trip. Some of the confusion related to answering the question may be related to
having two questions presented as one as part of this probe.
Probe 5: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
question]: Do you know whether other 4th-grade students had the activities or the activities were not
offered to 4th-grade students?
Five of 10 respondents said that all of the 4th graders go to the field trips together, three said they did
not know, and two said “no.”

Solutions to Improve Item
Respondent: NA
Interviewer: The probes in this question seem to cause some confusion, particularly Probe 4 where
there are perhaps too many components to the question for students to easily make sense of, and the
same may be true of Probe 4 as there are two distinct components to that probe as well. Additionally,
there seemed to be some issues with ensuring that respondents were choosing answers that only
applied to students’ time in the 4th grade. Perhaps this point could be further emphasized in the wording
of the item.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of the students seemed to understand the question as written, but there are some
questions regarding the validity of the time frame students were thinking of in their responses. Our
recommendation is to further emphasize that the question is referring to students’ time in the 4th grade,
perhaps by using the term “this year” either in the item or the answer choices.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 45

ETS CogLab Report: NIES|2015
Student #14

Revised/VH040200

During 4th grade, have you used books, videos, or other materials (including online resources)
about American Indian or Alaska Native people in school?
A. Yes, once or twice
B. Yes, three or more times
C. No

Item 14
Reading: NA
Understanding: It seemed that some respondents did not interpret the question solely in terms of
school resources.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-seven out of 30 responding students seemed to interpret the item correctly. There were varying
degrees of detail in respondents’ answers, but the gist of the question appeared to be intact. One
student seemed a little confused and stuck on the in school portion of the question and did not really
engage the idea of materials, and two students offered non sequitur responses such as, “How would you
treat people that went in war or something who built the Statue of Liberty or something” and, “Have
you found books or videos that teaches you languages and ... Do you know what a translator is on your
phone or something? That is something you could use. You could put a word down and it could translate
that. So like that. Or YouTube, you could look Indian words up and stuff.”
Probe 3: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up
question]: How many times have you used these materials in school?
Answers among the 27 respondents to Probe 3 varied somewhat widely with six students saying they
had used these materials two times, five indicating 10 or more times, four saying three to four times,
three saying only once, and other answers with one response were five times, “no,” “a lot,” “two times a
week,” and “I don’t know.” Additionally there were three difficult-to-categorize answers:
“During Lakota class we watch videos of how they speak and we watch legends like dinosaurs from the
end of the world and stuff.”
“Kind of a lot in Idaho history. We had to make dioramas about Native American tribes. I did Nez Perce.
We went on-line to go find stuff about it. I think that's all.”
“On my old school, they had spears way back when the days and then they had the sharp rocks on them.
They sharpen them and one time, they brought … we had a science experiment where we’re learning
about this and then they brought this one across like that and then it had one that goes through it and

ETS Post-Cognitive Interview Item Recommendations

Page 46

ETS CogLab Report: NIES|2015
it’s really [inaudible 00:19:11] and we looked at it and see … one had signs on it like paint writing on it
but it’s really tiny like this little thing when you look at them.”
Probe 4: What does the phrase “online resources” mean to you? Can you give me examples of “online
resources”?
In general, all 31 responding students seemed to understand this question across the board, although
the particulars of their answers varied quite a bit with regard to the components they chose to discuss.
Eight students mentioned computers, seven mentioned websites (general, not specific ones), seven
reiterated “online,” six mentioned Google specifically, four said something about the Internet, two
mentioned YouTube, two mentioned Wikipedia specifically, and one said something about MP3s and
listening to Native music. One said “I don’t know” and another gave a difficult-to-characterize answer
with the following: “Electricity and pictures that move.”
Probe 5: What does the phrase “in school” mean to you?
Twenty-six of 31 respondents seemed to understand what “in school” was referring to, one student said
they did not understand, and four gave difficult-to-categorize answers only peripherally related to
school, but still related to the item—having to do with school or online resources, just answers out of
place given the probe asked.
Probe 6: Do you think that answer choices A and B are clear? If not, can you give me examples of
other answer choices that would be more clear?
All 28 responding students felt that answer choices A and B were clear. One said they were not but did
not give an alternate to improve the choices, and one said, “C, I don’t really like because no because if I
never knew it, I’ll probably put C but I know it so I put B but A and B are good answers but C, it ain’t.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of students asked this question seemed to grasp the meaning and intent of the item.
The phrase “online resources” could possibly be replaced with the terms “Internet” or “websites” as
they may be more recognizable to students.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources”
with the phrase "internet resources" which is likely more recognizable to students.

ETS Post-Cognitive Interview Item Recommendations

Page 47

ETS CogLab Report: NIES|2015
Student #15

Revised/VH040205

During 4th grade, have you used books, videos, or other materials (including online resources)
about American Indian or Alaska Native people outside of school?
A. Yes, once or twice
B. Yes, three or more times
C. No

Item 15
Reading: NA
Understanding: A few of the respondents seemed to think this was the same question as Item 14 at
first.
Changing Answers: Three respondents changed their answers—two from B to A and one from A to B—
and one other answered A, but later said they had used materials “three or more times.”

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 31 respondents seemed to grasp the main components of this question regarding having used
materials outside of school. Again, there was some variation within answers, many specifically referring
to AI/ANs and others simply talking about using materials not on school grounds, but all seemed to
understand underlying concept of item.
Probe 3: [Conditional probe: If the participant selects answer choice A or B, then ask the follow-up
question]: How many times have you used these materials outside of school?
As with the previous item, all 18 responses to this question varied quite a bit. Six said one to two times,
three said three or more times, two said a “few” times, two said 10 times, and other answers occurring
only once were “at camp,” “not a lot,” “once a week,” or “eight or nine times.”
Probe 4: What does the phrase “outside of school” mean to you?
All 30 respondents to this question correctly identified what was meant by “out of school” with varying
details. Most indicated that they were thinking of home or at the library or at a friend or family
member’s home.
Probe 5: Do you think that answer choices A and B are clear? If not, can you give me examples of
other answer choices that would be more clear?
All 27 responses indicated that they thought the answer choices made sense. One said “no,” offering no
explanation; one had this to say after answering affirmatively that the answers made sense: “I don't
think they make sense. Because they're both the same,” and another student who answered in the
affirmative offered this caveat: “Yeah. Only B does though. Because why would you learn it if you’re not
learning much looking at it once or twice? It shouldn't be there. Should be three times or more.”
ETS Post-Cognitive Interview Item Recommendations

Page 48

ETS CogLab Report: NIES|2015
Solutions to Improve Item
Respondent: Differentiate answer choices—opinion was not shared by the majority of respondents.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of respondents seemed to understand this item with ease and interpret it as it was
meant. As with the previous question, the only change might be to consider replacing the phrase “online
resources” with the terms “Internet” or “websites” as they may be more recognizable to students.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to replace the phrase “online resources”
with the phrase "internet resources" which is likely more recognizable to students.

ETS Post-Cognitive Interview Item Recommendations

Page 49

ETS CogLab Report: NIES|2015
Student #16

Revised/VH040207

Do you have access to a school library, media center, or resource center with books, videos,
or other materials (including online resources) about American Indian or Alaska Native
people?
A. Yes
B. No

Item 16
Reading: NA
Understanding: Some issues that arose with the interpretation of the item came from confusion around
the term “media center” and students not interpreting the question solely in terms of school—also
taking community resources into account.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-six students out of 28 respondents seemed to generally understand the question as it related to
having a place to go to get information such as a library, and some of these students directly related it to
AI/ANs. Not all students seemed to interpret this question as being located in a school per se.
Probe 3: What does the phrase “school library, media center, or resource center” mean to you?
This probe drew a number of different answers from the 28 respondents, with students referencing
different pieces of the phrase “school library, media center, or resource center.” Ten students said
something about computer labs, nine mentioned libraries or books, five felt all of the different entities
were essentially the same thing, five mentioned resources, one said something about it being in school,
one said it would be elsewhere, and one said they did not know. The term “media center” seemed to
throw some students off, as did “resource center” (though to a lesser extent).
Prove 4: What does the phrase “have access” mean to you?
Twenty-three out of 31 respondents felt that “have access” was related to being able to go somewhere
or having permission to go somewhere—sometimes this was seen as a physical place, or the ability to go
to certain sites on the Internet. Three students indicated that they did not understand what “access”
meant, one thought the question was related to having the time to go somewhere, and another thought
it had to do with being physically close to a place. Four gave difficult-to-categorize answers such as,
“Access more food and more computers” and, “Really enjoy and really into learning about Native
American people.”

Solutions to Improve Item
Respondent: NA
ETS Post-Cognitive Interview Item Recommendations

Page 50

ETS CogLab Report: NIES|2015
Interviewer: Perhaps look at rephrasing the item to use easier-to-recognize words and phrases. For
instance, perhaps look at alternatives for “have access” and “media center.”

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The item seemed to be generally understandable by the majority of respondents. One
recommendation may be to simplify the language with regard to “access” and “media center.”
Additionally, if the aim is to specifically ask about these resources at school, that should be highlighted,
perhaps by only bolding the word “school” and leaving the other words without bolding, or alternatively
using the word “school” before every component of the question such as: “school library” “school media
center” or “school resource center.”

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically
asking about these resources in school by adding the bolded words "In school" to the beginning of the
item. For consistency with the proposed revision to the grade 4 student items VH040200 and VH040205,
ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is
likely more recognizable to students. The proposed revised item wording is consistent with other NAEP
student questionnaires, such as the 2014 TEL Probe questionnaire.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 51

ETS CogLab Report: NIES|2015
Student #18

New/VH040240

I put a lot of effort into my schoolwork.
A. This is not like me.
B. This is a little like me.
C. This is a lot like me.

Item 18
Reading: NA
Understanding: The grand majority of students were able to understand the question.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-six out of 31 respondents indicated that putting in effort equated to trying hard in one way or
another. Only one student said that they did not know, and one student mentioned teachers. Two
students gave hard-to-characterize answers: “I really think about what the question is trying to ask me”
and, “People effort into the schoolwork by learning about them.” There are potentially typos from the
transcription process.
Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: Can you give me examples of how you put a lot of effort into your schoolwork?
Out of 29 respondents, 19 gave examples of trying hard to show how they put effort into their
schoolwork. Four mentioned the importance of listening to teachers, two mentioned getting good
grades, and two mentioned studying. The following responses were mentioned once: doing work
quickly, doing homework, getting help when needed, showing one’s work, and taking time to do work.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.
ETS Post-Cognitive Interview Item Recommendations

Page 52

ETS CogLab Report: NIES|2015
Student #21

New/VH040382

I feel that I belong at school.
A. This is not like me.
B. This is a little like me.
C. This is a lot like me.

Item 21
Reading: One student misread the answer choices.
Understanding: One student seemed to have trouble grasping the overall concept of the question.
Changing Answers: One student changed their answer from C to A after the interviewer prompted them
to re-read the item’s answer choices (as they had been misread).

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 31 of the respondents grasped the meaning behind this item with some variation. Many felt that
they were being asked if they felt safe at school, or if they enjoyed school, or if they wanted to be at
school.
Probe 3: What does the phrase “belong at school” mean to you?
All but one of the 31 respondents seemed to understand the probe and all gave varying answers about
what the phrase “belong at school” meant to them. Six said that it had to do with being at school, seven
said it had to do with learning, five indicated that it meant you wanted to stay at school, four mentioned
college aspirations, two said that it felt good to be at school, one said that it meant being part of
something, one felt it meant wanting to stay at your school rather than going to another, and one did
not know. One gave a difficult-to-categorize answer: “I think it means did your parents put you in [art
00:28:43] school or did they have to put you in a private school because maybe the kids aren’t so good
at school to you. They treat you so mean but you treat them really nice.”
Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: Can you give me examples of when you feel you “belong at school”?
Of 27 respondents, eight mentioned friends, eight mentioned learning, six brought up feeling good at
school, two mentioned college, two mentioned simply being at school, and one said they were not sure.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 53

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: The grand majority of students understood the meaning and wording of this question and probes.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 54

ETS CogLab Report: NIES|2015
Student #24

New/VH040391

If I put in enough effort, I will succeed in school.
A. Strongly disagree
B. Disagree
C. Agree
D. Strongly agree

Item 24
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-eight of 30 respondents answered this probe in keeping with the meaning of the item, two said
that they did not know (which may be related to interview fatigue—particularly with what could be
construed as repetitive questions).
Probe 3: What does the phrase “put in enough effort” mean to you? Can you give me examples of
how you could put in enough effort?
Twenty-three of 31 responding students referenced trying hard, three mentioned getting good grades,
two said something about going to college, two mentioned listening to teachers and directions, one
mentioned homework, one said staying in school, and one said they did not know.
Probe 4: What does the phrase “succeed in school” mean to you? Can you give me examples of what
“succeed in school” would look like?
Answers among 31 respondents were varied for this question. Six referenced trying hard, six said
something about passing grades and moving from one grade to the next, six mentioned “graduation,”
six mentioned getting good grades, three referenced listening to teachers and directions, three said they
did not know and indicated the question was somewhat confusing, two spoke of college degrees, one
simply said “no,” and one mentioned homework.

Solutions to Improve Item
Respondent: NA
Interviewer: The question regarding putting in enough effort question was a little repetitive given Item
18.

ETS Post-Cognitive Interview Item Recommendations

Page 55

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: All students seemed to grasp the meaning and wording of this item.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 56

ETS CogLab Report: NIES|2015

Survey Instrument: Student Grade 8/Sequence I
Student #3

New/VH039931

Who taught you most of what you know about issues today that are important to American
Indian or Alaska Native people?
A. No one has taught me about issues today that are important to American Indian or
Alaska Native people.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ________________

Item 3
Reading: NA
Understanding: Most of the students understood the question without trouble; two had to be
redirected by the interviewer for a more appropriate interpretation.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty of 23 respondents seemed to grasp the meaning of the item, two said they did not know, and
one did not answer.
Probe 3: In answer choice B, what does the phrase “Family members” mean to you?
Thirteen of 23 respondents named different types of family members that the question brought to
mind, four mentioned that family members are people related to them, three mentioned that it would
be other tribal members as they are all family to some degree or another, and two mentioned love as a
component of the relationship. One gave a difficult-to-characterize response: “Which one of your family
members taught you the most of your people?”
Probe 4: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you?
Can you give me examples of “Tribal representatives or elders”?
Seven students (out of 21 respondents) mentioned that tribal representatives were people who
represented the tribe, five students gave specific names or relationships of people who were
representatives or elders, four mentioned people who had more knowledge, four said it had to do with
ETS Post-Cognitive Interview Item Recommendations

Page 57

ETS CogLab Report: NIES|2015
being older, three mentioned grandparents, one said that they were people higher up in the tribe, one
said they did not know, one said people on the council, and two gave uncharacterizable answers.
Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Eleven of 17 respondents indicated yes, three said no, and three said maybe. Most who indicated that
they did have a family member who was also a tribal representative noted that they thought of the
person as family first (one respondent was the exception).

Solutions to Improve Item
Respondent: NA
Interviewer: In the future, it might be helpful to separate out Probe 4 into two different questions, same
for Probe 5.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It seems that the overwhelming majority of respondents understood the question as it was asked.
One suggestion for potential change may be to keep the structure of the answers the same.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 58

ETS CogLab Report: NIES|2015
Student #4

New/VH040039

How often do you attend classes in school that are taught in an American Indian or Alaska
Native language?
A. My school does not offer courses that are taught in an American Indian or Alaska
Native language.
B. Never or hardly ever
C. Once or twice a month
D. Once or twice a week
E. Every day or almost every day

Item 4
Reading: NA
Understanding: Most of the respondents seemed to understand the item as it was, though some
students misconstrued the question and thought it had to do with learning about culture rather than
language.
Changing Answers: One student changed their answer from B to A as a result of answering Probe 6.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Seventeen of 23 responding students seemed to interpret the item correctly, five thought it had to do
with culture or tradition, and one student was confused.
Probe 3: What does the phrase “American Indian or Alaska Native language” mean to you? Can you
give me an example of “American Indian or Alaska Native language” that you can speak or know
about?
Twelve of 23 respondents gave examples of specific languages, three said they did not know, three
referred to “the language they speak,” two mentioned taking classes, two were confused by the
question, and one would not give an example.
Probe 4: How many American Indian or Alaska Native languages do you speak?
Fourteen out of 23 respondents said they spoke at least one language—or had some sort of knowledge
of one language. A few mentioned being able to understand but not speak. Five mentioned non-AI/AN
languages they could speak, four said they did not speak any AI/AN languages, and one student said
they spoke two AI/AN languages.
Probe 5: How would you answer this question if you speak more than one American Indian or Alaska
Native language?
Answers to this probe were varied among the 16 respondents, leading interviewers to posit that
hypothetical questions are difficult for many respondents to answer. Three students gave D as their new
ETS Post-Cognitive Interview Item Recommendations

Page 59

ETS CogLab Report: NIES|2015
answer, three were confused by the question, three said they would answer E, three stated that they do
not speak more than one language, two said their answer would stay the same, one said they would
speak more often (but did not specify how much more often), and one mentioned a non-AI/AN
language.
Probe 6: What is the difference between answer choices A and B?
Twelve students of the 23 respondents who responded to this probe responded in a seemingly
appropriate manner; however, there seemed to be a fair bit of confusion related to the fact that answer
choice A mentions school in the answer options but answer choice B does not. Five respondents gave
responses that indicated that they were confused by the question, two mentioned that family would
teach them, one said there were too many tribes at the school to offer language classes, one said there
was no difference between the two answer choices, and one felt they were the same.
Probe 7: If there were no courses offered in your school that are taught in an American Indian or
Alaska Native language, which answer choice would you select?
Fourteen of 22 students responding answered A, five answered B, and three answered D.

Solutions to Improve Item
Respondent: NA
Interviewer: Make answer choices consistent to cut back on confusion. Students seem to interpret only
answer choice A as being specifically related to school. They seem to interpret the other answer choices
as having more to do with use, and not necessarily related to classes or school.

Initial KAI Recommendation (Keep/Change/Discard)
Change: The item seems to get distorted by the way the answer choices are phrased. We recommend
making wording consistent and potentially highlighting the phrase “in school” if the interest is in
whether or not students learn a language at school and not at home, informally, or through a tribal or
cultural center.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that it should be highlighted that this item is specifically asking about
classes at school; therefore ETS recommends bolding phrase “in school” in the item stem. In answer
choice “A”, ETS recommends changing the word “courses” to the word “classes” to be consistent with
the item stem.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 60

ETS CogLab Report: NIES|2015
Student #5

New/VH040050

How do you rate yourself in speaking an American Indian or Alaska Native language?
A. I cannot speak an American Indian or Alaska Native language.
B. I can speak a few words or phrases.
C. I can speak well.

Item 5
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 23 respondents appropriately rephrased the question. All understood that the item was asking them
to determine how well they could speak in an AI/AN language.
Probe 3: In answer choice B, what does the phrase “I can speak a few words or phrases” mean to you?
Of the 23 respondents, 22 essentially reiterated the probe and said that they thought the question, “I
can speak a few words or phrases,” meant being able to speak a few words. One student said they did
not know.
Probe 4: In answer choice C, what does the phrase “I can speak well” mean to you?
All 21 students responding said that “I can speak well” meant speaking a language with fluency—the
amount of fluency ranged from one paragraph to being a fully fluent speaker.
Probe 5: Do you think that additional answer choices should be added between B and C? If yes, can
you give me examples of what those answer choices would be?
Out of the 23 respondents, 14 students did not feel that there was a need for additional answer choices.
Nine students indicated that there could be additional answer choices. One of the suggestions that came
up three times was the notion of being able to understand but not speak a language, or something
between speaking a few words and phrases and speaking a language well.

Solutions to Improve Item
Respondent: Add an additional answer choice between B and C.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Consider adding an additional answer choice between B and C.
ETS Post-Cognitive Interview Item Recommendations

Page 61

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 62

ETS CogLab Report: NIES|2015
Student #6

New/VH040067

How do you rate yourself in reading in an American Indian or Alaska Native language?
A. I cannot read in an American Indian or Alaska Native language.
B. I can read a few words or phrases.
C. I can read well.

Item 6
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 20 respondents interpreted the item as it was intended and understood that it was asking them to
determine how well they could read in an AI/AN language.
Probe 3: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?
Eleven of 12 respondents mentioned specific languages, and one mentioned basic literacy in a language
but did not specify which one.
Probe 4: [Conditional probe: If the participant selects answer choice B or C, then ask the follow-up
question]: Can you give me an example of “American Indian or Alaska Native language” that you can
read?
Eighteen of 20 respondents indicated that they felt that the question meant that you could read some
of a language but not all of it, or that you could make sense of some works and that you could
pronounce things but not necessarily understand it. Two students indicated that they did not know.
Probe 5: In answer choice C, what does the phrase “I can read well” mean to you?
Fourteen of 18 students responding said they thought it meant that you could read something with
ease (such as picking up a book and being able to read it). One mentioned comfort with reading, while
another pointed to the idea of ease of reading. Two students answered that they did not know.
Probe 6: Do you think that additional answer choices should be added between B and C? If yes, can
you give me examples of what those answer choices would be?
Fifteen of 21 respondents said that the answers were fine as they were. Six students felt that additional
answer options could be added—at least three of these students heralded back to their suggestions
from the last item having to do with speaking. Their suggestions were to add something about being

ETS Post-Cognitive Interview Item Recommendations

Page 63

ETS CogLab Report: NIES|2015
able to read but still stumbling over words, something between B and C, fluent reading, and sounding
out words but not making sense of them.

Solutions to Improve Item
Respondent: There were a few suggestions for additional response options between B and C.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Respondents generally seemed comfortable with the question. Consider making answer choices
uniform in phrasing.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 64

ETS CogLab Report: NIES|2015

Student #7

New/VH040082

Who taught you most of what you know about an American Indian or Alaska Native language?
A. No one has taught me about an American Indian or Alaska Native language.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): _____________________________

Item 7
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 22 students responding essentially restated the item using slightly different words.
Probe 3: What does the phrase “about an American Indian or Alaska Native language” mean to you?
There were varied answers among the 20 respondents. It seemed that many of the students did not
really understand what the probe was trying to get at. Twelve indicated that they were confused or did
not understand the probe, two said it had to do with knowing the language, two said it had to do with
Native languages, one said they did not know any Alaska Native languages, one restated the probe, one
said it had to do with the “old ways,” one said it meant you could write and speak a language, and one
thought it had to do with culture.
Probe 4: Can you give me examples of what you consider to be part of “American Indian or Alaska
Native language”?
Again, there was a lot of variation among the 20 respondents to the answers, and the probe seemed to
not make sense to a lot of the respondents. Eight were confused by the question, three mentioned
specific words in an AI/AN language, two mentioned specific languages, two said it had to do with the
tribe, three thought it might refer to words for animals, shapes, prayers, or songs. One student
mentioned tongue positioning for pronunciation, one mentioned pronunciation, and one student said
they did not have any examples.

ETS Post-Cognitive Interview Item Recommendations

Page 65

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Of the eight students answering the question, five said that they did have family members who were
tribal members, and all said they thought of them as family first. Two students said no, and one said
they were not sure.
Probe 6:
NA

Solutions to Improve Item
Respondent: NA
Interviewer: Probes 3 and 4 were confusing to students as they could be considered somewhat vague
questions. We recommend asking more specific probe questions in the future.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The item seems to be fine as is. Reconsider keeping answer choices phrased similarly. Probes
could use some re-tooling if they are to be used in the future.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 66

ETS CogLab Report: NIES|2015
Student #10

Item 10
Reading: NA
Understanding: One student reportedly had trouble making sense of answer choices.
Changing Answers: One student changed their answer several times while working through options for
the A portion of the question.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Thirteen of 21 students responding essentially repeated the question with varying degrees of sameness,
referring to whether or not they had learned about Native culture through different means, such as field
trips or presentations. Four students thought that the question was referring only to things that
happened within the school, two thought the question was only referring to what they had learned
about AI/AN cultural aspects (no mention of school), one student thought the probe was referring to
their thoughts on the item structure, one student interpreted the question as having to do with what
they had learned in class, and one student thought the question was referring to what presentations
they had given at school.
Probe 3: How do you know that other 8th-grade students attended, or that activities were not offered
to 8th-grade students?
A variety of answers were given among the 17 respondents for this probe, indicating some validity
issues with understanding the probe. Six students said that other students were there (indicating that
that was how they knew that other students attended), five were confused by the question, three said
that they were there and that was how they knew, two students said it was difficult to answer because

ETS Post-Cognitive Interview Item Recommendations

Page 67

ETS CogLab Report: NIES|2015
they were unsure about what was happening at other schools, one said they did not know, one said no
one was told, and another mentioned that they had gone in another grade.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
questions]: Do you know whether other 8th-grade students had the activities? Again, there was a
variety of answers among the 18 respondents suggesting that students may not have fully grasped the
probe. Thirteen said “yes,” four said they were there, three indicated that they did not understand the
question, one said that none of the students had activities, one simply said “no,” and another simply
said that they did know.
Probe 5: For questions a. through d., how would you answer this question if you did not attend these
activities and did not know whether other 8th graders attended these activities?
Again, there was wide variety among the 19 responses. Four students indicated that they were confused
by the question, two said B, seven said C, two said that these activities were not offered, one said “yes,”
one indicated they would still give the same answer, one student was not sure, and one student
quipped, “how would I know.”

Solutions to Improve Item
Respondent: One student suggested that the questions would be difficult for a new student not familiar
with activities prior to their arrival to answer.
Interviewer: Item and probes should be reconsidered for this question as they seem to confuse
respondents and the validity of the data is questionable.

Initial KAI Recommendation (Keep/Change/Discard)
Change: Perhaps it would make more sense to ask this question in a more simplified format as it was for
4th graders, or just make answer choices “yes” or “no.” Additionally, it may be helpful to determine the
most salient pieces of information to be gathered from asking the question and tailor the phrasing
better to best capture that—perhaps bold “through your school” and consider bolding “During the 8th
grade.”

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that the format of the item should be simplified by collapsing the answer
choices to “Yes” and “No”.

ETS Post-Cognitive Interview Item Recommendations

Page 68

ETS CogLab Report: NIES|2015
Student #14

Revised/VC996624

Here are some sentences about reading. Fill in one oval on each line to show whether the sentence
describes a person like you.

This is not
like me.

This is a
little like
me.

This is a lot
like me.

a. When my teacher talks about American
Indian or Alaska Native history or culture, I
try to read more about it.

A

B

C

Revised/
VH043704

b. I enjoy reading about American Indian or
Alaska Native people.

A

B

C

Trend/
VC996627

c. I enjoy reading about people who have
different traditions and cultures (ways of
life, customs) than I have.

A

B

C

Trend/
VC996628

Item 14
Reading: One student misread “sentence” as “sometimes.”
Understanding: Students had varied interpretations of the item that were all peripherally related to the
question at hand but were dynamic in the different pieces that students picked up on—outlined further
in Probe 1. This variation in response may be due to the fact that there are three different questions to
summarize in Item 14.
Changing Answers: One student changed their answer after responding to Probe 4 as they realized they
had confused or misread the answer choices for A and C.

Probes
Probe 1: Can you tell me, in your own words, what question a. is asking?
The 22 responses to the probe were varied with students picking up on different pieces of the overall
item questions. For example, eight students felt the question was asking them about learning more or
researching on their own, two interpreted the question as having to do with enjoying learning, two
commented on the structure of the item questions and answers, and two gave difficult-to-characterize
answers such as, “It is basically like a little test to see if you are that person or not” and, “Sounds like
they’re talking about what you would describe your friend as.” Other responses had to do with interest,
whether or not someone studied, reading, paying attention, if the teacher covered AI/AN
issues/cultures, if students thought of their tribe when teachers were talking about AI/ANs, and whether
or not students were interested in the topic. Again, the wide variety is likely due to the fact that there
are three questions and students were essentially being asked to sum them up into one.

ETS Post-Cognitive Interview Item Recommendations

Page 69

ETS CogLab Report: NIES|2015

Probe 3: In question a. what does the phrase “when my teacher talks about” mean to you?
All 23 respondents felt that the phrase “when my teacher talks about” referred to when their teacher
was teaching—either reading something to them, lecturing, or as part of a class activity.
Prove 4: Which answer choice would you select if your teacher had not talked about American Indian
or Alaska Native history or culture? Why would you select this answer choice?
Ten of 22 respondents said they would give A as an answer, six said they would choose B, and two said
they would choose C. Two students said they would “do more” but didn’t elaborate on whether or not
that meant they would change their answer to B or C, and two students said they would keep their
answer the same. One student was confused by the question. Interestingly, and primarily among the
students who answered A, five students indicated that if their teacher was not telling them about AI/AN
history or culture, then how would they know to go look anything up?

Solutions to Improve Item
Respondent: Possibly put answers in boxes rather than at top of column with corresponding letters.
Interviewer: Consider dropping Probe 3 as the concept is easily understood.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It is hard to tell if anything should be changed with this item given the variety of responses to
Probe 1. Again, it is very likely that the wide variety is due to students trying to describe all three
questions from this item at once rather than the item itself being confusing.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 70

ETS CogLab Report: NIES|2015
Student #16

New/ VH040389

How often do you help other students from your school with their schoolwork? For example,
you might help other students to study for a test, help them with a school project, or go over
their homework with them.
A. Never or hardly ever
B. Once or twice a month
C. Once or twice a week
D. Every day or almost every day

Item 16
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty of 23 respondents mentioned helping others with school work. Four specifically mentioned
helping classmates or peers with school work, one specifically mentioned friends, and one spoke about
helping younger students.
Probe 3: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of how you have helped other students with their schoolwork?
Are the other students that you have helped with their schoolwork in 8th grade? If not, what grade
are the students that you have helped with their schoolwork in?
Of the 19 respondents, 12 students mentioned helping other 8th graders, five said they helped students
in other grades, five mentioned siblings or family members. With regard to areas where help was
provided, five said they just worked through and explained things to other students, four mentioned
math specifically, four mentioned language or reading, three mentioned helping on projects, three
pointed to helping with homework, and one mentioned science.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Post-Cognitive Interview Item Recommendations

Page 71

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 72

ETS CogLab Report: NIES|2015
Student #17

Trend/VC996635

How often do any of the following people help you with your schoolwork? For example, they might
help you to study for a test, help you with a school project, or go over your homework with you. Fill in
one oval on each line.
Never
or
hardly
ever

Once
or
twice a
month

Once
or
twice a
week

Every
day or
almost
every
day

a. A parent or someone else from my
family

A

B

C

D

Trend/
VC996636

b. A teacher or another adult from my
school

A

B

C

D

Trend/
VC996637

A

B

C

D

Trend/
VC996639

d. A tribal representative or elder

A

B

C

D

New/VH018268

e. Someone else who lives in my
community or is a friend of my family

A

B

C

D

Trend/
VC996640

c. Another student

Item 17
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of 23 students responding felt the question was asking them about who helps them with their
homework, five thought the question was focused on how often others helped them with their
homework, one student made a comment about how the item is structured (rather than focusing on
content), and one student was focused on the question being about someone else who lived in the
community.
Probe 2: Can you tell me, in your own words, what question d. is asking?
Seventeen students out of 23 respondents answered the question in a manner that was in keeping with
the meaning of the item though most focused on tribal representatives over elders, likely because the
ETS Post-Cognitive Interview Item Recommendations

Page 73

ETS CogLab Report: NIES|2015
two are posed in a question together. Six students seemed confused by the question and gave answers
such as, “I try an elder that has been in, that has known a lot about schoolwork and stuff and knows a lot
about the tribal stuff,” or, “Do you have any... Do you have any - I don't know,” or, “Not really. Usually a
tribal representative or something or elder. I'm not really sure what it means ... like elder, or is that like
an older person?”
Probe 3: Can you tell me, in your own words, what question e. is asking?
Seventeen of 23 respondents seemed to understand Probe 3 with ease. Two mentioned elders, two
mentioned family members, and one mentioned tribal representatives. One pointed to the fact that
they lived far from their community so they were unsure how to answer and one thought the question
was focused on how often you were helped by the community member or family friend rather than
what the phrase meant. Two seemed to not fully grasp what the question was asking and gave varied
responses such as, “Say you're given a month and you had problems within the month, you see how
many ... which one of your parent, teacher, other student, tribal representative, or someone else helps
you,” and, “D, my ... one of the elders, Nana Grace has helped me doing a project on Native Americans.
She was really helpful. [Nana Grace is family].”
Probe 5: In question e., what does the phrase “someone else who lives in your community or is a
friend of your family” mean to you?
Sixteen out of 22 students responding answered the question in keeping with what the question was
asking, and a few mentioned that the question was very similar to Probe 3. Six mentioned neighbors, six
mentioned family friends, two spoke of family, two said something about random people in the
neighborhood, one noted their tutor, and one mentioned that they did not live in a community.
Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
questions]: Can you give me examples of these people who have helped you with your schoolwork?
Can you give me examples of how these people have helped you?
Ten of 20 respondents mentioned family—family included mothers (primarily), parents, grandparents,
aunts, and uncles. Seven noted their siblings or cousins, eight referenced teachers, four spoke of friends
helping, and two mentioned family friends. Two gave difficult-to-classify answers such as, “Like, for a
tournament... If I'm in a basketball tournament, I'll go to my friend Braden. His uncle is like.... He's like, a
coach for us, when tournaments come, so I'll ask him to help me. Like, "How should I do this?" Or, "How
should I play in the game?" And he'll help me do it.” Or, “If I have certain problems with a subject, they
make sure I focus on that one subject so I get it easier.”

Solutions to Improve Item
Respondent: One student wanted the additional answer choice “whenever they can.”
Interviewer: It would be helpful to ask about elders and tribal representatives separately as students
choose to respond about one and not the other.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It appears that the item is easily understood by most students.

ETS Post-Cognitive Interview Item Recommendations

Page 74

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this matrix item, but drop sub-item “e” VC996640.
Rationale: ETS recommends dropping this item (VC996640) to reduce student burden. Having five subitems about who is helping the student with his/her schoolwork is excessive. Asking about "someone
else who lives in your community/friend of your family" seems to involve a nuanced understanding that
is not being exhibited by some of the respondents to this question.
The similar grade 4 item VC759291 is also proposed to be dropped.

ETS Post-Cognitive Interview Item Recommendations

Page 75

ETS CogLab Report: NIES|2015
Student #19

Trend/ VC996644

During 8th grade, how many times have you talked to each of the following people about the classes
you should take in high school or about what you want to do after high school? Fill in one oval on
each line.

Never

One
time

Two or
three
times

Four or
more
times

a. A family member

A

B

C

D

Trend/VC996645

b. A teacher

A

B

C

D

Trend/VC996646

c. A school counselor

A

B

C

D

Trend/VC996647

d. Another student

A

B

C

D

Trend/VC996648

e. A tribal representative or elder

A

B

C

D

New/VH026209

A

B

C

D

f.

Someone else who lives in my
community or is a friend of my
family

Trend/VH026211

Item 19
Reading: One student misread the answer scale.
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
The majority of the 23 students providing answers seemed to understand the gist of the item but took
away different pieces from it, likely based on the number of answer options and trying to rephrase them
into a cohesive restatement of the question. Eight felt the question was focused on talking to someone
about future plans, five felt it had to do with how often, four interpreted the question as having to do
with talking to family members, two mentioned classes, two felt that the focus was who they had
spoken with, one talked about friends, and one seemed to think the question was tied to success.
Probe 2: Can you tell me, in your own words, what question e. is asking?
This question had a variety of answers among the 18 respondents with 10 students referencing tribal
representatives and seven mentioning elders. One student mentioned wisdom, one said something
about speaking with a school president, one mentioned family friends, one thought it had to do with
ETS Post-Cognitive Interview Item Recommendations

Page 76

ETS CogLab Report: NIES|2015
people who came to the school to talk about such topics, one mentioned their sister’s boyfriend, and
one student thought the question had to do with how many tribal representatives or elders they had
spoken with. The combination of elders and tribal representatives often seems to prevent students from
parsing out the differences between them. However, the overlap is often very great because most
representatives will be their elders, but not all elders are representatives.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Of the two respondents, one student said “yes” and that they thought of them as family first, and one
student said “no.”

Solutions to Improve Item
Respondent: NA
Interviewer: Potentially consider ways to split out elders and tribal representatives as different
categories or just be aware of the issues of overlap and what that may mean for data interpretation.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It appears that the question is easily understood by students.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 77

ETS CogLab Report: NIES|2015
Student #20

New/VH040157

How much do you agree with each of the following statements? Fill in one oval on each line.
Strongly
Disagree Agree
disagree

Strongly
agree

a. If I put in enough effort I will succeed
in school.

A

B

C

D

New/VH040158

b. If I put in enough effort in school I
will get into college.

A

B

C

D

New/VH040160

c. If I put in enough effort in school I
will get a good job.

A

B

C

D

New/VH040161

Item 20
Reading: NA
Understanding: NA
Changing Answers: One student changed their answer at Probe 5 from “strongly agree” to “agree” due
to the idea that some people do not have enough money to go to college.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Fourteen of 20 respondents interpreted the question as having to do with the correlation between
effort and success. Six students spoke to the item structure.
Probe 3:
Nineteen students of the 23 responding spoke about trying hard and working hard in school, four
mentioned doing homework, two pointed to paying attention, one mentioned passing class, and one
mentioned grades.
Probe 4:
Fourteen of 23 respondents referred to getting good grades, 11 referenced graduation, three pointed to
good behavior, one mentioned getting an education, and one spoke about their future.
Probe 5:
All 23 respondents seemed to struggle a bit with this probe because it was so straightforward a
question. Twenty-one said it meant getting into college; of these 21, three mentioned scholarships, and
two mentioned going to a college of choice. Other answers were that the question had to do with
money (one student), further education (three students), being smart (one student), having a career

ETS Post-Cognitive Interview Item Recommendations

Page 78

ETS CogLab Report: NIES|2015
(one student), and how big an achievement that would be for someone from a small town (one
student).
Probe 6:
Nineteen of 23 respondents interpreted this question as having to do with being able to get the job that
you want—your job of choice. Specific jobs mentioned varied among the 23 responses such as lawyers,
doctors, and teachers were mentioned as was having good hours and pay and not working at
McDonalds. Other professions mentioned were architecture, being a judge, photographer, business
owner, and river runner among others.

Solutions to Improve Item
Respondent: NA
Interviewer: Probe 5 is very straightforward and thus actually a little confusing to answer for some
students, as if they suspect that it may be a trick question.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 79

ETS CogLab Report: NIES|2015
Student #21

New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student tardiness

A

B

C

D

New/VH040217

b. Student health problems

A

B

C

D

New/VH040219

c. Student misbehavior in class

A

B

C

D

New/VH040221

d. Physical conflicts among students

A

B

C

D

New/VH040223

e. Bullying

A

B

C

D

New/VH040226

f.

A

B

C

D

New/VH040227

A

B

C

D

New/VH040231

Low student motivation

g. Low teacher expectations

Item 21
Reading: One student stumbled over a few words but reading did not seem generally to be a problem.
Understanding: Two students seemed to be interpreting the item in terms of how it related to them
personally, as if the focus of the question were on them rather than on the school they attended.
Changing Answers: There were no changing answers, but there was some difficulty choosing answers.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Sixteen students out of the 19 respondents felt the item and subsequent questions wanted them to
reflect on problems at their school, one student seemed to interpret the probe on a personal level, one
said they did not know what the question was asking, and one thought it had to do with expectations.
Probe 3: What does the phrase "student tardiness” mean to you?
Twenty of the 23 students responding felt the question had to do with lateness—both to school and to
class. Five students felt it also had to do with absence from school, and one said, “If they actually want
to go to school and just learn new things.”
Probe 4: In question a., what does the phrase "student health problems" mean to you?
Thirteen students out of 23 respondents felt the question was asking about students getting ill or sick,
four mentioned obesity, three brought up asthma, one said cancer, one noted broken limbs, one said
autism, one mentioned drugs or alcohol, one talked about scoliosis, and one mentioned disabilities in
general.
ETS Post-Cognitive Interview Item Recommendations

Page 80

ETS CogLab Report: NIES|2015
Probe 5: In question c., what does the phrase “student misbehavior in class” mean to you?
Out of all 23 respondents, 12 students felt that the question had to do with students’ disruptive
behavior or “acting up” in class, eight mentioned talking in class or talking back to the teacher, five said
it included not listening to the teacher, and four said it meant not doing work.
Probe 6: In question d., what does the phrase “physical conflicts among students” mean to you?
Nineteen of 23 respondents indicated fighting, four mentioned teasing or bullying, and two interpreted
the question as also being related to verbal conflicts like teasing.
Probe 7: In question e., what does the phrase “bullying” mean to you?
Fourteen of 23 respondents talked about verbal conflicts such as teasing, picking on others, and talking
behind someone’s back. Six felt that bullying could also be physical, and five spoke of other aspects such
as the jealousy of bullies, the troubles with being “different,” and students sparring with one another.
Probe 8: In question f., what does the phrase “low student motivation” mean to you?
This probe and the following one seemed to cause a little bit of confusion to all 23 respondents as they
had a harder time explaining what they thought the concepts meant without generally restating the
phrase they were being asked about. For instance, 14 students felt that “low student motivation” meant
that students had no motivation. Some explained a bit more by saying that some students saw no point
in trying hard at school, or were not being encouraged to do better by friends or teachers. Nine
mentioned that low motivation meant not trying hard, and one said it meant sleeping in class.
Probe 9: In question g., what does the phrase “low teacher expectations” mean to you?
As with the previous probe, many of the 23 respondents’ answers fell into the same category as many
students tended to rephrase the phrase they were being asked to interpret. To that end, 18 students felt
that “low teacher expectations” meant that teachers did not expect much from their students, that they
did not care and were not motivating students. Four students said that they did not know what the
question was trying to ask, and one interpreted the question as having to do with whether or not people
had high expectations of their teachers.

Solutions to Improve Item
Respondent: NA
Interviewer: It is hard to say if students were answering for their 8th grade class or their school overall.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Perhaps consider what is included when asking about the school in general versus just asking
about the 8th grade.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students
as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of
ETS Post-Cognitive Interview Item Recommendations

Page 81

ETS CogLab Report: NIES|2015
the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.
Drop sub-item “g” VH040231.
Rationale: ETS recommends dropping this item (VH040231) to reduce student burden. This sub-item
caused much more confusion for respondents than any of the other sub-items within this item.

ETS Post-Cognitive Interview Item Recommendations

Page 82

ETS CogLab Report: NIES|2015

Survey Instrument: Student Grade 8/Sequence II
Student #1

New/VH039733

Who taught you most of what you know about American Indian or Alaska Native history?
A. No one has taught me about American Indian or Alaska Native history.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 1
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Fourteen of 25 responding students felt the item was asking them to think about who had taught them
the most of what they knew about AI/AN history. Four said they did not know, three mentioned history
and culture, three felt the question was asking if they knew about AI/AN history, and one student felt
the question had to do with tribal elders.
Probe 3: What does the phrase “American Indian or Alaska Native history” mean to you? Can you give
me examples of “American Indian or Alaska Native history” that you have learned?
Eight out of 23 responding students simply reiterated that the question was asking about history, four
spoke about culture, four mentioned specific tribes, and three felt it had to do with AI/AN lifestyles of
the past. Four respondents mentioned drumming, art, dance, or storytelling. Two mentioned language,
two mentioned religion, and one student brought up ancestors.
Probe 4: In answer choice B, what does the phrase “Family members” mean to you?
Twelve students out of 24 respondents mentioned specific family members such as grandparents,
parents, aunts and uncles, siblings, and cousins. Five said that “family members” were people who were
“related,” five reiterated the term “family,” three mentioned that they considered the reservation or
part of their family. One student mentioned friends, and one talked about love.
Probe 5: In answer choice E, what does the phrase “Tribal representatives or elders” mean to you?
Can you give me examples of “Tribal representatives or elders”?
ETS Post-Cognitive Interview Item Recommendations

Page 83

ETS CogLab Report: NIES|2015
Eleven of 25 respondents spoke of family, six talked about tribal councils or chair people, three
mentioned being old and/or wise, two students mentioned specific people, six mentioned the tribe. The
responses to this question were fairly dispersed due to the fact that the question covers two groups of
people who may or not may not overlap in terms of their roles as elders or tribal representatives.
Probe 6: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Seven out of 12 students responding indicated that they did have a family member who was a tribal
representative. Of these seven, four said they thought of this person as a family member first, two did
not specify, and one said they thought of the person as a tribal representative first. Five students
reported that they did not think that they had a family member who was also a tribal representative.
Probe 7: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
Only one student chose F as their answer for the item and this student gave a specific name.
Probe 8: What does the phrase “most of what you know” mean to you?
This question seemed to confuse students somewhat as it is a straightforward question, which at times
leads to students either repeating almost verbatim what the question was, or trying to come up with an
answer that is different. Of the 17 students responding, six felt the question was asking them about the
most or the majority of what they knew, five thought it meant what they had learned, three felt it had
to do with language, two thought it was asking about everything that they knew, and two said they did
not know.
Probe 9: Which answer choice would you select if multiple people had taught you about American
Indian or Alaska Native history? How would you decide which answer choice to select?
As with previous hypothetical probes, this one did not generate the same kind of reasoning that may
have been needed to answer the question completely. Of the 17 students responding, seven said that
they would choose B, five said they would choose D, two said E, one said F, one said that they did not
know, and one said they would not choose more than one answer.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of the students seemed to grasp the meaning of the item overall. Consider keeping
all answer choices phrased in a similar fashion, and consider if there is a need to separate out tribal
representatives from elders or to keep the question as is.

ETS Post-Cognitive Interview Item Recommendations

Page 84

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 85

ETS CogLab Report: NIES|2015
Student #2

New/VH039760

Who taught you most of what you know about American Indian or Alaska Native traditions
(way of life, customs)?
A. No one has taught me about American Indian or Alaska Native traditions.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 2
Reading: One student misread the answer for E.
Understanding: One student initially thought this was a repeat of Item 1.
Changing Answers: One student initially chose answer A but then quickly changed to B.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Fourteen of 23 responding students interpreted this question as having to do with who had taught them
the most about what they knew about AI/AN traditions, two mentioned how much they knew, one
thought the question was the same as the last, one simply said “no.” Five gave difficult-to-classify
answers ranging from, “About American Indian Native tribes. It is about our tradition,” to, “What they
did or what they do, the people, American Indians” or, “What do they do? We have Christmas,
Thanksgiving, what do they do? ["They?"] The tribes.”
Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native traditions”
mean to you? Can you give me examples of “American Indian or Alaska Native traditions” that you
have learned?
Eight of 25 responding students mentioned dancing and drumming, six brought up festivals and
powwows, three spoke of religion, three talked about food ways and hunting, two brought up specific
tribes, two mentioned language, one discussed leaders, one brought up clothes, and yet another
mentioned ceremonies.
Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Four of six students responding to this question said that they did not have family members who were
tribal representatives, and two said they did and thought of them as family first. Five had already
answered the question previously.
ETS Post-Cognitive Interview Item Recommendations

Page 86

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: The majority of students appear to understand the question.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 87

ETS CogLab Report: NIES|2015
Student #8

New/VH040103

Who taught you most of what you know about American Indian or Alaska Native systems of
counting?
A. No one has taught me about American Indian or Alaska Native systems of counting.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 8
Reading: NA
Understanding: Twelve students expressed confusion over the term “systems of counting” and did not
know what it referred to. This calls into question the validity of the responses to this question.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twelve out of 25 responding students indicated that they did not know what the question was trying to
ask or what systems of counting were. Though several students came up with answers, it seemed that
there was a lot of guessing as to what “systems of counting” meant. Six focused on the part about who
taught them about systems of counting, four guessed that the question meant counting in an AI/AN
language, three thought it might have to do with tribes being able to keep tabs on their population
counts, one thought it might have to do with money.
Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native systems of
counting” mean to you? Can you give me examples of “American Indian or Alaska Native systems of
counting” that you know about?
Eleven of 22 respondents said they did not know, eight said that they thought it had to do with
language, two brought up population counts, and one mentioned money.
Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Only two students responded to this probe and they thought of these people as family first. One
student had already been asked the question and had already answered it.

ETS Post-Cognitive Interview Item Recommendations

Page 88

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]:
Who is the person?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: Define what is meant by “systems of counting.”

Initial KAI Recommendation (Keep/Change/Discard)
Discard: Students clearly did not grasp what the question was trying to ask but would often provide
answers B-E, thus calling the validity of such responses into question.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that the phrase "systems of counting” is not clear; therefore ETS
recommends revising the phrase to "ways of counting".
This recommendation is aligned with the similar grade 4 item (VH040103).

ETS Post-Cognitive Interview Item Recommendations

Page 89

ETS CogLab Report: NIES|2015
Student #9

New/VH039963

Who taught you most of what you know about American Indian or Alaska Native symbols and
designs used in mathematics?
A. No one has taught me about American Indian or Alaska Native symbols and designs
used in mathematics.
B. Family members
C. Friends
D. Teachers
E. Tribal representatives or elders
F. Someone else (please specify): ___________________________

Item 9
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
The 25 responses given for this probe varied greatly as students were not sure what the question was
asking or what “symbols and designs used in mathematics” meant. Five students focused on the part of
the question regarding who taught them what they know, three mentioned math, four said they did not
know, one said shapes, and one referenced algebra. Nine gave answers that were difficult to categorize.
Some of these are shared below:
“About the symbols they used to tell each other what to do and yeah…”
“What does it mean like they’re never direct like Egyptians people really have their [inaudible 00:07:07]
so they want to know what it is, probably like that.”
“I guess it’s the way they do stuff, like making a teepee. They would have to do all this counting and
stuff.”
“In mathematics, I'm pretty sure I know that symbols and designs, but I'm not really sure what it is doing
in mathematics.”
Probe 3: In answer choice A, what does the phrase “American Indian or Alaska Native symbols and
designs” mean to you? Can you give me examples of “American Indian or Alaska Native symbols and
designs” that you know about?
Eleven of 24 responding students felt that “symbols and designs” had to do with decorative symbols,
usually in art. Six suggested that perhaps they were variations on plus, minus, greater than, division, and
ETS Post-Cognitive Interview Item Recommendations

Page 90

ETS CogLab Report: NIES|2015
multiplication symbols in math. Five said they did not know, and two said that they did not think that
symbols and designs were used in math.
Probe 4: [Conditional probe: If the participant selects answer choice B, then ask the follow-up
questions]: Do you have any family members who are also tribal representatives? If yes, do you first
think of them as your family members or as tribal representatives?
Only one student was asked the question; they answered “no.”
Probe 5: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: Who is the person?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: This item should either be clarified or removed.

Initial KAI Recommendation (Keep/Change/Discard)
Discard: Remove the question as there is a lack of clear understanding about what the question is
asking. The lack of understanding calls into question the validity of the answers given by students.

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS agrees with KAI that this item should be dropped. Respondents had difficulty interpreting
the phrase "symbols and designs", and the specific reference to math caused additional confusion.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 91

ETS CogLab Report: NIES|2015
Student #11

Revised/VH039972

During 8th grade, how often have you used any of the following materials in school? Fill in one oval
on each line.

Once or
twice a
month

Once
or
twice a
week

Every
day or
almost
every
day

A

B

C

D

Revised/VH039974

b. Books, videos, or other materials
(including online resources) not
related to American Indian or
Alaska Native people, traditions,
or cultures

A

B

C

D

Revised/VH040079

c. Books, videos, or other materials
(including online resources)
about current issues that are
important to American Indian or
Alaska Native people

A

B

C

D

Revised/VH040083

Never
or
hardly
ever
a. Books, videos, or other materials
(including online resources)
about American Indian or Alaska
Native history, traditions, or
cultures (way of life, customs)

Item 11
Reading: One student misread question row B, one misread the question as “today” rather than “8th
grade.”
Understanding: Five respondents expressed some confusion over how to make sense of the item and
the different questions posed within it.
Changing Answers: One student changed their answer to row B from A to B after a discussion of the
item’s meaning. Another student expressed some difficulty choosing an answer for row A and row C.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
The 25 responses to this question were varied in part perhaps because respondents were being asked to
restate three different questions as one. Four students thought it had to do with how often resources
were used, four felt it was asking if they had ever used them, two restated the question, one thought it
ETS Post-Cognitive Interview Item Recommendations

Page 92

ETS CogLab Report: NIES|2015
had to do with how often they used their AI/AN culture in their everyday life, and one thought it had to
do with learning traditions. Two felt that the question was asking about where they learned about AI/AN
culture and traditions, two said they did not know, and two restated all three portions of the question in
their own words. Six students gave difficult-to-categorize answers such as:
“The bad stuff that's happening, not the bad stuff but the important stuff. [Happening to what?]
Something that’s important to them. [Who?] To the American Indians and Alaska Natives.”
“Like what's going on right now, in that culture or whatever.”
“Traditions they used, or how they used to work their life.”
Probe 3: In question b., what does the phrase “books, videos, or other materials not related to
American Indian or Alaska Native people, traditions, or cultures” mean to you? Can you give me
examples of these books, videos, or other materials?
There were a variety of answers to this question from the 25 respondents. Twelve students referenced
doing research or project work on topics not having to do with AI/ANs, three talked about history books,
four mentioned going online to do research about anything (or looking at Facebook or viral videos), one
student thought the question still had to do with AI/ANs, three said they did not know, and two
mentioned research in general.
Probe 4: In question a., what does the phrase “online resources” mean to you? Can you give me
examples of “online resources”?
Twelve of 25 respondents specifically mentioned going to websites to learn more about their AI/AN
culture, nine mentioned Google, four talked about Bing, two mentioned YouTube, two mentioned
Facebook, one said Twitter, one mentioned the media, and one said anything on the Internet.
Probe 5: What does the phrase “in school” mean to you?
As this question was very straightforward, it produced some interesting answers among the 22
respondents where students seemed to be unsure of the answers because the answer seemed so
obvious. Sixteen said that “in school” meant in school, two said they did not know, one thought it had to
do with education broadly, one said it had to do with what they were learning, and five gave difficult-toclassify answers:
“That means like spent in at max.”
“Having someone come in and talk to us about it or lessons with video clips about it.”
“Using books, reading, talking about it socially.”
“Boring. [Interviewer prompts.] Prison, learning, get educated, getting you ready for high school.”
Probe 6: Have you used any of the books, videos, or materials listed in questions a. through c. outside
of school, but not for “in school” class work or assignments? If yes, do you consider those activities to
be “in school” class work or assignments?
Ten of the 21 students responding said that they had used materials outside of school and five of those
considered that work to be school-related. Five said they did not consider that work school-related. Ten
students said that they did not use materials outside of school, and one said probably.

ETS Post-Cognitive Interview Item Recommendations

Page 93

ETS CogLab Report: NIES|2015
Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It is difficult to make a recommendation on this particular item given the lack of validity in how
students interpreted the item, which may be due in large part to having to summarize three questions
into one.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 94

ETS CogLab Report: NIES|2015
Student #12

Revised/VH040104

During 8th grade, how often have you used any of the following materials outside of school? Fill in
one oval on each line.
Never
or
hardly
ever

Once or
twice a
month

Once or
twice a
week

Every
day or
almost
every
day

a. Books, videos, or other
materials (including online
resources) about American
Indian or Alaska Native history,
traditions, or cultures (way of
life, customs)

A

B

C

D

Revised/VH040106

b. Books, videos, or other
materials (including online
resources) not related to
American Indian or Alaska
Native history, traditions, or
cultures

A

B

C

D

Revised/VH040109

c. Books, videos, or other
materials (including online
resources) about current issues
that are important to American
Indian or Alaska Native people

A

B

C

D

Revised/VH040111

Item 12
Reading: NA
Understanding: Two students had to be redirected about the question referring to materials “outside of
school.”
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Fifteen of 24 respondents said that the question was asking about what they did outside of school, two
restated the question, and one said they did not understand. Six gave hard-to-qualify answers such as:

ETS Post-Cognitive Interview Item Recommendations

Page 95

ETS CogLab Report: NIES|2015
“It's asking you how often you would go and look up stuff about Native Americans or the way of life or
what's going on with them in oppose to how much you don't look up about them while you're still on
the Internet.”
“It's talking about outside and we just came back from outside. I get to play basketball, kickball,
stickball.”
“Things you don't do outside of school.”
“What you do the most or what you like to do.”
Probe 3: What does the phrase “outside of school” mean to you? Do you consider “outside of school”
to include all activities that are for schoolwork and not for schoolwork?”
Out of 25 respondents, 15 said that “outside of school” meant outside of school. This question seemed
to make students uncomfortable because of how easy it was. Three said doing things on their own time,
two said it had to do with hanging out with friends, two said they did not know, one said being at home,
and two gave hard-to-categorize answers:
“Like at home or at people's houses or powwows or something. [All activities included?] Like kind of for
school.”
“Mmhmm. Sometimes, but whenever it gets kind of fun for friends and play. I'll start reading just like-“
Probe 4: In question b., what does the phrase “books, videos, or other materials not related to
American Indian or Alaska Native people, traditions, or cultures” mean to you?
Sixteen of 24 respondents said that the phrase in question B just meant researching different topics that
did not have to do with AI/ANs. Four thought it had to do with their or another AI/AN tribe, and four
said they did not know.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: It seems that most respondents seem to understand this question based on responses to the
item.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 96

ETS CogLab Report: NIES|2015
Student #13

Revised/VH040207

Do you have access to a school library, media center, or resource center with books, videos,
or other materials (including online resources) about American Indian or Alaska Native
people?
A. Yes
B. No

Item 13
Reading: NA
Understanding: One student said they were unsure what “media center” meant.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of 22 students responding to the question mentioned having access to different materials;
only six mentioned in school. One student said they did not know and two thought the question had to
do with where you would go if you did not have access.
Probe 3: What does the phrase “school library, media center, or resource center” mean to you?
Nineteen of 24 respondents essentially defined the idea of “school library, media center, or resource
center” as a place to go to find materials. However, it became clear that many students were unsure of
what a resource or media center were (10), though they often hazarded guesses, and it seemed that
many students interpreted “resource center” as a place outside of school. This is perhaps because
“library” is the place they are most familiar with, and also it is the only term with “school” before it.
Probe 4: What does the phrase “have access” mean to you?
Twenty-three of 24 students responding felt that “have access” meant being allowed, having
permission; one said that “have access” did not mean anything to them.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: Consider changing “school library, media center, or resource center” to terms more familiar to
students. School library seemed universally known, whereas the other two options were either
unknown to students or they thought they were not school-based.

ETS Post-Cognitive Interview Item Recommendations

Page 97

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to clarify that the item is specifically
asking about these resources in school by adding the bolded words "In school" to the beginning of the
item. For consistency with the proposed revision to the grade 4 student items VH040200 and VH040205,
ETS recommends replacing the phrase “online resources” with the phrase "internet resources" which is
likely more recognizable to students. The proposed revised item wording is consistent with other NAEP
student questionnaires, such as the 2014 TEL Probe questionnaire.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 98

ETS CogLab Report: NIES|2015
Student #15

New/VH040130

Here are some sentences about your school. Fill in one oval on each line to show whether the
sentence describes a person like you.

a. I put a lot of effort into my schoolwork.

b. I want to be one of the best students in my
class.
c. I enjoy being challenged in my classes.

d. I feel that I belong at school.

This is
not like
me.

This is a
little like
me.

This is a
lot like
me.

A

B

C

New/
VH040131

A

B

C

New/
VH040134

A

B

C

New/
VH040137

A

B

C

New/
VH040138

Item 15
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what question a. is asking?
Fourteen of 21 responding students said that the question was getting at whether or not they tried hard
at school. Three commented on the structure of the item, three mentioned liking school, and one
restated the overall question with all four parts.
Probe 2: [Conditional probe: For question a., if the participant selects answer choice B or C, then ask
the follow-up question]: Can you give me examples of how you put a lot of effort into your
schoolwork?
Twenty of 23 respondents said the question was about trying hard in school such as paying attention or
taking one’s time to do good work. One student mentioned getting help from others to succeed, and
two gave difficult-to-classify answers:
“Okay. Monday and Tuesday, I told you, we were in Greensboro. Tuesday night, whenever we came
back, we had PTA that night. We got our report cards. Before we got report cards, I went to all of my
teachers and I asked them to make sure if I'd missed anything, what it was, or if I could get it now, or if I
could get it tomorrow at school or whenever. I don't want to miss out of anything, and I don't want to
ETS Post-Cognitive Interview Item Recommendations

Page 99

ETS CogLab Report: NIES|2015
be behind whenever we get back in class. Everybody's like, "I know, I know," and I'm sitting there like "I
don't really know". If that's the case, I should have just stayed at home. I shouldn't have come to school
[inaudible 00:22:30].”
“Well me, people try to split me up from my best friend but then we go right back to each other so like I
feel like when I am with her I get my work done and stuff but people don't think that because we goof
off a lot but then we get our work done.”
Probe 3: Can you tell me, in your own words, what question b. is asking?
Eleven of 24 respondents mentioned being the best or the top of their class, six mentioned getting good
grades, four said it was about being the smartest, four indicated that it was about being the teacher’s
pet, three said it had to do with being better than others, three mentioned good behavior as a
component, and two spoke about being popular.
Probe 4: Can you tell me, in your own words, what question c. is asking?
Twelve out of 24 responding students indicated that enjoying a challenge had to do with harder
material, nine said that it had to do with liking a challenge, five mentioned competition with others, and
one said they did not know.
Probe 5: Can you tell me, in your own words, what question d. is asking?
Ten of 24 respondents said that belonging had to do with feeling like you should be there, six said it was
about feeling safe, four mentioned it had to do with liking school, one mentioned popularity, one said it
was about one’s future, and one indicated that they did not know. One gave a difficult-to-categorize
answer: “Going against other people in my classroom.”
Probe 7: In question b., what does the phrase “one of the best students” mean to you?
Twelve out of 24 students responding said that “one of the best students” meant being one of the top
students, six felt it had to do with grades, six brought up the topic of good behavior, one felt it had to do
with being the fastest, and one indicated that they did not know what it meant.
Probe 8: In question c., what does the phrase “being challenged” mean to you? Can you give me
examples of you “being challenged in your classes”?
When asked about being challenged, 13 of 24 responding students mentioned having harder material to
work with, five said it had to do with putting in more effort and pushing oneself, four talked about
competing with others, one said it had to do with getting help when needed, and one said they did not
know.
Probe 9: In question d., what does the phrase “belong at school” mean to you?
Eight of 23 respondents said that they felt they belonged at school because being there felt like they fit
in, four said they felt safe, four said you should be at school, four indicated that it was because you were
good at school, and three said to learn.

ETS Post-Cognitive Interview Item Recommendations

Page 100

ETS CogLab Report: NIES|2015
Probe 10: [Conditional probe: For question d. if the participant selects answer choice B or C, then ask
the follow-up question]: Can you give me examples of when you feel you “belong at school”?
Of the 23 respondents to the question, 11 said that they felt good at school, eight replied that it had to
do with friends, two said that they felt they belonged at school when they were learning, one said that
they had to be at school, and one simply replied “no.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is. All respondents seemed to understand the item as written.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 101

ETS CogLab Report: NIES|2015
Student #18

Revised/VH040140

During 8th grade, have you participated in any of the following activities organized by your school?
Fill in one oval on each line.
Yes

No

a. Sports team

A

B

Revised/VH040143

b. Drama club

A

B

Revised/VH040146

c. Music club, orchestra, or band

A

B

Revised/VH040147

d. Cultural dances or other American Indian or Alaska
Native activities (for example, drum groups)

A

B

Revised/VH040150

e. Academic club (for example, math club)

A

B

Revised/VH040152

Item 18
Reading: One student had a hard time reading “orchestra.”
Understanding: Four students had some difficulty defining some of the item’s different components.
Changing Answers:

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of 21 responding students essentially restated the question in general terms regarding
whether or not they had taken part in any of the after-school activities mentioned. Two students talked
about interest rather than participation.
Probe 3: In question a., what does the phrase “sports team” mean to you?
Twenty-three respondents gave a multitude of examples of what they considered sports teams. Nine
mentioned basketball, seven brought up football, four mentioned track or cross-country, three talked
about baseball, three mentioned soccer, three said volleyball, two said tennis, two said softball, one said
golf, and one said wrestling.
Probe 4: In question b., what does the phrase “drama club” mean to you?
Eleven of 22 responding students felt that “drama club” entailed theatre or acting in plays, seven said
they did not know, three said they did not have a drama club at their school, one said they’d never done
it, and one thought it had to do with dance. One interpreted the question as students creating drama
and referenced groups of girls who created drama.

ETS Post-Cognitive Interview Item Recommendations

Page 102

ETS CogLab Report: NIES|2015
Probe 5: In question c., what does the phrase “music club, orchestra, or band” mean to you?
Eighteen of 21 responding students interpreted “music club, orchestra, or band” as being related to
playing instruments, although nine students also mentioned choir or singing. Three brought up dance as
an example.
Probe 6: In question d., what does the phrase “cultural dances or other American Indian or Alaska
Native activities” mean to you? Do you think the example “drum groups” is an example of cultural
dances? If not, can you give me examples of what “cultural dances” would be?
When asked about cultural dances, 14 of 22 respondents talked about cultural dances, and five students
mentioned powwows. Something interesting about other responses to this question is that most of the
examples were not school-based but rather from outside of school. Additionally there were several
students who, when prompted, said that drum groups were a good example; but there was also a
contingency that pointed out that drum groups were not a dance.
Probe 7: In question e., what does the phrase “academic club” mean to you?
Fourteen of 22 respondents used math as an example of an academic club, three felt that it had to do
with smart kids, two mentioned spelling bees, two brought up science, one mentioned history, one said
they did not know, and another student said that they did not have academic club at their school.
Probe 8: In addition to the activities listed in questions a. through e., do you think any additional
activities should be included? If yes, can you give me examples of additional activities?
Fourteen of 23 responding students felt that no additional activities needed to be added to the list. The
following categories had one student each suggesting that it be added: AI/AN-specific organizations,
photography, choir, tutoring club, chess, science, art, and gardening.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Consider putting word “school” in front of each activity to ensure that students remember that
the question is contextually based on activities offered by the school—if indeed that is the key
component being measured.

ETS Recommendation (05/02/2014)
Keep this matrix item, but drop sub-item “b” VH040146.
Rationale: ETS recommends dropping this item (VH040146) to reduce student burden. This sub-item
caused much more confusion for respondents than any of the other sub-items within this item.

ETS Post-Cognitive Interview Item Recommendations

Page 103

ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations

Page 104

ETS CogLab Report: NIES|2015
Student #20

New/VH040157

How much do you agree with each of the following statements? Fill in one oval on each line.
Strongly
Disagree Agree
disagree

Strongly
agree

a. If I put in enough effort I will succeed
in school.

A

B

C

D

New/VH040158

b. If I put in enough effort in school I
will get into college.

A

B

C

D

New/VH040160

c. If I put in enough effort in school I
will get a good job.

A

B

C

D

New/VH040161

Item 20
Reading: One respondent skipped a word when reading the item.
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty of 21 respondents felt that the question was asking about trying hard to do well in school. One
student restated the question, one thought it was a repeat of an earlier question, and one said they did
not know.
Probe 3: What does the phrase “put in enough effort” mean to you?
All 23 respondents to this probe said that “put in enough effort” meant trying hard.
Probe 4: In question a., what does the phrase “succeed in school” mean to you? Can you give me
examples of what “succeed in school” would look like?
Seventeen of the 23 students responding said that “succeed in school” meant passing or graduating.
Eight felt it had to do with getting good grades, three said it was related to being the best, and two
mentioned good behavior.
Probe 5: In question b., what does the phrase “get into college” mean to you?
Twenty of 23 respondents felt that “get into college” meant going to college. One mentioned
scholarships, two had difficult-to-categorize answers: “That you did succeed in school and you did what
you are doing and you did it really good” and “It means, I don't know the word for it ... it gets harder, it-“

ETS Post-Cognitive Interview Item Recommendations

Page 105

ETS CogLab Report: NIES|2015
Probe 6: In question c., what does the phrase “get a good job” mean to you? Can you give me
examples of what “a good job” would be?
Fourteen for 23 respondents mentioned that “get a good job” meant having a job of one’s choice. The
options listed at least once by students were: doctor, lawyer, teacher, astronomer, sports player, desk
job, restaurant owner, physicist, engineer business owner, and others. Five students indicated that a
good job meant having enough money, and two said that it was not working at McDonalds.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 106

ETS CogLab Report: NIES|2015
Student #21

New/VH040216

To what extent is each of the following a problem in your school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student tardiness

A

B

C

D

New/VH040217

b. Student health problems

A

B

C

D

New/VH040219

c. Student misbehavior in class

A

B

C

D

New/VH040221

d. Physical conflicts among students

A

B

C

D

New/VH040223

e. Bullying

A

B

C

D

New/VH040226

f.

A

B

C

D

New/VH040227

A

B

C

D

New/VH040231

Low student motivation

g. Low teacher expectations

Item 21
Reading: One student misread “moderate.”
Understanding: A few students had some trouble with some of the vocabulary used in the item, one had
trouble understanding “extent,” another had a problem with “physical conflict,” and some grappled with
adequately understanding “low student motivation” and “low teacher expectations.”
Changing Answers: One student changed their answer in row A from C to B after the word “extent” was
explained to them.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eighteen of 20 students responding felt the question was asking them about problems in their school.
One person felt the question was asking if any of the problems were personally experienced; one
respondent’s answer was dropped due to losing audio.
Probe 3: What does the phrase “student tardiness” mean to you?
All 22 respondents said that “student tardiness” meant being late—there was a combination of
respondents who thought this was just for class or for school overall. Additionally three students
thought that it meant being absent from school entirely.

ETS Post-Cognitive Interview Item Recommendations

Page 107

ETS CogLab Report: NIES|2015
Probe 4: In question a., what does the phrase “student health problems” mean to you?
Of the 22 respondents, 10 students felt that “student health problems” meant being sick or ill. Five
mentioned asthma, two brought up diabetes, one mentioned obesity, one brought up cancer, one said
broken limb, one student mentioned “disease” in general, and one talked about nutrition.
Probe 5: In question c., what does the phrase “student misbehavior in class” mean to you?
Eleven students out of 22 respondents felt that the probe about misbehavior could be characterized by
not paying attention or listening, seven said it had to do with being disruptive in class, three said it was
about being rude, one said talking in class, and one mentioned students acting up.
Probe 6: In question d., what does the phrase ““physical conflicts among students” mean to you?
Of the 22 respondents, 17 respondents felt that “physical conflicts” consisted of fights. Interestingly,
three students interpreted the question as having to do with physical activity such as sports and
exercise, and two said it meant bothering others.
Probe 7: In question d., what does the phrase "bullying" mean to you?
Fifteen of 23 students responding said that bullying had to do with picking on others (often those
younger or smaller), two mentioned teasing, two said it was being rude or mean, two mentioned talking
badly about someone, one student felt that it involved physical conflicts, and another student felt it had
to do with problem students.
Probe 8: In question f., what does the phrase “low student motivation” mean to you?
Of the 22 respondents, 13 indicated that “low student motivation” had to do with not trying in school,
three said it was students who did not want to be at school, two felt that they were students who
needed help, one indicated that they were kids with problems either at home or at school, one student
said they did not know, and two gave hard-to-characterize answers in the following:
“Sad, guilty, or it's nothing.”
“No, I think that's a large extent. Our teachers are always, even if a student doesn't think they can do it,
our teacher helps us, "You can do it. You can do it. Don't give up." I'm also in student council, so if
there's ever anybody that's feeling like they can't go down, because I'm the president, they will come
and talk to me. They'll ask for my advice. I also feel like I'm a guidance counselor. I guess they feel more
comfortable talking to me about it than they would an adult. I guess I would understand more. Low
student motivation, no, because we're always trying to help in any way that we can.”
Probe 9: In question g., what does the phrase “low teacher expectations” mean to you?
Students struggled a bit with “low teacher expectations.” Ten of the 22 respondents said that it was
teachers who expect little and do not push their students. Three respondents said it was teachers who
were not doing their jobs, two referenced mean teachers, one said it was teachers who did not
discipline, and four students said they did not know. There were two difficult-to-categorize answers:
“When the teacher would tell you to do something you wouldn't do it and you would like tell her no or
yeah…”
“Like teachers they need to go on the right path, trying to help them out, do everything good to them.
Just try to help them out on math, reading, science if a student won't know and a teacher will tell him
ETS Post-Cognitive Interview Item Recommendations

Page 108

ETS CogLab Report: NIES|2015
how to do it, give them the rules and everything. Like Rhonda Fry she helped us out on our rules
learning from mathematics.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Consider rephrasing “physical conflicts among students” to “physical fights among students” and
consider easier phrasing for “low student motivation” and “low teacher expectations.”

ETS Recommendation (05/02/2014)
Revise this matrix item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students
as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of
the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.
Drop sub-item “g” VH040231.
Rationale: ETS recommends dropping this item (VH040231) to reduce student burden. This sub-item
caused much more confusion for respondents than any of the other sub-items within this item.

ETS Post-Cognitive Interview Item Recommendations

Page 109

ETS CogLab Report: NIES|2015

Survey Instrument: Teacher Grade 4/Sequence I
Teacher #1

New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional
practices for American Indian or Alaska Native students in your classroom?
A. Not at all
B. Small extent
C. Moderate extent
D. Large extent

Item 1
Reading: NA
Understanding: Two respondents had a difficult time determining what “culturally specific instructional
practices for AI/AN students” meant. There was some question as to whether or not that meant
teaching about the culture or teaching in a culturally appropriate manner.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Fifteen teachers (out of 23 respondents) said that they felt the question was asking them to reflect on
whether they use cultural topics when teaching that are applicable to AI/AN students. Eight respondents
thought it might have to do with culturally appropriate practices of engaging and meeting the needs of
AI/AN students.
Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me
examples of what a reasonable time frame would be?
Twenty-one of the 24 respondents felt that the time frame of two years was appropriate. Of the three
who did not, reasons had to do with either being very new to teaching, or (for long-tenured teachers)
that five years might be a better measure.
Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the
last 24 months? If not, can you give me examples of what “the last two years” would be?
Of the 23 respondents, 20 teachers thought of “the last two years” as the last two school years, while
three thought of it as the last 24 months.

ETS Post-Cognitive Interview Item Recommendations

Page 110

ETS CogLab Report: NIES|2015
Probe 5: What does the phrase “culturally specific instructional practices for American Indian or
Alaska Native students” mean to you?
Twenty-four teachers responded to this item. Of these, 12 felt that “culturally specific instructional
practices” had to do with how one might make their teaching more culturally appropriate or responsive
with regard to being aware of and honoring cultural differences among their AI/AN students (i.e.,
adapting their teaching methods to meet the cultural needs of their students). Eleven mentioned
teaching AI/AN topics, three mentioned working to incorporate AI/AN vocabulary into their teaching,
and two said they made an effort to invite outside speakers to present on different AI/AN-specific
topics.
Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me an example for culturally specific instructional practices for American or
Alaska Native students that you have implemented in your classroom?
Seven teachers (out of 18 respondents) reported making lesson plans more interactive and hands-on so
as to better engage their AI/AN students, seven said they chose topics of interest for AI/AN students,
and four mentioned being mindful of cultural practices (e.g., not touching children, limited eye contact).
One mentioned having little leeway in the school-prescribed curriculum to offer other topics, and one
teacher again mentioned bringing in speakers to share different topics with the class.

Solutions to Improve Item
Respondent: NA
Interviewer: The number and type of probes for this question were reasonable. The question about
participants’ answers should come before asking them to restate the question as that seems to be the
natural flow of the interaction.

Initial KAI Recommendation (Keep/Change/Discard)
Change: It seems like the question could be clarified to better define for teachers if they are reflecting
on topics or instructional practice as the difference between the two. The rather in-depth analysis many
teachers seemed to give the question while trying to determine what it was actually asking suggests that
there is some lacking consensus on the actual subject of the question.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 111

ETS CogLab Report: NIES|2015
Teacher #2

Revised/VH040275

During the last two years, how many times have you attended professional or communitybased development programs (such as in-service classes and workshops, including online
classes) aimed at developing culturally specific instructional practices for American Indian or
Alaska Native students?
A. Never Skip Pattern.
B. 1 or 2 times
C. 3 or 4 times
D. 5 or more times

Item 2
Reading: One participant misread “culturally” as “cultural.”
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-one of 22 respondents interpreted the question appropriately. One answer was a non-sequitur
response.
Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me
examples of what a reasonable time frame would be?
Fifteen teachers (of the 19 respondents) said that they did think that two years was a reasonable time
frame, three said no (again, usually calling for either less or more time in the form of one or five years),
and one respondent said it depended on what the question was asking.
Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the
last 24 months? If not, can you give me examples of what “the last two years” would be?
Of the 17 who responded, 11 teachers said they thought of the last two years in school years, six said
they were thinking of 24 consecutive months.
Probe 5: What does the phrase “professional or community-based development programs” mean to
you?
Fourteen teachers out of 21 respondents felt that professional programs would have to do with
receiving training by professionals or the district on specific topics or in other areas to help them
maintain their teaching certification. Six felt that community programs would entail some kind of
training given specifically by the community to address community needs, and six additional
respondents said that community programs might be programs or workshops help by tribes. Two
mentioned that maybe this would have to do with culturally specific training.
ETS Post-Cognitive Interview Item Recommendations

Page 112

ETS CogLab Report: NIES|2015
Probe 6: What does the phrase “in-service classes” mean to you?
The answers varied greatly in specifics for this question, but all teachers (24 respondents in total) were
aware of what in-service classes were and what they meant for them and their certification.
Descriptions used to answer this question included that in-service classes were required, offered by the
school, offered by the district, a workshop, focused on academic topics, for credit, served as knowledge
enrichment, done in-school when students were not in school, were a class with other teachers, were
part of their job, a professional development day, and required for licensure.
Probe 7: When you answered this question, did you consider activities that were not in-service classes
and workshops or online classes? If yes, can you give me examples of what these other activities are?
Twelve teachers (out of 22 respondents) said “yes” and nine said “no.” Examples of other activities
mostly related to school and teacher training; six had to do with culturally specific learning related to
AI/AN students.
Probe 8a: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: a. What development programs have you attended?
Of the 14 respondents, six teachers said that they had attended development programs sponsored by
the school, three mentioned conferences, three mentioned presentations [Note: it was difficult to
determine if these were through the school, community, conference, or other], one mentioned a local
historical center and one said s/he still took classes.
Probe 8b: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: b. Were any of the programs you attended taught as online classes?
Eight teachers (out of 13 respondents) said “no” to online classes, three said “yes,” and three indicated
that they had done some online activities though these ranged from looking things up on the Internet to
attending a webinar.
Probe 8c: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
questions]: c. Have you attended development programs that are not through in-service classes?
Seven of 13 respondents said “yes,” three said “no,” and three indicated that they possibly had.
Examples were a conference, a class, and learning through peers.

Solutions to Improve Item
Respondent: NA
Interviewer: Some of the probes have multiple question components imbedded into one probe, making
the answers somewhat messy.

Initial KAI Recommendation (Keep/Change/Discard)
Change: The question is generally fine as is, but may be helpful to highlight the fact that the question is
specifically geared to learning more about AI/AN students.

ETS Post-Cognitive Interview Item Recommendations

Page 113

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 114

ETS CogLab Report: NIES|2015
Teacher #3

New/VH040281

To what extent have you implemented lessons learned from these professional or community-based
development programs in your classroom?
A. Not at all
B. Small extent
C. Moderate extent
D. Large extent

Item 3
Reading: NA
Understanding: There was one question about whether or not the time frame for the question was still
two years.
Changing Answers: One teacher changed their response from C to D when considering their response.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 17 respondents, 15 teachers appropriately restated the Item 3 question as having to do with
whether or not they were using gained knowledge in their classrooms. Two gave difficult-to-categorize
answers with the following:
“Have you taught a lesson that was geared on a topic maybe that your community had a need for in
your classroom?”
“As a part of that class that I took when I first came here, I had to teach a lesson in class and I also had to
come back to my specific class and teach a lesson.”
Probe 3: What does the phrase “lessons learned” from these programs mean to you?
Twelve of the 16 respondents felt that “lessons learned” essentially meant that they had gained
knowledge that they could put into practice. Three felt the question specifically related to new
knowledge gained. One gave a difficult-to-qualify answer:
“Hopefully there’s a positive outcome and that you want your lesson to be... That whoever is attending
get something that’s an eye-opener for them for their community, help them develop whether it’s their
students or the teachers. Make them more aware. I think awareness maybe overall.”
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of how you have implemented lessons learned from these
professional or community-based development programs in your classroom?
Answers to this probe varied in terms of topics noted as implementing lessons learned and new
knowledge. Of the 17 respondents, four mentioned that particular teaching practices they had learned
ETS Post-Cognitive Interview Item Recommendations

Page 115

ETS CogLab Report: NIES|2015
and made use of in their classrooms. Other responses had to do with using AI/AN stories (three), or
cultural practices into their teaching (two). The following responses were only mentioned by one
teacher for each topic: Singapore math, cultural awareness, common core, history, AI/AN language,
social justice, 3D, and using projects.

Solutions to Improve Item
Respondent: NA
Interviewer: Skip pattern not obvious at first.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 116

ETS CogLab Report: NIES|2015
Teacher #4

Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the
last two years? Fill in all ovals that apply.
A. State
B. District
C. Tribal education department
D. Indian education professional associations
E. College or university
F. Other (please specify): ________________

Item 4
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 16 respondents, 11 teachers felt that the question was trying to get at who sponsored or paid for
the programs in which they had taken part over the last two years. Three felt the question had to do
with where they were trained. Two gave difficult-to-classify answers:
“I interpret it what avenues are you pursuing to get professional development. Is it all just coming from
your school, is it all just coming from the state, or are you personally going out and looking for other
opportunities to get that development.”
“Well, I kind of think the point of it is to find out how effective…If you have a program and it was from a
certain place, how effective was it. Or what are the resources that are out there that you can take
advantage of, like are they coming from the state, are they coming from new district, from tribal, from
Indian education, or are they coming from colleges or universities? So that’s kind of what the question
means to me.”
Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: What is this funding source?
When asked about the funding source, the eight responses for this probe pointed to the BIE (two), the
tribe (two), Teach for America (two), STEP (one), PACER (one), NSF (one), and one “I don’t know.”

Solutions to Improve Item
Respondent: NA
ETS Post-Cognitive Interview Item Recommendations

Page 117

ETS CogLab Report: NIES|2015
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 118

ETS CogLab Report: NIES|2015
Teacher #5

Trend/VE012668

To what extent do you use the following to assess student progress? Fill in one oval on each line.
Not
at all

Small Moderate Large
extent
extent
extent

a. State assessments

A

B

C

D

Trend/VE012670

b. District assessments

A

B

C

D

Trend/VE012672

c. Assessments developed by American
Indian or Alaska Native organizations

A

B

C

D

Trend/VE012673

d. Tests supplied by textbook publishers
(for example, end of unit or chapter
tests)

A

B

C

D

Trend/VE012674

e. Teacher-made tests or quizzes

A

B

C

D

Trend/VE012675

f.

A

B

C

D

Trend/VE012676

g. Group projects

A

B

C

D

Trend/VE012678

h. Oral responses of students during class
discussions

A

B

C

D

Trend/VE012681

A

B

C

D

New/VH040300

i.

Performance-based assessments

Assessments to evaluate English
language proficiency

Item 5
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eight teachers (out of 21 respondents) thought the question was asking them what assessments they
used, seven thought the question referred to how they assess their students, two thought the question
was asking how often they assessed students, and one felt the question had to do with how well s/he
assessed students. Four respondents gave difficult-to-qualify answers [Note: one is too long to include in
this report]:

ETS Post-Cognitive Interview Item Recommendations

Page 119

ETS CogLab Report: NIES|2015
“Well, because one of the things…Going back to the gentleman who gave me the same, basically
American Indian students learned differently. If you're going to assess their performance, I would argue
it's not just American Indian community that I know may sound differently. If you are only using things
like state assessments and districts assessment, you're really not getting at the whole picture and I
wholeheartedly agree with that especially since a lot of those things I see have definite [inaudible
00:21:40].”
“I only think this because part B surprised me so much because I haven't actually seen any assessments
developed by American Indian or Alaska Native organizations, so in my mind it is just to cue my
awareness, are you using assessments that are going to be fair and specific and incorporating the culture
into the assessment? So often that can be a determining factor in success.”
“I think it's demonstrating more than asking that there's a lack of input from...In assessments from
American Indian and Alaska Native organizations. It could be how important I would state the
assessments in my own teaching. Yeah, I mean look at that stuff. Usually the teacher made it tough, the
day to day quizzes and I have districts to place our tests for things like Math. I supposed that helps guide
us to see clearly what the students need more help in kind of thing with their stem of understanding.”
Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency”
mean to you?
This question seemed to confuse many respondents: none of the 23 respondents reported using an
assessment to evaluate English language proficiency. All 23 teachers who responded had a notion of
what English language proficiency testing consisted of, but it seemed from their answers that this was a
confusing question. It is hard to say whether or not this is because most seemed to associate language
proficiency assessment specifically with students of foreign descent.
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask
the follow-up question]: Can you give me examples of how you have used “assessments to evaluate
English language proficiency” to assess student progress?
Answers to this probe varied greatly, suggesting some validity issues with teacher response choices to
the item. Respondents lacked the ability to explain what they were doing in their own classrooms to
assess English language proficiency. Of the 15 respondents, four teachers used school-based tests as
their answer (i.e., tests given to students across the board at the school level), three mentioned that
they might better understand a student’s English proficiency through the use of math problems, one
used team meetings to go over test results (bit did not specify a type of test), one said s/he simply
listened, one said reports, one said the language proficiency assessment, and one said s/he did not
know. There were three answers difficult to categorize:
“Well, the assessments to evaluate English language, they're only used with students who speak two
languages. [Inaudible 00:18:39] students when English isn't necessarily their first language, yeah.”
“I know at first for me, I was like, I don't teach English, I teach Ojibwe. So I have to try and teach the
structure and the makeup of the language and how it's set up different than English. But I don't test
English so for me, that's opposite of what I do. Then after I thought, when you told me to focus in on it
then I thought oh, maybe it's for first speakers who are in maybe an ELL program.”

ETS Post-Cognitive Interview Item Recommendations

Page 120

ETS CogLab Report: NIES|2015
“Maybe we’re practicing for our NMSBA tests that are coming up. From the very beginning, kids will
come in and I would ask them a question, they would say no, yes or this and that. Then we have to tell
them, you need to answer me in complete sentence, complete thought. We have to practice like this so
by the time the tests come around, hopefully, they’ll be able to answer in complete sentences and not
just by one or two words.”
Probe 5: In question c., what specific organizations did you consider when answering this question?
Of the 21 respondents, nine teachers said they did not know of any specific organizations, three said
that they did not consider any specific organization, two teachers mentioned the BIA, two mentioned
Indian Education departments, and two said they thought of tribal organizations. One reported using the
local text, one thought of AI/AN organizations, and one specifically mentioned NIEA.
Probe 6: When answering this question did you consider NIES information about student progress? If
yes, in which question specifically did you consider NIES information about student progress?
All 21 respondents said that they had not considered the NIES when answering the question, with many
noting that they had never heard of the NIES.

Solutions to Improve Item
Respondent: NA
Interviewer: Is this question meant to be specific to AI/AN students? If so, that should probably be
explicit in the question asked.

Initial KAI Recommendation (Keep/Change/Discard)
Change: If this item is geared toward AI/AN students, consider highlighting the specification. Also,
determine whether or not the item is referring specifically to assessments used by teachers or by the
school. And last but not least, is there a time frame to be associated with this question?

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking
about AI/AN students.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 121

ETS CogLab Report: NIES|2015
Teacher #6

New/VH043923

Do you teach reading/language arts to grade 4 students?
A. Yes  Go to Question 7
B. No  Go to Question 8

Item 6
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of the 20 respondents seemed to understand the meaning of the question. One response
recorded consisted of “No, I didn’t.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 122

ETS CogLab Report: NIES|2015
Teacher #7

Revised/VH040381

How often do you integrate materials about the following topics into your reading/language arts
lessons? Fill in one oval on each line.

Never

At least
once a
year

At least
once a
month

At least
once a
week

Every day
or almost
every day

a. American Indian or Alaska
Native culture or history

A

B

C

D

E

Revised/VH0
40385

b. Current issues affecting
American Indian or Alaska
Native people or
communities

A

B

C

D

E

Revised/VH0
40386

Item 7
Reading: NA
Understanding: NA
Changing Answers: One respondent changed their answer for question A to C from D, and for question
B from D to C.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 19 respondents interpreted the item as it was intended.
Probe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history”
mean to you?
Twenty out of 20 respondents either worked to explain or gave examples of the meaning of “AI/AN
culture or history.” Some respondents felt the question was straightforward, thus leading to redundant
answers. It would be better in the future to simply ask for examples to determine understanding of the
concept.
Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska
Native people or communities” mean to you?
Eighteen teachers (out of 18 respondents) gave answers indicating that they did indeed have an
understanding of the phrase “current issues affecting AI/ANs.” Examples given in the course of
answering included: treaties, land rights, sovereignty, historical trauma, economies, casinos, lifestyle,
health, and poverty.

ETS Post-Cognitive Interview Item Recommendations

Page 123

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b.,
or to both, then ask the follow-up question]: Can you give me examples of how you have integrated
materials about the topics into your reading/language arts lessons?
Five of the 16 respondents said they used state history to integrate topics into their lessons. Five said
they used stories, traditional and otherwise. Three mentioned crafts from creating regalia to basket
weaving, one mentioned language, and one mentioned dance. Issues of how much leeway teachers had
with their state or district curriculum was again brought up by one respondent.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 124

ETS CogLab Report: NIES|2015
Teacher #8

New/VH043941

Do you teach mathematics to grade 4 students?
A. Yes  Go to Question 9
B. No  Go to Question 10

Item 8
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty teachers (out of 20 respondents) seemed to easily understand what the item was asking.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 125

ETS CogLab Report: NIES|2015
Teacher #9

Revised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in
one oval on each line.

Never

At least
once a
year

At least
once a
month

At least
once a
week

Every day
or almost
every day

a. American Indian or Alaska
Native culture or history

A

B

C

D

E

Revised/VH
040390

b. Current issues affecting
American Indian or Alaska
Native people or communities

A

B

C

D

E

Revised/VH
040392

Item 9
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of 18 respondents, all 18 teachers understood the item.
Probe 3: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b.,
or to both, then ask the follow-up question]: Can you give me examples of how you have integrated
materials about the topics into your mathematics lessons?
Only 11 respondents provided answers for Probe 3. These ranged from teachers using questions about
ancient or current populations, measurements on how to build things (such as teepees or drums),
counting in an AI/AN language, culturally specific word problems, data from current issues such as water
rights, and mapping and scaling using math.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Post-Cognitive Interview Item Recommendations

Page 126

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 127

ETS CogLab Report: NIES|2015
Teacher #10

New/VH043953

How much do you agree with each of the following statements about the materials available in your
school library, media center, or resource center? Fill in one oval on each line.
Strongly
disagree

Disagree

Agree

Strongly
agree

a. The number of books and materials
available for 4th grade students is
sufficient.

A

B

C

D

New/VH043957

b. The quality of the books and
materials available for 4th grade
students is satisfactory.

A

B

C

D

New/VH043959

c. The number of books and materials
specific to American Indian and
Alaska Native culture available for
4th grade students is sufficient.

A

B

C

D

New/VH043964

d. The quality of the books and
materials specific to American
Indian and Alaska Native culture
available for 4th grade students is
satisfactory.

A

B

C

D

New/VH043966

Item 10
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen teachers (out of 21 respondents) accurately explained that the question had to do with
whether or not teachers thought there were enough satisfactory books for 4th graders. Only 10
specifically mentioned AI/AN-specific books.
Probe 3: In questions a. and c., what does the phrase “is sufficient” mean to you?
All 21 respondents mentioned amounts of books in their response to what “sufficient” meant to them.
One felt that the question had to do with quality, one said access, and one felt that it had to do with
averages (i.e., sufficiency serving as another word for average, as does satisfactory).
ETS Post-Cognitive Interview Item Recommendations

Page 128

ETS CogLab Report: NIES|2015
Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you?
All 24 respondents provided an answer to the probe. Nine teachers felt that the question about what
“satisfactory” means talked about the idea of averages. Five felt the question was talking about quality,
four mentioned if the books were contemporary, four said if books were appropriate for the reading
age, three mentioned the condition of the books, two said it meant if there were enough books, and one
felt it had to do with whether or not books were relevant.
Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean
to you?
Fourteen of the 24 respondents said that the quality of books and materials had to do with content (i.e.,
whether it was appropriate), seven mentioned the condition of the books, seven mentioned whether or
not the books were contemporary, and four felt it had to do with whether or not the books were for the
appropriate reading age.
Probe 6: In questions c. and d., what does the phrase “specific to American Indian and Alaska Native
cultures” mean to you?
Of the 23 respondents, 11 teachers said that they felt that materials specific to AI/AN cultures should be
historically accurate or accurate in general. Ten mentioned the word “specific” either in terms of
explaining what they thought of the phrase posed to them, or sometimes having to do with the notion
that materials should be about specific tribes or groups. Three teachers mentioned that materials should
be authored by AI/AN writers.

Solutions to Improve Item
Respondent: There should be an “I don’t know” category.
Interviewer: The way the probes were phrased generated quite a deal of reflection on the part of
teachers, but it is notable that the vast majority of teachers focused only on books as “materials.”

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 129

ETS CogLab Report: NIES|2015
Teacher #11

New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student absenteeism

A

B

C

D

New/VH040413

b. Student tardiness

A

B

C

D

New/VH040414

c. Student health problems

A

B

C

D

New/VH040416

d. Student misbehavior in class

A

B

C

D

New/VH040422

e. Physical conflicts among students

A

B

C

D

New/VH040424

f.

A

B

C

D

New/VH040425

g. Low student aspirations

A

B

C

D

New/VH040427

h. Low teacher expectations

A

B

C

D

New/VH040428

i.

A

B

C

D

New/VH040430

Bullying

Low family involvement

Item 11
Reading: NA
Understanding: One teacher wanted to know if the timeline for this question was still two years.
Another seemed to not understand the question but answered all subsequent probes appropriately.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty-two teachers (out of 22 respondents) seemed to appropriately understand what the question
was trying to invoke.
Probe 3: In question a., what does the phrase “student absenteeism” mean to you?
Nineteen of the 24 respondents felt “absenteeism” meant students not at school; seven felt it had to do
with the rate at which students were missing school.
Probe 4: In question b., what does the phrase “student tardiness” mean to you?
Of the 24 respondents, 21 teachers said that “tardiness” meant being late; five teachers mentioned the
rate at which students were late.
ETS Post-Cognitive Interview Item Recommendations

Page 130

ETS CogLab Report: NIES|2015
Probe 5: In question c., what does the phrase “student health problems” mean to you?
Thirteen of the 24 teachers who responded to the probe felt that the question regarding student health
problems involved students being sick. Five mentioned asthma, four talked about diabetes, three
brought up allergies, two mentioned diet and nutrition, two noted disabilities (including learning
disabilities), two mentioned physical problems, two noted mental problems, and one mentioned dental
and one vision problems. Other issues mentioned were epilepsy and ADD.
Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you?
Eleven teachers (out of 24 respondents) mentioned disruptive or distracting behavior as constituting
student misbehavior. Nine brought up students not following the rules, and five mentioned students
being disrespectful or talking in class.
Probe 7: In question e., what does the phrase “physical conflicts among students” mean to you?
Of the 24 respondents, 18 teachers felt that physical conflicts among students constituted fighting. Six
teachers mentioned students putting their hands on each other, five mentioned hitting, three brought
up kicking, and two said pushing.
Probe 8: In question f., what does the phrase “bullying” mean to you?
Eighteen of the 24 respondents said that bullying constituted repeated behavior such as picking on
others. Ten teachers mentioned verbal bullying such as name calling, while cyber bullying was also
brought up by three teachers. Nine teachers felt that bullying could also be physical.
Probe 9: In question g., what does the phrase “low student aspirations” mean to you?
Fourteen teachers (out of 23 respondents) mentioned motivation of students as an issue in low student
aspirations. Nine brought up students not feeling like they had much to look forward to with regard to
their futures, and seven mentioned a lack of self-esteem.
Probe 10: In question h., what does the phrase “low teacher expectations” mean to you?
All 24 teachers responded to the probe. Of these, 13 teachers felt that low teacher expectations were
akin to teachers not pushing their students, and 12 mentioned teachers with low standards for their
students’ achievements.
Probe 11: In question i., what does the phrase “low family involvement” mean to you?
All 24 teacher responses were consistent on ideas of what might constitute family involvement, but not
all teachers answered the question in terms of what they thought low family involvement entailed.
Different aspects discussed included whether or not parents volunteered at the school, sent notes to the
teacher, were responsive to teacher communication, attended parent-teacher conferences, attended
school events, etc.

Solutions to Improve Item
Respondent: One respondent asked about whether or not they should be reflecting on the class level,
grade level, or school level.

ETS Post-Cognitive Interview Item Recommendations

Page 131

ETS CogLab Report: NIES|2015
Interviewer: Is this item meant to elicit reflection on the entire school (which may be K-12 sometimes)
or just for the 4th grade?

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Be more specific about the level on which teachers should be focus when answering the item.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students
as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of
the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 132

ETS CogLab Report: NIES|2015

Survey Instrument: Teacher Grade 8/Sequence I
Teacher #1

New/VH040247

During the last two years, to what extent have you implemented culturally specific instructional
practices for American Indian or Alaska Native students in your classroom?
A. Not at all
B. Small extent
C. Moderate extent
D. Large extent

Item 1
Reading: NA
Understanding: Two respondents had trouble with the term “culturally specific instructional practices”
and what that meant.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Validity of this item is questionable: 16 out of 23 respondents felt that “implementing culturally specific
instructional practices” had to do with covering topics related to AI/ANs, four felt like it had to do with
actual teaching practices, one said that s/he did not know, and two expressed that they were not sure
what the term referred to. As one of the unsure noted:
“I think the thing that catches me about it is what they might mean by an instructional practice because
I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s
why I would go with a small extent. I actually started a class called Engineering from a Native
Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best
practices for all students.”
Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me
examples of what a reasonable time frame would be?
Of the 23 respondents, 19 expressed that they thought two years was a good time frame. One felt that
four to five years was a better time frame, two said that one year was a better time frame, and one felt
that six months was a more appropriate time frame. One response was hard to characterize:
“I have done it throughout my career and I have taught many different grade levels. I think that we
always have Native students in classes. We also have Hispanic and Non-Natives in other schools I have
taught in and I think that everybody needs to know how rich the Native culture is. Sometimes we are

ETS Post-Cognitive Interview Item Recommendations

Page 133

ETS CogLab Report: NIES|2015
marginalized to Native American Day. The Hispanics are Cinco de Mayo. That just makes me angry. I
think that we need to be integrated and respected for the rich history that we have got.”
Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the
last 24 months? If not, can you give me examples of what “the last two years” would be?
Twenty-one of 22 respondents felt that “the last two years” referred to school years. One said 24
months.
Probe 5: What does the phrase “culturally specific instructional practices for American Indian or
Alaska Native students” mean to you?
There was quite a bit of variance in this question; 15 of 23 respondents felt that “culturally specific
instructional practices” referred to incorporating cultural material into lessons, six felt it had to do with
culturally sensitive teaching practices, three said language, three said they were unsure what the phrase
meant, and one gave a difficult-to-characterize answer:
“To me, the Lakota culture is an oral culture, and there's a lot more listening and talking in my room, in
part because I'm a big talker, but also that's just traditionally how Native parents would have educated
their children. I think it becomes vague because there are so many, I'm sure this is exactly what you
want on this recording, I think it becomes vague because there are so many generations of Native
parents that have led to the students that are in my classroom today that were not raised in the
traditional household, traditionally by the Native parents.”
Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me an example for culturally specific instructional practices for American or
Alaska Native students that you have implemented in your classroom?
There were 19 respondents for this question, and the answers varied: six respondents indicated that
they implemented culturally specific practices through language, four mentioned science and nature,
three said arts and crafts, two mentioned Native American History Month, one reported doing so
through a school bulletin, one mentioned ELL practices, one said history, one reported not
implementing practices, one described being mindful of eye contact, one said honoring shyness, and
one described pairing students.

Solutions to Improve Item
Respondent: One respondent suggested that an example of what is meant by “culturally specific
instructional practices” would be helpful.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: It seems that this question is an engaging one; however, as one of the respondents suggested,
it would be very helpful to be specific about what is meant by “culturally specific instructional
practices,” whether that refers to teaching methods or incorporating culturally appropriate material.
And if both are intended, then perhaps this could be reconfigured as a matrix question.

ETS Post-Cognitive Interview Item Recommendations

Page 134

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 135

ETS CogLab Report: NIES|2015
Teacher #2

Revised/VH040275

During the last two years, how many times have you attended professional or communitybased development programs (such as in-service classes and workshops, including online
classes) aimed at developing culturally specific instructional practices for American Indian or
Alaska Native students?
A. Never Skip Pattern.
B. 1 or 2 times
C. 3 or 4 times
D. 5 or more times

Item 2
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 21 respondents, 20 adequately noted that the question was asking about attending other
training opportunities. Only 12 specifically mentioned AI/ANs in their restatement of the answer, one
respondent thought the question had to do with implementing what was learned from the trainings,
and one thought it had to do with language.
Probe 3: Do you consider “the last two years” to be a reasonable time frame? If not, can you give me
examples of what a reasonable time frame would be?
Fifteen of 19 respondents said that the last two years was a reasonable amount of time. Three said no,
that it should be a longer time frame (again, five years was a common suggestion), and one said it
should be less time.
Probe 4: Do you consider “the last two years” to be this school year and last school year, or to be the
last 24 months? If not, can you give me examples of what “the last two years” would be?
Of 17 respondents, 10 said that they thought of the last two school years as school years, and five said
they thought of it as 24 months. One said “year round,” and another said they were thinking of this year.
Probe 5: What does the phrase “professional or community-based development programs” mean to
you?
Responses from 20 respondents were somewhat varied to this probe. Ten felt that the professional
development programs were provided by the school or district. Answers about community-based
programs ranged with eight respondents essentially saying that they thought this meant that the
community was involved—though it varied by respondent as to whether they thought it was hosted by
ETS Post-Cognitive Interview Item Recommendations

Page 136

ETS CogLab Report: NIES|2015
the community or just involved the community. Five specifically mentioned AI/AN communities, four felt
that the programs were offered by an outside source. Other comments were that it had to do with
cultural integration or that it was something you could get credit for, and one person mentioned
conferences.
Probe 6: What does the phrase “in-service classes” mean to you?
Ten of 20 respondents said that in-service classes were offered by the district, seven said they were held
in school, three said they involved professional development, and two respondents seemed to think that
it had to do with shadowing someone else (like a student teacher).
Probe 7: When you answered this question, did you consider activities that were not in-service classes
and workshops or online classes? If yes, can you give me examples of what these other activities are?
Eight of 22 respondents said “no,” and seven said that other activities they were thinking of were
community events. Two mentioned assemblies, one said they went to see a tribal council meeting, one
mentioned conferences, one said they were thinking of a partnership with a local organization, one said
they taught online classes and learned from that experience, and one said they did not know.
Probe 8a: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: a. What development programs have you attended?
Five of 13 teachers responding mentioned outside speakers when asked about other development
programs they had attended. Five mentioned AI/AN groups, four noted cultural training, three brought
up nearby universities as resources, one mentioned community organizations, one said Teach for
America, one said GLAD (guided language acquisition design), and another said they did not remember.
Probe 8b: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the followup questions]: b. Were any of the programs you attended taught as online classes?
Eleven of 13 respondents said that they had not attended any online classes. Two said that they had but
that they were not Native-specific.
Probe 8c: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
questions]: c. Have you attended development programs that are not through in-service classes?
Seven of 14 responding teachers said that they had attended development programs that were not
through in-service classes and six said they had not. One said they did not remember. Of those who said
yes, conferences, community events, presentations, and workshops were mentioned.

Solutions to Improve Item
Respondent: One respondent suggested a larger time frame than two years for this item.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: We recommend highlighting American Indian/Alaska Native students.

ETS Post-Cognitive Interview Item Recommendations

Page 137

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 138

ETS CogLab Report: NIES|2015
Teacher #3

New/VH040281

To what extent have you implemented lessons learned from these professional or community-based
development programs in your classroom?
A. Not at all
B. Small extent
C. Moderate extent
D. Large extent

Item 3
Reading: NA
Understanding: NA
Changing Answers: One respondent changed their answer from B to D in response to Probe 3.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eleven of 13 respondents to this question felt that the question had to do with whether or not they had
implemented lessons learned. Two gave hard-to-classify answers in the following:
“I think this is trying to ask that, I’m saying that the professional and community-based development
programs in my classroom has happened according to the language that I teach and the history that I
bring. However, somebody has to teach some others like me so that all of us teachers can do what I do.”
“I think the thing that catches me about it is what they might mean by an instructional practice because
I am not certain whether they mean my curriculum or how I’m actually teaching my curriculum, so that’s
why I would go with a small extent. I actually started a class called Engineering from a Native
Perspective, but that doesn’t necessarily have to do with instructional practices because I try to use best
practices for all students.”
Probe 3: What does the phrase “lessons learned” from these programs mean to you?
Eight out of 15 respondents felt the question had to do with what implementation they had done of
lessons learned. Seven indicated that it had to do with what they had taken away from trainings. Two
responses were hard to categorize:
“I’m 65 years old. I think some of these lessons learned, I think I need to know some something about
the Southwest, you know, for me to make inference of what they do and make a direct connection to
maybe the Ojibwe way of doing that also. Let’s say, they do maple sugar right now here in
Minnesota...During this time, what are they doing in New Mexico? What are they doing in the Pacific
Northwest? What do they do up in Six Nations Iroquois Confederacy area? What are they doing over
here about the Blue Corn Festival? What’s going on with the drum? These kind of things. People do not
know anything about that and that’s what I think is real. We have a living, truly beautiful, living culture,
ETS Post-Cognitive Interview Item Recommendations

Page 139

ETS CogLab Report: NIES|2015
but it’s all in books and these kids are going to read and make their worksheets and they’re going to be
done and get graded, move on.”
“Well, I was hoping that'd be the in-service lessons that we learned. Well, as an educator, it's just
something that you learned in the classroom situation or something like that, not necessarily learned on
the street (which could be ‘lessons learned,’ but I meant it literally).”
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of how you have implemented lessons learned from these
professional or community-based development programs in your classroom?
Six of the 14 respondents gave examples of including specific topics into their teaching (not all of these
answers were specific to AI/ANs), six responded that they had made sure to be more culturally sensitive
in their teaching methods, two brought up professional development in general, and one mentioned the
GLAD program.

Solutions to Improve Item
Respondent: One respondent noted that they found the item too Euro-centric.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Is this question meant to invoke whether or not teachers implement lessons learned in general or
whether they are implementing lessons learned specifically targeting AI/AN students? If the latter, it
should be noted in the question and highlighted.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 140

ETS CogLab Report: NIES|2015
Teacher #4

Revised/VH040284

Who sponsored the professional or community-based development programs you attended in the
last two years? Fill in all ovals that apply.
A. State
B. District
C. Tribal education department
D. Indian education professional associations
E. College or university
F. Other (please specify): ________________

Item 4
Reading: NA
Understanding: One subject indicated that they were a little confused about answer option C regarding
the tribal education department and whether that referred specifically to a department on their
reservation or if it was meant more generally.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twelve of 14 respondents felt that question was asking who had sponsored any community-based
development programs they may have attended. One respondent thought the question was asking if
they had gone, and two responses were hard to categorize:
“I'm a little confused about whether that means a department specifically within our tribe that we have
on our reservation or if that's just more general. I know CCSD puts on trainings. I don't know whether
the ones that I attended were sponsored by CCSD. I don't have the impression that they were. I have the
impression that it was more like our individual schools reach out to the tribal reservation and set it up
that way, but I do know that the district offers trainings.”
“Professional development, which we do it every day, or we have it every other week. We do it every
other week, because I'm the head of the math department. We have some kind of peer share, we pull
videos up from the web site teaching channel and I implement it a large extent. We have a lady coming
in also, doing professional development, she comes about every three months and she asked us to read
a book and to implement it into our teaching.”

ETS Post-Cognitive Interview Item Recommendations

Page 141

ETS CogLab Report: NIES|2015
Probe 3: [Conditional probe: If the participant selects answer choice F, then ask the follow-up
question]: What is this funding source?
Of the five teachers who responded, one said they did not know, one said Teach for America, one said
ELO (unsure of what this stands for), one said the school where they taught, and one said the Minnesota
Literary Council.

Solutions to Improve Item
Respondent: One respondent said, “To me it automatically makes me think of the school district, which
out here, like for our school, is kind of meaningless, because we're the only school in our school district.
Tribal Education Department, Indian Education Professional Associations, college university, for me, I
guess the people that actually sponsored mine were a college or a university, Indian Education
Professional Associations, because I feel like the Tribal Education Department is nested within that, and
then the Tribal Education Department and the district.”
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 142

ETS CogLab Report: NIES|2015
Teacher #5Trend/VE012668
To what extent do you use the following to assess student progress? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. State assessments

A

B

C

D

Trend/VE012670

b. District assessments

A

B

C

D

Trend/VE012672

c. Assessments developed by American
Indian or Alaska Native organizations

A

B

C

D

Trend/VE012673

d. Tests supplied by textbook
publishers (for example, end of unit
or chapter tests)

A

B

C

D

Trend/VE012674

e. Teacher-made tests or quizzes

A

B

C

D

Trend/VE012675

f.

A

B

C

D

Trend/VE012676

g. Group projects

A

B

C

D

Trend/VE012678

h. Oral responses of students during
class discussions

A

B

C

D

Trend/VE012681

A

B

C

D

New/VH040300

i.

Performance-based assessments

Assessments to evaluate English
language proficiency

Item 5
Reading: NA
Understanding: One respondent noted that they did not think of grades as a measure of student
progress and further indicated that they did not know what “assessments developed by AI/AN
organizations” meant.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 23 respondents, 22 thought the question had to do with how they assessed their students. One
respondent thought it had to do with whether or not AI/AN students could understand questions.

ETS Post-Cognitive Interview Item Recommendations

Page 143

ETS CogLab Report: NIES|2015
Probe 3: In question i., what does the phrase “assessments to evaluate English language proficiency”
mean to you?
Out of a total of 22 respondents to this question, 12 referenced either ELL, ESL, or ELP as assessments
for English language. Many of these respondents additionally commented that they were unsure if
AI/AN students received these assessments, mentioned that the counselor administered these tests, or
noted that all of their students were English speakers. Eight teachers responding felt that the question
was simply asking about testing English, and two indicated that they did not know.
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question i., then ask
the follow-up question]: Can you give me examples of how you have used “assessments to evaluate
English language proficiency” to assess student progress?
There were 12 respondents for this question and three said they looked at vocabulary, two mentioned
writing assignments, and one said ESL. One brought up that they were a math teacher and another said
they were a science teacher—both indicated that they did not do English assessments. One teacher said
they used oral assessments and another said they used the ACCESS test but said this was used when
they taught in the Caribbean. Two gave difficult-to-classify answers:
“That’s the only thing that I’m thinking about when I think of assessments to evaluate English. Specific
involves what is in that test or other examples?”
“The only thing I can think of is the language program that OLC has and then Red Cloud has their own
language program too so they develop their own curriculum, all of that stuff.”
Probe 5: In question C, what specific organizations did you consider when answering this question?
Eight of 21 respondents said that they had not considered any specific organizations when answering
questions, seven mentioned some specific AI/AN organizations or groups including tribes. One said the
state, four said they did not know, and one said the Northwest Regional Lab.
Probe 6: When answering this question did you consider NIES information about student progress? If
yes, in which question specifically did you consider NIES information about student progress?
Out of 20 respondents, 17 indicated that they had not considered the NIES, many noting they had never
heard of it before. One said yes and two gave difficult-to-classify answers:
“Yeah. When I read it, I actually froze up because I am anti-assessments or anti-standardized
assessments. When I read that, that's all I felt right away and it makes me defensive because I do not
follow this prescription of those types of assessments. That being said though, my students, they have
high success rates. They're high achievers. The way I assess them is at a different level. No. It's just a
feeling. I just have a general stereotypical feeling, I guess.”
“I guess when I read it, I was really thinking about how I assess my students in class and where I get that
from.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA
ETS Post-Cognitive Interview Item Recommendations

Page 144

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: Consider clarifying the question by adding “in your classroom” if the question is intended to
capture what teachers themselves are doing rather than all assessments used at school. Otherwise, this
item seems generally to be understood.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking
about AI/AN students.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 145

ETS CogLab Report: NIES|2015
Teacher #6

Trend/VE012771

Do you teach reading/language arts to grade 8 students?
A. Yes  Go to Question 7
B. No  Go to Question 8

Item 6
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Interestingly the 21 responses given varied more than expected, likely because the question is
straightforward, causing some respondents to overly analyze the question. Seventeen thought the
question was asking them if they taught reading or language arts. Seven of these specifically mentioned
the 8th grade, and one said middle school. One teacher simply said “no,” and three teachers gave
difficult-to-categorize answers:
“Teaching them strategies on how to understand what they're reading and what they're writing, how to
write sentences.”
“I think specifically if you have a reading language arts license, I would have answered yes.”
“I think that’s a departmentalized kind of a thing. We have reading teachers. We have language arts
teachers. As an elementary teacher, this would’ve referred to me living on a reservation. [I either
00:32:34] do everything.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 146

ETS CogLab Report: NIES|2015

ETS Post-Cognitive Interview Item Recommendations

Page 147

ETS CogLab Report: NIES|2015
Teacher #7

Revised/VH040381

How often do you integrate materials about the following topics into your reading/language arts
lessons? Fill in one oval on each line.

Never

a. American Indian or Alaska
Native culture or history

b. Current issues affecting
American Indian or Alaska
Native people or communities

A

A

At least
once a
year

B

B

At least
once a
month

At least
once a
week

C

C

D

D

Every day
or almost
every day

E

Revised/
VH0403
85

E

Revised/
VH0403
86

Item 7
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of 10 respondents, six mentioned AI/ANs specifically when answering the question about
integrating materials. Four simply mentioned implementing culturally specific material.
Probe 3: In question a., what does the phrase “American Indian or Alaska Native culture or history”
mean to you?
Answers from the 10 respondents to this question were all over the map. Three teachers mentioned
stories, four mentioned history, one pointed to tradition, one said identity, and one said that they did
not know.
Probe 4: In question b., what does the phrase “current issues affecting American Indian or Alaska
Native people or communities” mean to you?
All nine respondents pointed to current issues (such as the Keystone pipeline); other themes mentioned
by the remaining respondents were treaties, mascots, suicide, government, issues with the state,
alcohol, land rights, and suffering.

ETS Post-Cognitive Interview Item Recommendations

Page 148

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b.,
or to both, then ask the follow-up question]: Can you give me examples of how you have integrated
materials about the topics into your reading/language arts lessons?
The 10 responses received for this question were very varied. Five respondents pointed to covering
issues in their classes, two said they talked about history, one teacher said something about MLK Day
(the rest of the answer had to do with having an African American president and needing an AI/AN
president), one said they had students read Native authors, and one pointed to Native American History
Month.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is. Respondents seem comfortable with the item. We suggest asking for examples in the
probes rather than asking what phrases mean to respondents as the simplicity of the question often
leads to unnecessarily complicated answers.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 149

ETS CogLab Report: NIES|2015
Teacher #8

Trend/VE012783

Do you teach mathematics to grade 8 students?
A. Yes  Go to Question 9
B. No  Go to Question 10

Item 8
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of 20 respondents, 17 felt the question was asking them if they were math teachers; of these 17,
eight teachers mentioned 8th graders. Three teachers stated that the question was trying to get at what
one’s subject area was.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 150

ETS CogLab Report: NIES|2015
Teacher #9

Revised/VH040387

How often do you integrate materials about the following topics into your mathematics lessons? Fill in
one oval on each line.

Never

a. American Indian or Alaska
Native culture or history
b. Current issues affecting
American Indian or Alaska
Native people or communities

A

A

At least
once a
year

B

B

At least
once a
month

C

C

At least
once a
week

D

D

Every day
or almost
every day

E

Revised/
VH0403
90

E

Revised/
VH0403
92

Item 9
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All six respondents understood the question to be asking how they implemented AI/AN-specific culture
or current issues into their teaching.
Probe 3: [Conditional probe: If the participant selects answer choice B, C, D, or E to questions a. or b.,
or to both, then ask the follow-up question]: Can you give me examples of how you have integrated
materials about the topics into your mathematics lessons?
Out of eight respondents, six spoke to using local context—whether that be local animals or fruits or
history. One mentioned using the local language and one made a statement about Native American
History Month but did not elaborate on what that looked like in their math class.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.
ETS Post-Cognitive Interview Item Recommendations

Page 151

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 152

ETS CogLab Report: NIES|2015
Teacher #10

New/VH040395

How much do you agree with each of the following statements about the materials available in your
school library, media center, or resource center? Fill in one oval on each line.
Strongly
disagree

Disagree

Agree

Strongly
agree

a. The number of books and materials
available for 8th grade students is
sufficient.

A

B

C

D

New/VH040399

b. The quality of the books and
materials available for 8th grade
students is satisfactory.

A

B

C

D

New/VH040401

c. The number of books and materials
specific to American Indian and
Alaska Native culture available for
8th grade students is sufficient.

A

B

C

D

New/VH040402

d. The quality of the books and
materials specific to American
Indian and Alaska Native culture
available for 8th grade students is
satisfactory.

A

B

C

D

New/VH040404

Item 10
Reading: NA
Understanding: Two respondents assumed that rows A and B also referred to AI/AN materials.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 20 respondents felt that the question was asking about the types of materials available to their
students. Ten teachers specifically mentioned the 8th grade and eight mentioned AI/ANs specifically.
Probe 3: In questions a. and c., what does the phrase “is sufficient” mean to you?
Answers from 22 respondents to this question varied greatly with seven saying they thought that
“sufficient” meant enough, five said it meant minimum or adequate, four said it was a subjective term,
four said it referred to quality, four felt it meant availability, one said variety, and one gave a hard-toqualify answer: “It's looking at resources in the building so that if we do want to work in ... I'm thinking
ETS Post-Cognitive Interview Item Recommendations

Page 153

ETS CogLab Report: NIES|2015
especially in terms of history and language arts. I'm doing things in ... Type into it to Native American
culture and kind of geared toward that, do we have the resources in the building.”
Probe 4: In questions b. and d., what does the phrase “is satisfactory” mean to you?
Again, answers from 22 respondents were very varied to this question. Thirteen said that “satisfactory”
meant meets minimum quality, two said the term was the same as “sufficient.” One said it had to do
with reading level, one said it was needed to be current, one said it was a subjective term, one felt it had
to do with the condition of books, one said it meant “enough,” and another said it meant “needs work.”
Two gave hard-to-qualify answers:
“Let’s have a Native-American section here, don’t I? Look in the library yourself. Check it out. Look
around. Where’s the Native-American side? You’ll find out. Go to [Harlem 00:38:02]. Go over there. You
know what, these 2 schools are American Indian studies [sites 00:38:10].”
“Sufficient means enough, enough that we can ... whatever it is that we’re doing in our classroom. Do
we have enough books that we can use in order to get across to our students what it is that we’re
teaching, you know, that objective? Do we have not only specific things, but do we have a variety of
things to where the kids can do a lot of research? Do they have a lot of places to go to find materials
from different tribes or different languages and other customs? We just don’t have that.”
Probe 5: In questions b. and d., what does the phrase “the quality of the books and materials” mean
to you?
Eight of 22 respondents said that the question had to do with the content of books, for instance if it was
up-to-date and accurate. Six said it meant if the quality was good, four said it was the condition of
materials, two said it was a subjective question, one said it should be by Native authors, one said it
should be at appropriate reading level for grade, one mentioned variety, and one said it should be
chosen by Native teachers.
Probe 6: In questions c. and d., what does the phrase “specific to American Indian and Alaska Native
cultures” mean to you?
Nineteen out of 21 respondents said it had to do with AI/ANs, and four mentioned that it had to do with
being authored by AI/ANs.

Solutions to Improve Item
Respondent: Add an “I don’t know” option.
Interviewer: Consider specifying in rows A and B that the question refers to school books.

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Respondents generally seemed to grasp what the question was asking.

ETS Recommendation (05/02/2014)
Keep this item.

ETS Post-Cognitive Interview Item Recommendations

Page 154

ETS CogLab Report: NIES|2015
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 155

ETS CogLab Report: NIES|2015
Teacher #11

New/VH040406

What proportion of your American Indian or Alaska Native students will complete the 8th grade?
A. None
B. A few
C. Some
D. Most
E. All
F. I don’t know.

Item 11
Reading: NA
Understanding: It was hard for some teachers who taught multiple grades to specify about their 8th
grade students. Additionally, in responding to Probe 1, many respondents did not mention their AI/AN
students specifically, but this does not necessarily have bearing on understanding. We recommend
highlighting the term “American Indian or Alaska Native” to ensure that is the focus for respondents.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Nineteen of 21 respondents said they were being asked to predict how many of their students would
pass the 8th grade, and five mentioned AI/AN students specifically.
Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what
extent does having a high or low proportion of American Indian or Alaska Native students influence
the answer choice you selected?
There were 22 responses to this question, and given the incredibly wide variety of answers, there was a
total lack of validity for this probe. The term “proportion” in the question led respondents to give actual
numbers, percentages, or numbers of their AI/AN students compared to the rest of the class. And their
answers about whether or not the proportions had an influence were often not answered, though they
usually fell into three categories: it makes no difference, makes it easier to answer because the
proportion is small, or makes it easier to answer because the proportion is high (usually for schools
where AI/AN population is 100%).
Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the
follow-up question]: Please explain why you selected the answer choice. In other words, how did you
figure out the answer?

ETS Post-Cognitive Interview Item Recommendations

Page 156

ETS CogLab Report: NIES|2015
There were 21 responses to this question, and again, the answers varied greatly. Some teachers gave
answers that heralded back to the previous probe having to do with whether or not they had a high or
low proportion (thus making it easy to calculate), while other respondents made blanket statements
about expecting most of their students to pass. It is hard to say if the wide variety and vague answers
given to this probe are a result of the way the probe is written or, more likely, that teachers are sensitive
about the subject of the question.
Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?
When asked if it was difficult or easy to answer, 15 of the 21 respondents said it was easy, though
several of these respondents noted that they thought this was because they were at a smaller school
and had smaller classes, whereas teachers at larger schools may have trouble. Three said it was difficult
because they didn’t know who their AI/AN students were, and another said they did not like to admit to
the numbers of where students were. Lastly there were three difficult-to-categorize answers, one of
which was the following: “To answer totally honestly, it could have been written better and I could have
answered it easily. But since it was, "A proportion," is a math-bound, and there's no proportions listed,
in my view, anyhow, that made it difficult. Not the question ... or, the concept itself was not difficult,
just the way it was worded.”
Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of
other answer choices that would be more clear?
Out of 20 responses to this question, 18 respondents felt that the answers were clear, while two felt
that the question was a little difficult as it was vague. As one stated, “Well, you need to know who
they're comparing with. Who are you comparing with? You don't have a comparison, so how can you
make a proportion if you don't have a comparison. To answer those, if I was to do it, this is how I'd do it,
I'd be thinking okay, whites 60-40. We had 54 last year, we have 60 this year. It's more than some, I'd
say most, because I think of the kids that left, I'd be thinking of the kids that left school. One expelled,
went home at Christmas. Our turnover rate is about 45% here.”

Solutions to Improve Item
Respondent: One respondent said, “I think the proportion part should be taken out, because honestly if
somebody is taking that survey and they haven't like recently looked at the, I don't go around thinking
I've got this many of these students. I don't think people would do that. So, I just think the question
maybe should ask, how many of your Native American students will complete the 8th grade, or do you
think a big amount would? But the proportional part, I don't like that.”
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: Given the wide variety of answers and the push back from some of the respondents, consider
alternate wordings for this item. Also consider bolding “American Indian or Alaska Native students.”

ETS Recommendation (05/02/2014)
Revise this item.
ETS Post-Cognitive Interview Item Recommendations

Page 157

ETS CogLab Report: NIES|2015
Rationale: ETS recommends revising this item to simplify the phrase "What proportion", which caused
confusion for the respondents, by changing it to the phrase "About how many".

ETS Post-Cognitive Interview Item Recommendations

Page 158

ETS CogLab Report: NIES|2015
Teacher #12

New/VH040408

What proportion of your American Indian or Alaska Native students will be prepared for high school?
A. None
B. A few
C. Some
D. Most
E. All
F. I don’t know.

Item 12
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 20 respondents to this question felt that the question was referring to how many of their students
would be prepared for high school. AI/AN students were only referenced by a few respondents, and it
should be kept in mind that some of the schools where interviews took place were 100% AI/AN, though
the majority were not.
Probe 3: What proportion of your 8th grade students are American Indian or Alaska Native? To what
extent does having a high or low proportion of American Indian or Alaska Native students influence
the answer choice you selected?
There were 17 responses to this question. Six teachers said that they had a small proportion of students;
four said they had a large proportion. Seven said that the proportion had no effect, three said that it did.
One teacher said they did not know. Two teachers had difficult-to-categorize responses:
“Most because it has to be a pair with what I answer for 11 if I believe that most of my 8th grader were
going to pass then most of them will be ready for high school as well.”
“Just like the last one. I think that the answer choices can be more specific. What percentages or
different words to use. But I think that I had a good sense of what most meant related to some or all. So
I think it was okay.”

ETS Post-Cognitive Interview Item Recommendations

Page 159

ETS CogLab Report: NIES|2015
Probe 4: [Conditional probe: If the participant selects answer choice A, B, C, D, or E, then ask the
follow-up question]: Please explain why you selected the answer choice. In other words, how did you
figure out the answer?
The 18 respondents for this question gave very varied answers that ranged from counting the number
of students to answers about how students pass no matter what, to pointing out that if the question
actually used the term “8th graders,” that would change how they answered, to saying things about how
they know their students pretty well and that is what they based their answers on.
Probe 5: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?
Twelve of the 19 respondents said that the question was easy to answer, six said it was hard, and one
said “in the middle.”
Probe 6: Do you think that answer choices A through F are clear? If not, can you give me examples of
other answer choices that would be more clear?
Fifteen of the 17 respondents said that the answer choices were clear, and two said they were not—
largely because they were seen as vague.

Solutions to Improve Item
Respondent: One teacher felt that “proportion” was the wrong term and should be changed to
"portion." Another suggested that the word “prepared” should be clarified as having to do with
academic preparedness or social preparedness.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Consider changing the term “proportion” and specifying what is meant by “prepared.”
Additionally, specify if teachers should be reflecting on their 8th grade students.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to simplify the phrase "What proportion", which caused
confusion for the respondents, by changing it to the phrase "About how many".

ETS Post-Cognitive Interview Item Recommendations

Page 160

ETS CogLab Report: NIES|2015
Teacher #13

New/VH040411

To what extent is each of the following a problem in your school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student absenteeism

A

B

C

D

New/VH040413

b. Student tardiness

A

B

C

D

New/VH040414

c. Student health problems

A

B

C

D

New/VH040416

d. Teen pregnancies

A

B

C

D

New/VH040417

e. Drug or alcohol use by students

A

B

C

D

New/VH040421

f.

Student misbehavior in class

A

B

C

D

New/VH040422

g. Physical conflicts among students

A

B

C

D

New/VH040424

h. Bullying

A

B

C

D

New/VH040425

i.

Low student aspirations

A

B

C

D

New/VH040427

j.

Low teacher expectations

A

B

C

D

New/VH040428

A

B

C

D

New/VH040430

k. Low family involvement

Item 13
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Twenty of 21 respondents understood what the item was asking. One respondent provided a difficultto-quantify answer: “Behavior, I guess involvement from students, teachers, and the community to
parents.”
Probe 3: In question a., what does the phrase “student absenteeism” mean to you?
All 21 respondents felt that “student absenteeism” meant students missing school.

ETS Post-Cognitive Interview Item Recommendations

Page 161

ETS CogLab Report: NIES|2015
Probe 4: In question b., what does the phrase “student tardiness” mean to you?
All 21 respondents said that “student tardiness” meant students being late. Most of the respondents
interpreted this as being late to school, others as late to class, and still others as late to both school and
class.
Probe 5: In question c., what does the phrase “student health problems” mean to you?
Answers varied for this question from the 21 teachers providing answers. Nine mentioned being sick,
four said it could be emotional, three brought up asthma, two said lice, two said diabetes, two
mentioned ADHD, two said it would be a physical issue, two said allergies, one mentioned obesity, two
brought up hygiene, and one said broken bones.
Probe 6: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then ask
the follow-up question]: Can you give me examples of how teen pregnancies are a problem in your
school?
There were only seven respondents to this question about teen pregnancies. Two said bullying of the
girls was a problem, two said that missing school was an issue, and four gave difficult-to-qualify
responses:
“After hours there's not enough supervision. We had three or four pregnant. That got pregnant the last
year that were 8th graders. Some of them got pregnant at home, at Christmas and came back. So it all
comes down to talking to your kids, interacting with your kids, and basically talking to them. But then
that doesn't help.”
“I had a pregnant 8th grader last year. I feel that’s a problem, not that I know her. She’s great but I feel
when you’re pregnant and you’re in 8th grade, that’s a problem. I think we have four, not 8th graders.
We had one 8th grader and then I think three or four high school girls that were pregnant last year. We
have 50 students in our high school. We have 60 in our middle school. Proportionality wise, that’s a
good number.”
Probe 7: In question e., what does the phrase “drug or alcohol use by students” mean to you?
All 21 respondents felt that “drug or alcohol use among students” meant whether or not students were
using drugs. Most respondents expanded to explain that they thought that most of this was occurring
outside of school, though some said it could happen in school.
Probe 8: In question f., what does the phrase “student misbehavior in class” mean to you?
There were 20 responses to this question. Seven said that student misbehavior had to do with students
not following rules, five mentioned students disrupting class, four said it was about students acting up,
one said it was students being rude, another felt it was about students making bad choices, another said
insubordination, and yet another said it had to do with students not listening.
Probe 9: In question g., what does the phrase “physical conflicts among students” mean to you?
Sixteen out of 21 respondents used the word “fight” to describe what they felt “physical conflicts among
students” meant to them. Three used the word “physical” and two mentioned pushing.

ETS Post-Cognitive Interview Item Recommendations

Page 162

ETS CogLab Report: NIES|2015
Probe 10: In question h., what does the phrase “bullying” mean to you?
Out of 21 respondents, eight said it was systematic or repeated behavior, seven felt that “bullying” was
verbal in nature, six mentioned cyber bullying, four felt it was physical in nature, and four brought up
students being picked on, harassed, or intimidated.
Probe 11: In question i., what does the phrase “low student aspirations” mean to you?
When asked about “low student aspirations,” the 21 respondents had a variety of answers. Thirteen felt
that it had to do with a lack of initiative or lack of vision for the future, six said it had to with low selfesteem, one said “yes,” and another said that they did not know.
Probe 12: In question j., what does the phrase “low teacher expectations” mean to you?
Of the 20 responses received for this question, six said that they thought “low teacher expectations”
had to do with teachers underestimating students, five indicated that it was about teachers who did not
set that bar high, three mentioned teachers who do not care, two said it was about teachers who had
given up, one said that low expectations led to bad behavior, one said it had to do with the obstacles
teachers face, one mentioned little administrative support, and one simply disagreed that it was a
problem.
Probe 13: In question k., what does the phrase “low family involvement” mean to you?
There were 20 respondents on the issue of “low family involvement.” Seven of these mentioned parents
missing parent-teacher conferences, seven mentioned low investment or engagement with children’s
education, six said it was about parents not attending school activities (such as opportunities to
volunteer, or going to games or concerts), five said it was about not communicating with teachers, one
pointed to the fact that they were at a boarding school, and another mentioned that by the 8th grade,
many parents see their children as adults and so are more hands-off.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: All respondents seem to understand the item and what it is asking, although one point may be
that the question does not specify a grade level or a time frame. The time frame issue did not come up
during participants’ answers, but it became clear that some teachers were answering for the entire
school (as stated in the question) while others were responding only for 8th grade.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students
as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of

ETS Post-Cognitive Interview Item Recommendations

Page 163

ETS CogLab Report: NIES|2015
the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 164

ETS CogLab Report: NIES|2015

Survey Instrument: Administrator Grade 4/Sequence I
School #1

Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.
A. Regular public school
B. Charter public school
C. Bureau of Indian Education contracted or grant school
D. Bureau of Indian Education operated school
E. Other nonpublic school

Item 1
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of 11 respondents, six felt the question was asking about who funded or who had authority over
their school, and five saw the question asking what kind of school it was.
Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain
why.
Eight of the 12 respondents said that they did not think they needed to supply more than one answer,
and four said that they could choose more than one answer due to the way the funding of their school
was set up.
Probe 4: In answer choice E, what does the phrase “other nonpublic school” mean to you? Can you
give me examples of “other nonpublic schools?”
Six of the 11 respondents mentioned parochial schools as an example, five brought up private schools,
three mentioned homeschooling, one said tribal schools, and one said alternative schools.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 165

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 166

ETS CogLab Report: NIES|2015
School #2

New/VH040433

Is your school a boarding school?
A. Yes
B. No

Item 2
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 11 respondents appropriately responded to the item and said that a boarding school was one where
students lived at the school.
Probe 3: What does the phrase “boarding school” mean to you?
Of the 11 teachers responding, 10 indicated that it meant living on campus and two mentioned dorms.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
questions]: Are there any students at your school that do not board? If yes, what proportion of your
students do not board? To what extent does having a high or low proportion of students who do not
board influence the answer choice you selected?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 167

ETS CogLab Report: NIES|2015
School #3

New/VH040437

Is your school located on a reservation/on tribal land?
A. Yes
B. No

Item
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 11 respondents, eight essentially said that the question was asking if their school was on a
reservation, one said “tribal land,” and two gave difficult-to-categorize answers:
“This one is a double-sided question because to the [tribe], we are on tribal land, all of California but,
no, we are not on the reservation. We know. Whether the site that that school is on happens to coincide
with the reservation that's been established by the Federal Government and it doesn't. But for the tribal
land, because it's supposed to all tribal land, they still hold on to view that gap. We are on tribal land
whenever or anywhere in this region.”
“That the property that is around us is tribal property, and that obviously we have a lot of tribal
members living in this area, it doesn’t mean to me that the school is sitting on tribal land, but
everywhere around us is and I don’t know if it is tribal land right here, I’m not sure. It’s all around us, I
know that.”
Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?
When asked what “located on a reservation/tribal land” meant, nine of the 12 respondents said that it
meant being on a federal reservation, three felt that reservations and tribal land were the same thing,
and one said that it meant self-governance.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Post-Cognitive Interview Item Recommendations

Page 168

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 169

ETS CogLab Report: NIES|2015
School #4

New/VH040477

Is your school an American Indian or Alaska Native language immersion school?
A. Yes
B. No

Item
Reading: NA
Understanding: One respondent was confused by what “immersion school” meant.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of the 11 respondents for this question, nine said it meant teaching in another language, one said
they did not know what the phrase meant, and one said, “I think it might be getting at trying to see
whether to track whether languages that might become extinct are still being taught. It might be looking
also at curriculum.”
Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school”
mean to you?
Of the 12 respondents for this question, one restated that they did not know what an immersion school
was, nine said it meant teaching in an AI/AN language all day, one mentioned teaching in another
language, and one talked about how their Native language was gone.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
question]: What proportion of students in your school participates in your schools’ language
immersion program?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.

ETS Post-Cognitive Interview Item Recommendations

Page 170

ETS CogLab Report: NIES|2015
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 171

ETS CogLab Report: NIES|2015
School #5

Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the
following ways? Fill in one oval on each line.
Yes

No

a. Academic club (for example, math club)

A

B

Revised/VH040576

b. Arts club

A

B

Revised/VH040577

c. Cultural dances or activities (for example, drum groups)

A

B

Revised/VH040578

d. Drama club

A

B

Revised/VH040580

e. Making school curriculum decisions

A

B

Revised/VH040581

f.

A

B

Revised/VH040584

g. Parent-teacher conferences

A

B

Revised/VH040588

h. Parent-teacher organizations

A

B

Revised/VH040589

i.

School sports

A

B

Revised/VH040592

j.

Volunteer programs

A

B

Revised/VH040596

A

B

Revised/VH040597

Open houses or back-to-school nights

k. Other (please specify): _____________________________

Item
Reading: NA
Understanding: NA
Changing Answers: One respondent changed their answers once realizing that the answers were
supposed to relate to families. Another respondent changed their answers once they reached the
probes specific to each question section.

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 12 respondents to this question, 10 felt it was asking about parent involvement in the school, one
thought it had to do with the tribe, and one felt it had to do with what the school was involved in.

ETS Post-Cognitive Interview Item Recommendations

Page 172

ETS CogLab Report: NIES|2015
Probe 3: Are your answer choices for questions a. through k. specific to the families of your grade 4
American Indian or Alaska Native students, or do they apply to the families of all American Indian or
Alaska Native students in your school?
Eight of the 11 respondents said they considered the question as relating to all of the school’s families,
two said just the AI/AN families, one said all families K-4, one said all 3-5 grades, and another said all K5. In several of these schools, it was noted that the population of AI/AN students and families was very
high.
Probe 4: In question a., what does the phrase “academic club” mean to you?
Out of 12 respondents, nine brought up math, six said science, five said reading/language, two said
social studies, one said history, one said spelling, one said to enhance subject material, and one said it
was after school.
Probe 5: In question b., what does the phrase “arts club” mean to you?
Ten out of the 11 respondents said that “arts club” meant that it was fine art, three just said their school
did not have such an activity, and one mentioned music.
Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you?
Seven of the 12 responding administrators said that they felt that “cultural dances or activities” were
AI/AN-related. One mentioned drums, three said powwows, and one brought up ORF (did not explain
what this stands for).
Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the
follow-up question]: Can you give me examples of cultural dances or activities? Do you consider
“drum groups” a good example of “cultural dances or activities”?
Four out of 10 responding administrators said that drum groups were a good example, two mentioned
powwows, three mentioned different kinds of crafts, one said pit house activities, one mentioned sweat
lodges, one brought up hand games, another said that for their tribe it was rattles and not drum groups,
one mentioned songs, and one had the following to say:
“To me it means either assemblies or students doing these things. They’re either taking lessons or
showing off with what they’ve learned where they is some theme to what they’re engaged in. [Drum
groups good example?] AC/DC plays drum too so that’s a little weird to have it be the one thing that
somehow the drums are ethnic or somehow. I don't know. I think because this questionnaire is in its
infancy. Someone is just writing down some random stuff. That’s probably an example that needs to be
refined with a couple of more activities. That would be clear if you decided to lift this as extracurricular
or parent involvement. It would depend on what category they’re putting that in.”
Probe 8: In question d., what does the phrase “drama club” mean to you?
Nine of the 11 responses were that plays or theater constituted a drama club, four mentioned the
performing arts in general, poetry was mentioned twice, and music, dance, and art were each
mentioned once.

ETS Post-Cognitive Interview Item Recommendations

Page 173

ETS CogLab Report: NIES|2015
Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?
When asked about making school curriculum decisions, seven of the 11 responding administrators said
that it meant having curriculum input from parents, and five mentioned parent advisory committees.
Probe 10: In question f., what does the phrase “open houses or back-to-school nights” mean to you?
All 11 respondents appropriately characterized back-to-school nights as open houses or opportunities to
involve families in familiarizing themselves with the school.
Probe 11: In question g., what does the phrase “Parent-teacher conferences” mean to you?
Again, all 11 respondents were in agreement that “parent-teacher conferences” meant parents and
teachers meeting to discuss children’s progress.
Probe 12: In question h., what does the phrase “Parent-teacher organizations” mean to you?
All 11 respondents saw parent-teacher organizations as an opportunity for parents and teachers to work
together for the improvement of the school.
Probe 13: In question i., what does the phrase “school sports” mean to you?
All 11 respondents spoke about sports in terms of identifying what sports constituted a school sport
(and to that end basketball, baseball, soccer, volleyball, track, and softball were mentioned). And some
administrators pointed to the fact that their elementary schools did not have school-sponsored sports
but rather that was something managed by the wider community or to be found in middle or high
school.
Probe 14: In question j., what does the phrase “Volunteer programs” mean to you?
Volunteer programs were seen by all 10 respondents to correspond with non-paid work on the part of
community or family members.
Probe 15: What proportion of your students are American Indian or Alaska Native? To what extent
does having a high or low proportion of American Indian or Alaska Native students in your school
influence the answer choice you selected?
The 10 answers to this question on the proportion ranged from 2% to 33% to 100%. Seven said that the
proportion amount had no effect on their answers, one said they did not know, one said it made
answering easier, and one said this was very influential in how they chose to answer.
Probe 16: [Conditional probe: If the participant selects answer choice A for question k, then ask the
follow-up question]: What are the other ways in which families of your American Indian or Alaska
Native students are involved with your school?
Among the nine answers given to this question, the following themes arose: Bible study, festivals,
budget decisions, leadership groups, grad school programs, reading to kids (mentioned twice), fund
raising, and career day.

Solutions to Improve Item
Respondent: NA
Interviewer: NA
ETS Post-Cognitive Interview Item Recommendations

Page 174

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Change: Consider bolding “families of AI/AN students.”

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 175

ETS CogLab Report: NIES|2015
School #6

Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions
and culture are offered at your school each year? Fill in one oval on each line.
None

1

2

3

4 or
more

a. Year-long course

A

B

C

D

E

Trend/VC962954

b. Semester-long course

A

B

C

D

E

Trend/VC962956

c. Workshop or unit

A

B

C

D

E

Trend/VC962957

d. Clubs

A

B

C

D

E

Trend/VC962958

e. Other programs (for example, study
groups before or after regular class
periods)

A

B

C

D

E

Revised/VH018332

Item
Reading: NA
Understanding: Three respondents were confused about some of the terms used in this item such as
“year-long” vs. “semester long” or “other courses,” and another felt that the question was better geared
toward middle or high schools.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
All 11 respondents felt that the question was asking whether or not their schools were teaching about
AI/AN culture.
Probe 3: What does the phrase “study groups before or after regular class periods” mean to you?
All 11 respondents said that “study groups before or after regular class periods” meant before or afterschool programs either as activities or additional help for students.
Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E to question e., then
ask the follow-up question]: Can you give me examples of other programs offered at your school?
Answers from the seven respondents were varied and included youth development, AI/AN-specific
activities such as field trips, summer camp, tutoring, study groups, and, “The counselor could possibly
have say maybe one. The counselor could pull a group of kids and work with them specifically. There
could be one there. Again, there's just ...”
ETS Post-Cognitive Interview Item Recommendations

Page 176

ETS CogLab Report: NIES|2015
Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of
other answer choices that would be more clear?
All 12 administrators responded to this question and five felt that the answer choices (or at least part of
the answer choices) were clear, but nine of the responses indicated that there was some confusion over
the answer choices. One mentioned that the chart was unnecessary, one pointed to the possible need
for a “daily basis” category, one felt that E was not clear, three felt that the answer choices were in
keeping with high school or middle school rather than elementary, another felt that clubs was repetitive
of the other questions. Points made about the structure and content of questions are below:
“Like I said, instead of how many of the following courses or programs are offered each year, I would
maybe word it like is American Indian or Alaska Native traditions and culture taught in your school in any
of the following methods maybe. Not methods, but ... How is the American Indian or Alaska Native
traditions and culture taught in the curriculum, or offered in the curriculum at your school? Again, we
don't have a year-long course, and we don't have a semester-long course, but the teachers do teach
units that would address that.”
“It’s just kind of silly to have CBB but whatever. That’s a problem with this. You got to come up of some
other way to be like answer to 6B is B. You know what I mean? There’s a different way to do that
somehow. I don't know. [Number is not clear?] Right. If the answer to 6B is B then you haven’t clarified.
I would do it like 6.1, 6.2, 6.3, 6.4, the answer is B so that you get a little bit clearer there. A, B, C, D, E,
there’s no 6 associated with it. You should associate this as 6A, 6B, 6C, 6D, 6E not just to the .1.”
“Yeah, this is ambiguous...Yes, because it is a different groups, we have a couple different workshops
and then we have one that goes every week, two times a week, so it's a same answer, two for if you
meet two times or if it is two times a week. See what I'm saying?”

Solutions to Improve Item
Respondent: Some respondents felt that the chart was unnecessary. Course structure is different in the
4th grade and so the structure of answers is inappropriate or inapplicable. Row D is repetitive of Item 5.
Change language from “courses” to “class.” Find a different numbering system for question rows or
answers—don’t use letters, just use numbers. “Courses” does not allow for summer programs or afterschool activities.
Interviewer: Consider including “language” along with traditions and culture in the question.

Initial KAI Recommendation (Keep/Change/Discard)
Change: Given the respondent feedback, particularly to Probe 5, this question should be changed to
better reflect the realities of elementary school. Additionally, may consider highlighting if this question
is only to refer to the 4th grade, as it currently does not.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to a more simplified format by collapsing the answer
choices to “Yes” and “No”. The answer choices for this existing item caused confusion among
ETS Post-Cognitive Interview Item Recommendations

Page 177

ETS CogLab Report: NIES|2015
respondents. Also, response frequencies indicate that the strong majority of the responses fall under
the following two out of the five categories: “None” or “1”. As a result of changing this item to a
“Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs
about American Indian or Alaska Native traditions and culture offered at your school each year?”.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 178

ETS CogLab Report: NIES|2015
School #7

Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your
school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student absenteeism

A

B

C

D

Trend/VC963030

b. Student tardiness

A

B

C

D

Trend/VC963031

c. Student health problems

A

B

C

D

Trend/VC963032

d. Student misbehavior in class

A

B

C

D

Trend/VC963034

e. Physical conflicts among students

A

B

C

D

Trend/VC963035

f.

A

B

C

D

Trend/VC963037

g. Low student aspirations

A

B

C

D

Trend/VC963038

h. Low teacher expectations

A

B

C

D

Trend/VC963039

i.

Low family involvement

A

B

C

D

Trend/VC963040

j.

Inadequate transportation for
students

A

B

C

D

New/VH040475

Bullying

Item
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Out of 12 respondents, 10 felt that the question was referring to problems at the school. One person
specifically mentioned problems for AI/AN students, one person felt it had to do with problems for
students, staff, and family, and another mentioned that they were thinking of all students.
Probe 3: In question a., what does the phrase “student absenteeism” mean to you?
All 12 respondents felt that this question was asking about students who were missing from school or
class or both.
ETS Post-Cognitive Interview Item Recommendations

Page 179

ETS CogLab Report: NIES|2015
Probe 4: In question b., what does the phrase “student tardiness” mean to you?
All 12 respondents thought this question was about asking about students who were late to school or
class or both.
Probe 5: In question c., what does the phrase “student health problems” mean to you?
Of the 12 respondents providing an answer to this question, six mentioned children being sick or ill, four
brought up diabetes, three said asthma, two mentioned mental or emotional problems, one said dental,
one said sleep deprivation, one mentioned physical issues, one brought up allergies, one noted obesity,
and one mentioned lice.
Probe 6: In question d., what does the phrase “student misbehavior in class” mean to you?
There were 12 respondents for this question and four mentioned disruptive behavior as student
misbehavior in class, three said it was students not listening, two said that it had to do with poor
classroom management, two noted rule breaking and two brought up interrupting and disrespect.
Probe 7: In question e., what does the phrase “physical conflicts among students” mean to you?
When asked about physical conflicts, out of 12 responses, four mentioned pushing, three said shoving,
three used the word “fight,” three said it had to do with students putting their hands on one another,
two said hitting, one said bullying, and one said it had to do with classroom management.
Probe 8: In question f., what does the phrase “bullying” mean to you?
When discussing bullying, seven of the 12 respondents said that the behavior had to be consistent and
repetitive, five said it was generally verbal, three mentioned physical aspects, one brought up cyber
bullying, one said that it had an effect on children’s self-esteem, and one said it was related to poor
classroom management/poorly organized adults. Interestingly it seemed that many of the
administrators did not think that many of the complaints about bullying at their schools were actual
bullying, but rather just kids being kids.
Probe 9: In question g., what does the phrase “low student aspirations” mean to you?
Out of the 12 respondents, six intimated that low student aspirations were due to the low bar that
teachers set, four mentioned motivation on the part of students, one said it had to do with students not
trying hard, one brought up disabilities, and one said that some children did not like coming to school
because they were bullied.
Probe 10: In question h., what does the phrase “low teacher expectations” mean to you?
With regard to low teacher expectations, five of the 12 respondents said that it had to do with teachers
setting a low bar, two said it was because teachers were overwhelmed or had given up, one noted poor
organization among teachers, one said it was teachers who did not bother to push their students, and
three gave hard-to-qualify answers:
“A lot of them coming up with that problem. Is there a problem? But I don’t see. Teachers aren’t coming
to us and saying that there’s a problem.”
“That says to me how much the administration expects of them as far as putting effort into educating
our children.”

ETS Post-Cognitive Interview Item Recommendations

Page 180

ETS CogLab Report: NIES|2015
“I would say ... Overall, here the teachers actually have really high expectations. I still answered that
small extent, because I think sometimes ... One of the areas we struggle with as a school is we have a
high level of proficiency. About 80% of our kids are proficient, but there's still that other 20%. We've got
to focus on that. Even the students that are already proficient, we don't show a lot of growth. It's how
are we challenging the kids that are already proficient, because proficient isn't really good enough.”
Probe 11: In question i., what does the phrase “low family involvement” mean to you?
Of the 12 respondents to this question, nine felt that low family involvement was related to families not
physically coming to events or activities, and four mentioned parents being non-responsive to teacher
communication.
Probe 12: In question j., what does the phrase “inadequate transportation” mean to you?
There were 12 responses to this question and of those, eight felt that “inadequate transportation”
meant not having a way to get to and from somewhere, one interpreted it as students having dangerous
transportation options, and three gave hard-to-categorize answers:
“Somebody is trying to figure out. That’s a kind of different category. That’s like a jacked up district.”
“That means they're asking if the ability to transport our students hurts their education.”
“There’s none. I don’t see any inadequate transportation.”

Solutions to Improve Item
Respondent: Consider rephrasing "considering entire student body" or eliminating it completely; also
consider increasing the scale to five or adding proportion/percentage. One respondent did not care for
the term "extent." Another suggested answer categories: rarely, sometimes, usually, always.
Interviewer: Is this question meant to apply to the entire school (which may be K-8 or higher) or just to
the 4th grade?

Initial KAI Recommendation (Keep/Change/Discard)
Change: This item seems generally understandable. Consider making it specific to 4th grade. Additionally,
should administrators be reflecting on all students or just AI/AN students?

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN students
as some schools will likely consist of a mixture of both AI/AN and non-AI/AN students. The purpose of
the NIES survey is to capture information that is specific to AI/AN students, teachers, and schools;
therefore the items should reference "AI/AN" when necessary in order to capture the most useful data.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 181

ETS CogLab Report: NIES|2015
School #8

Trend/VC963041

Which of the following types of student and family services are offered at your school on a regular
basis? Fill in one oval on each line.
Offered

Not
offered

a. Health services for students such as a school nurse

A

B

Trend/VC963042

b. Health services for families such as a community clinic

A

B

Trend/VC963043

c. Social or counseling services for students

A

B

Trend/VC963045

d. Social or counseling services for families

A

B

Trend/VC963047

e. Adult education programs for all ages

A

B

Trend/VC963048

f.

A

B

New/VH017944

A

B

New/VH017945

Tutoring services for students

g. Career counseling services for students

Item
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 11 responses given, nine concentrated on providing or offering services to families, and two
focused on providing services to students.
Probe 3: What does the phrase “a regular basis” mean to you?
Answers from the 12 respondents were varied in how they determined what “a regular basis” meant.
Five said it meant regularly occurring, other answers were that it depended, would be most of the week,
weekly, daily, three times a week, not once a year, all the time, or two times a month.
Probe 4: In question f., what does the phrase “tutoring service for students” mean to you?
There were 12 responses to this question. Ten were that “tutoring services” meant extra help for
students, one person said it was to help students elevate their scores, and another simply said “yes.”

ETS Post-Cognitive Interview Item Recommendations

Page 182

ETS CogLab Report: NIES|2015
Probe 5: [Conditional probe: If the participant selects answer choice A to question f., then ask the
follow-up question]: Can you give me examples of the tutoring services offered at your school?
Eleven out of 12 respondents said that their school offers tutoring before, after, and during classes. Only
one respondent said that their school did not have such offerings.
Probe 6: In question g., what does the phrase “career counseling services for students” mean to you?
With regard to career counseling services, it is difficult to determine the group that the 12 respondents
were thinking about when giving their answers. At least four mentioned that they thought the question
was more appropriate for secondary-level students, two mentioned job fairs generally, and seven said
that their schools offered career counseling—something it is hard to imagine that 4th graders would be
involved with given their age.
Probe 7: [Conditional probe: If the participant selects answer choice A to question g., then ask the
follow-up question]: Can you give me examples of the career counseling services offered at your
school?
There were only seven responses to this question. Three mentioned job fairs, three brought up outside
speakers, and two mentioned teachers as people providing career information to students. Again, it is
difficult to say if the focus here was purely on 4th grade students.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep/Change: Consider bolding if the question is focused on the whole school or only 4th graders.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 183

ETS CogLab Report: NIES|2015
School #9

New/VH040496

To what extent do your school’s 4th grade curricula emphasize preparing American Indian or
Alaska Native students for life in their community, tribe, group, pueblo, or village?
A. Not at all
B. Small extent
C. Moderate extent
D. Large extent

Item
Reading: NA
Understanding: One respondent found the item difficult to answer but could not clearly convey why,
and also noted that they were from Guatemala originally and so had a different definition for what a
pueblo was.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 12 responses given to this question, seven mentioned preparedness, and six thought that the
question meant incorporating the culture.
Probe 3: What does the phrase “preparing American Indian or Alaska Native students for life in their
community, tribe, group, pueblo, or village” mean to you?
Out of 12 responses, five mentioned the tribe with regard to preparing AI/AN students, one said they
did not know, and three mentioned the community. Several respondents gave hard-to-qualify answers:
“That’s really a touchy question there, because I am not Native American and some of this is allowing
them to have excuses why they are late for school and my opinion is that I think we ought to treat
everybody the same. And they want to be treated different and they think the teachers need to learn
the culture of the Native American and accept that they’re late all the time or they’re not at school and
it’s hard for me to deal with this one. (Chuckle.)”
“It it's specific to American Indian or Alaska Native in living like ... the dynamics on a reservation or
something like that, I don't know the answer to that.”
“That means that how much emphasis is being put in and to get the kids ready to utilize their way of
living and resources to be better citizens in the future. However, when we say, “To what extent do your
school curriculum emphasize,” if I just put the 4th grade as a block of learning time, then I’m talking
about the curriculum that we, all the schools have, the buy curriculum. But if our school has learning
every day, that our school is an active school with activities related to their life, I think we are doing a
better, a very good job in that.”
ETS Post-Cognitive Interview Item Recommendations

Page 184

ETS CogLab Report: NIES|2015
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D, then ask the follow-up
question]: Can you give me examples of how your school’s 4th grade curricula emphasize preparing
American Indian or Alaska Native students for life in their community, tribe, group, pueblo or village?
There were eight responses to this question. Three administrators stated that they did not think that
their 4th grade curricula did emphasize preparing AI/AN students for life in the different communities
listed. Three said that their schools offered culturally relevant programming, and two brought up
encouraging common core educational components.
Probe 5: What does the word “pueblo” mean to you?
Of the 11 responses given, eight felt that “pueblos” were related to American Indian people in the
Southwest, one person (from Guatemala) said that it meant city, one said “house drum,” and another
said that it did not really have any meaning to them.
Probe 6: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?
When asked if the item was easy or difficult to answer, four of the 12 respondents said it was easy, and
eight felt that it was difficult. Of those indicating that it was difficult, the reasons had to do with the fact
that several respondents did not see “community” and “tribe” (among the other phrases) as being the
same thing. For instance, many felt that the community referred to being among the wider community,
whereas being with the tribe was a separate kind of community. Additionally, many respondents
mentioned that the item’s emphasis on the 4th grade curricula was problematic because they often do
not emphasize cultural learning that may happen at the school level. This is important as many curricula
are approved at the district or state level.

Solutions to Improve Item
Respondent: Expand examples on what is meant by “preparing youth for their community.”
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: It seems that there was some confusion over the multiple terms to describe what were seen as
different kinds of communities. The language around “curriculum” caused many respondents confusion
over how they should answer the question as it is currently worded. If the wording could be changed to
be more inclusive around the curriculum piece then it should be kept. Consider removing the word
“community” unless rephrasing it to “tribal community.”

ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS recommends dropping this item as respondents indicated that life in their community is
not necessarily the same as life in their tribes. Respondents indicated that the school curricula may
succeed in preparing students for life in their community, for instance, but not necessarily life in their
tribe. In addition, not all students may have a “tribal life” and may live in a community without other

ETS Post-Cognitive Interview Item Recommendations

Page 185

ETS CogLab Report: NIES|2015
AI/AN families. Furthermore, sub-items "b" and "c" may be more applicable to higher grades, as
expressed by some of the respondents, therefore they may not be appropriate for grade 4.

ETS Post-Cognitive Interview Item Recommendations

Page 186

ETS CogLab Report: NIES|2015

Survey Instrument: Administrator Grade 8/Sequence I
School #1

Revised/VH040432

Which of the following describes your school best? Please fill in one oval only.
A. Regular public school
B. Charter public school
C. Bureau of Indian Education contracted or grant school
D. Bureau of Indian Education operated school
E. Other nonpublic school

Item
Reading: NA
Understanding: NA
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 10 respondents, six administrators felt the question item asked them to identify school type,
three felt it asked about the school’s funding source, and one felt the question was trying to determine
whether or not the school was effective.
Probe 3: Do you think that you need to select more than one answer choice? If yes, please explain
why.
All 11 administrators who responded reported needing only one answer choice.
Probe 4: In question e., what does the phrase “other nonpublic school” mean to you? Can you give me
examples of “other nonpublic schools?”
Of the 11 responses to this probe, six pointed to religious or parochial schools, five focused on private
schools or institutions, and two highlighted charter schools as types of “other nonpublic schools.” One
administrator cited alliance or alternative schools.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 187

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 188

ETS CogLab Report: NIES|2015
School #2

New/VH040433

Is your school a boarding school?
A. Yes
B. No

Item
Reading: NA
Understanding: None of the respondents demonstrated any difficulty understanding the question;
however, one administrator pointed out that some lay/regular survey participants may not be familiar
with the term “boarding school.”
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 11 respondents, 10 administrators felt that the question asked whether the school boarded
students/students lived at the school, while one felt that the question was looking at the type of school
operated, specifically.
Probe 3: What does the phrase “boarding school” mean to you?
Seven administrators (out of 12 respondents) defined “boarding school” in terms of providing students
with housing or students living on campus or in dorms. Two referred to students living on campus or at
the school “24/7,” while one felt that students in boarding schools lived independently while on
campus. Three respondents made it a point to reference the historical context of boarding schools in
AI/AN communities.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
questions]: Are there any students at your school that do not board? If yes, what proportion of your
students do not board? To what extent does having a high or low proportion of students who do not
board influence the answer choice you selected?
Two administrators responded to this probe. The first reported a 60% boarding population, while the
other noted that 33 of the 460 enrolled students boarded. Neither respondent felt that this proportion
influenced their answer choice selection.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 189

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 190

ETS CogLab Report: NIES|2015
School #3

New/VH040437

Is your school located on a reservation/on tribal land?
A. Yes
B. No

Item
Reading: NA
Understanding: One respondent admitted to experiencing some difficulty determining the meaning of
the phrase “on a reservation/on tribal land.”
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the nine respondents, six felt that the question was asking if the school was located on land owned
by the tribe or within the boundaries of the reservation, while one felt that it was asking about which
group or entity had jurisdiction over the provision of educational services. Two administrators
referenced special or complex circumstances in their responses:
“Yes, it was a reservation land originally here in Oklahoma, but because of the Treaty it still is identified
with us, but it’s not reservation land. It’s not set aside specifically anymore, just for the tribe its set aside
for individuals. But you have to know that kind of in your head, because some people will say, ‘Yeah, I’m
on a reservation land.’ But in Oklahoma we are not.
“Up here, this could be a little confusing but I'm interpreting it whether it's on reservation or tribal land,
which the answer would be no. The uniqueness of Wellpinit is we're on a reservation but we're on
publics land within the reservation.”
Probe 3: What does the phrase “located on a reservation/on tribal land” mean to you?
Nine administrators (out of 12 respondents) felt that “reservation or tribal lands” referred to
reservations, Native-owned land, or federal land set aside for tribes. One felt that it referred to land
where taxes were not paid on the land, while one other administrator felt that the term was nested
within a historical definition. One admitted to being unsure of the definition of the phrase:
“That’s where I’m struggling. I know what a reservation is but I thought it was the same thing. Me being
a non-Native maybe that’s why I don’t know.”

Solutions to Improve Item
Respondent: NA
Interviewer: NA
ETS Post-Cognitive Interview Item Recommendations

Page 191

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Change: While none of the respondents demonstrated difficulty responding to the question, a few
noted situations where the presence of trust lands could make it more challenging to answer. Consider
adding “trust land” to the list of tribal land options within the text of the question.
Another respondent noted that nontribal members or individuals unfamiliar with the concept or
definition of tribal lands/reservations may experience confusion or difficulty when answering the
question.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 192

ETS CogLab Report: NIES|2015
School #4
New/VH040477
Is your school an American Indian or Alaska Native language immersion school?
A. Yes
B. No

Item
Reading: NA
Understanding: One respondent thought the question was asking about American Indian and Alaska
Native languages separately. A second administrator demonstrated some difficulty interpreting the
phrase “immersion program.”
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 11 respondents, eight felt that the question was trying to ask if an AI/AN language was used all
day or included in the general school curriculum. Two administrators felt that the question only asked if
the school ran an immersion program, while one felt that the question wanted to determine if staff and
students spoke a Native language instead of simply teaching it.
Probe 3: What does the phrase “an American Indian or Alaska Native language immersion school”
mean to you?
Seven administrators (out of 12 respondents) felt that an immersion program incorporated an AI/AN
language throughout the day and throughout course curriculum. One felt that the term meant that the
school had certified teachers to provide appropriate language immersion instruction. Five of the
administrators discussed current or past plans to implement an immersion program in their responses.
Probe 4: [Conditional probe: If the participant selects answer choice A, then ask the follow-up
question]: What proportion of students in your school participates in your schools’ language
immersion program?
NA

Solutions to Improve Item
Respondent: NA
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 193

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 194

ETS CogLab Report: NIES|2015
School #5

Revised/VH040574

Are families of your American Indian or Alaska Native students involved with your school in the
following ways? Fill in one oval on each line.
Yes

No

a. Academic club (for example, math club)

A

B

Revised/VH040576

b. Arts club

A

B

Revised/VH040577

c. Cultural dances or activities (for example, drum groups)

A

B

Revised/VH040578

d. Drama club

A

B

Revised/VH040580

e. Making school curriculum decisions

A

B

Revised/VH040581

f.

A

B

Revised/VH040584

g. Parent-teacher conferences

A

B

Revised/VH040588

h. Parent-teacher organizations

A

B

Revised/VH040589

i.

School sports

A

B

Revised/VH040592

j.

Volunteer programs

A

B

Revised/VH040596

A

B

Revised/VH040597

Open houses or back-to-school nights

k. Other (please specify): _____________________________

Item
Reading: NA
Understanding: One respondent appeared to equate references to clubs and activities with actual
school course activities.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the 10 respondents, eight felt that the question was asking about types and levels of family
involvement, while two felt that the question asked about the opportunities for involvement offered to
families.
Probe 3: Are your answer choices for questions a. through k. specific to the families of your grade 8
American Indian or Alaska Native students, or do they apply to the families of all American Indian or
Alaska Native students in your school?
Six administrators (out of nine respondents) referenced all families when answering. Two referenced
only the families of 8th grade students, and one limited their answer to AI/AN families.
ETS Post-Cognitive Interview Item Recommendations

Page 195

ETS CogLab Report: NIES|2015
Probe 4: In question a., what does the phrase “academic club” mean to you?
When asked about academic clubs, eight out of 10 respondents felt that the term referred to academic
or course-specific groups; one felt that it referred to a group for gifted or talented students; and one felt
that it referred to a type of peer-based mentoring.
Probe 5: In question b., what does the phrase “arts club” mean to you?
Six of the 10 administrators who responded felt that the item referred to any visual or performance art.
Three administrators, rather than defining the term, described past or present examples of arts clubs at
the school, while one simply noted that the school did not offer such an activity.
Probe 6: In question c., what does the phrase “cultural dances or activities” mean to you?
Ten administrators responded to the probe. Of these, two made general references to powwows and
drum groups, while four cited specific examples of cultural activities offered at or by the school. Three
administrators discussed the traditional and cultural value of such activities in their responses. One
respondent defined the probe item as visual or performance art.
Probe 7: [Conditional probe: If the participant selects answer choice A for question c., then ask the
follow-up question]: Can you give me examples of cultural dances or activities? Do you consider
“drum groups” a good example of “cultural dances or activities”?
Of the nine respondents, five described powwows and drum groups offered by or at the school. Two
respondents described specific types of dance taught to students, while one made a more general
reference to teaching students traditional types of dance. One administrator failed to provide an
example, simply noting that drum groups are a good example of traditional activity. Three other
administrators similarly noted that drum groups are, in fact, a good example.
Probe 8: In question d., what does the phrase “drama club” mean to you?
All but one administrator (out of 10 respondents) described “drama club” in terms of acting, plays,
stage, or theater. The 10th administrator’s response described the school’s plans to develop a student
drama group.
Probe 9: In question e., what does the phrase “making school curriculum decisions” mean to you?
Of the 11 respondents, six described the probe item as parental involvement or input into curriculum or
course decisions. Three administrators felt it referred to a specific council or committee that included
parents. One administrator’s response defined it simply as parental feedback on school curriculum,
while one other administrator noted that only the state drove curriculum-based decisions for the school.
Probe 10: In question f., what does the phrase “open houses or back-to-school nights” mean to you?
Ten of the 11 respondents to this probe defined this probe item in terms of a specific school event
where parents met or interacted with teachers and school staff. One administrator referred to it as a
type of public relations activity.

ETS Post-Cognitive Interview Item Recommendations

Page 196

ETS CogLab Report: NIES|2015
Probe 11: In question g., what does the phrase “Parent-teacher conferences” mean to you?
All 10 responses described “parent-teacher conferences” as a meeting between parents and teachers
(both formal and informal in nature).
Probe 12: In question h., what does the phrase “Parent-teacher organizations” mean to you?
Three administrators (out of 10 respondents) specifically referred to PTAs or PTOs in their responses,
while five made more general references to some type of parent-teacher committee. One respondent
described a past example of this type of organization, while two others noted that no such organization
currently existed.
Probe 13: In question i., what does the phrase “school sports” mean to you?
Of the 10 responses describing “school sports,” six referred to a specific sport (e.g., football, basketball,
and track) and four referred to sports as a collective, general category.
Probe 14: In question j., what does the phrase “Volunteer programs” mean to you?
Six administrators (out of 10 respondents) described “volunteer programs” as occasions when parents
or family members provided assistance to the school. Two administrators described specific programs or
examples in their responses, while two others noted that no such programs currently exist.
Probe 15: What proportion of your students are American Indian or Alaska Native? To what extent
does having a high or low proportion of American Indian or Alaska Native students in your school
influence the answer choice you selected?
Of the nine total responses given, one administrator reported a proportion of 2%, one a proportion of
20%, and one a proportion of 33%. Five administrators reported a proportion of 95% or greater. Two
respondents noted that these proportions had no influence over their answer selection. One respondent
found this probe very confusing and needed to reread it before responding, while three other
respondents found the probe too confusing to answer.
Probe 16: [Conditional probe: If the participant selects answer choice A for question k, then ask the
follow-up question]: What are the other ways in which families of your American Indian or Alaska
Native students are involved with your school?
Of the seven responses provided, two described alumni activities, one described families serving as
chaperones or assisting with supervision during field trips, and four described more miscellaneous
activities (e.g., cheerleading, band, and archery contests).

Solutions to Improve Item
Respondent: One respondent noted that not all question row items were available or offered at the
school.
Interviewer: NA

ETS Post-Cognitive Interview Item Recommendations

Page 197

ETS CogLab Report: NIES|2015
Initial KAI Recommendation (Keep/Change/Discard)
Change: Several respondents noted that some of the question rows listed did not apply as they were not
available at the school. Consider adjusting the answer choices to include an option for “N/A” or “Do not
offer/Not available.” Also specify and emphasize if the question focuses on all AI/AN families or just the
families of 8th grade students.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 198

ETS CogLab Report: NIES|2015
School #6

Trend/VC962952

How many of the following courses and programs about American Indian or Alaska Native traditions and
culture are offered at your school each year? Fill in one oval on each line.
None

1

2

3

4 or
more

a. Year-long course

A

B

C

D

E

Trend/VC962954

b. Semester-long course

A

B

C

D

E

Trend/VC962956

c. Workshop or unit

A

B

C

D

E

Trend/VC962957

d. Clubs

A

B

C

D

E

Trend/VC962958

e. Other programs (for example, study
groups before or after regular class
periods)

A

B

C

D

E

Revised/VH018332

Item
Reading: NA
Understanding: Two respondents expressed some confusion when reading and answering the question
item, one because the answer choices are not specifically offered at the respondent’s school, while the
other could not understand the question’s meaning.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
One respondent (out of 12 respondents) was not sure what the question was trying to ask. Seven
administrators felt the question was asking about AI/AN culture-specific classes and courses. Two
administrators felt that the question explored the types of cultural programming offered at the school,
outside of a traditional curriculum, and one thought the question wanted to know how culture was
taught at the school. One respondent thought the question was asking about their own level of
knowledge of Native culture.
Probe 3: What does the phrase “study groups before or after regular class periods” mean to you?
One of the 11 respondents was unsure of the definition of the probe item. Four defined the phrase in
terms of activities taking place outside of the classroom or regular school day. Three respondents
described it as a formal group or tutoring program. Three administrators described the phrase as topicrelated discussion groups (one), independent study (one), or follow-up study or instruction (one).

ETS Post-Cognitive Interview Item Recommendations

Page 199

ETS CogLab Report: NIES|2015
Probe 4: [Conditional probe: If the participant selects answer choice B, C, D, or E for question e., then
ask the follow-up question]: Can you give me examples of other programs offered at your school?
The four responses to the probe referred to the Century 21 program (one), the MEFE (one), a program
involving NASA (one), and a professional performance group (one).
Probe 5: Do you think that answer choices A through E are clear? If not, can you give me examples of
other answer choices that would be more clear?
All 11 administrators who responded noted that the answer choices seemed clear.

Solutions to Improve Item
Respondent: One administrator suggested rephrasing some of the question rows to read more clearly:
"That just meant a year-long course ... or and programs so it's really, is there a specific course, is there a
specific program that they're talking about?"
Another respondent suggested separating out questions, as the current format can be confusing and
overly complicated to answer.
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: Consider providing examples of or definitions for the question row items. This should help
alleviate some of the respondent confusion. Also specify if these courses are school-wide or 8th gradespecific.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to a more simplified format by collapsing the answer
choices to “Yes” and “No”. The answer choices for this existing item caused confusion among
respondents. Also, response frequencies indicate that the strong majority of the responses fall under
the following two out of the five categories: “None” or “1”. As a result of changing this item to a
“Yes/No” format, ETS recommends revising the item stem to “Are the following courses and programs
about American Indian or Alaska Native traditions and culture offered at your school each year?”.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 200

ETS CogLab Report: NIES|2015
School #7

Trend/VC963029

Considering all of the students in your school, to what extent is each of the following a problem in your
school? Fill in one oval on each line.
Not at
all

Small
extent

Moderate
extent

Large
extent

a. Student absenteeism

A

B

C

D

Trend/VC963030

b. Student tardiness

A

B

C

D

Trend/VC963031

c. Student health problems

A

B

C

D

Trend/VC963032

d. Teen pregnancies

A

B

C

D

New/VH040470

e. Drug or alcohol use by students

A

B

C

D

Trend/VC963033

f.

Student misbehavior in class

A

B

C

D

Trend/VC963034

g. Physical conflicts among students

A

B

C

D

Trend/VC963035

h. Bullying

A

B

C

D

Trend/VC963037

i.

Low student aspirations

A

B

C

D

Trend/VC963038

j.

Low teacher expectations

A

B

C

D

Trend/VC963039

A

B

C

D

Trend/VC963040

A

B

C

D

New/VH040475

k. Low family involvement
l.

Inadequate transportation for
students

Item
Reading: NA
Understanding: One respondent was unsure of the time frame covered by the question. This same
individual later interpreted the question as specific to AI/ANs rather than to the entire school.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
One administrator (out of eight respondents) answered in reference to specific answer rows rather than
the entire question. Five administrators felt the question was asking about the types of problems or
issues experienced at the school. Two administrators thought the question focused on issues that
specifically inhibited students’ ability to learn or succeed in school. One respondent felt that the
question was trying to explore the school’s management abilities.

ETS Post-Cognitive Interview Item Recommendations

Page 201

ETS CogLab Report: NIES|2015
Probe 3: In question a., what does the phrase “student absenteeism” mean to you?
Of the 12 respondents, 11 defined the probe phrase in terms of the student being away from or out of
school without an excuse. One respondent described it as a lack of parental support.
Probe 4: In question b., what does the phrase “student tardiness” mean to you?
Ten administrators (out of 11 respondents) described “tardiness” as being late for school or class. One
administrator described a circumstance where the student’s physical distance from the school could
impact his or her ability to come to school on time.
Probe 5: In question c., what does the phrase “student health problems” mean to you?
All 11 respondents referred to the phrase in terms of a health or medical condition. Among these
responses, two specifically referred to diabetes, one referred to lice, one to mental health issues, one to
the flu, and one to any condition requiring medication.
Probe 6: In question e., what does the phrase “drug or alcohol use by students” mean to you?
Twelve administrators responded to this probe. Of these, seven defined the phrase as the use or abuse
of drugs, alcohol, or other illegal substances. Four administrators described the prevalence of the issue
in their school setting; one described a specific instance of addressing the issue with students. One
administrator’s response described the school’s response to the issue in general.
Probe 7: In question f., what does the phrase “student misbehavior in class” mean to you?
Of the 11 respondents, 10 described the phrase as some kind of problem behavior or misconduct. One
administrator defined it more specifically as behavior resulting in some kind of formal discipline.
Probe 8: In question g., what does the phrase “physical conflicts among students” mean to you?
Ten of the 12 respondents described the phrase as physical fighting or harm done to other students,
with one other respondent detailing a specific instance of physical conflict that took place in the school.
One administrator’s response was that the item was not an issue.
Probe 9: In question h., what does the phrase “bullying” mean to you?
The 12 respondents to this probe had varying definitions of bullying. Five described it as harassing
another student or causing another student stress or harm. Four described specific examples of bullying,
with one other noting that bullying is something that happens more than once (i.e., the behavior is
repetitive). Three defined it in terms of one student overpowering another, while two respondents
noted that the term itself is difficult to define:
“I hear this all the time, large extent. I think it may be due to a definition problem. They just don't
understand the definition of it. So any type of verbal words looking at.”
“This one is so hard. Everyone got their own definition for it. I would say people lump a lot of things into
bullying that may not be…”

ETS Post-Cognitive Interview Item Recommendations

Page 202

ETS CogLab Report: NIES|2015
Probe 10: In question i., what does the phrase “low student aspirations” mean to you?
Eleven of the 12 respondents defined the probe item as students lacking goals, direction, or motivation.
One respondent’s answer described a specific example of low student aspirations.
Probe 11: In question j., what does the phrase “low teacher expectations” mean to you?
Seven administrator responses (out of 12 respondents) consisted of general descriptions of teacher
expectations. Four administrator responses described the high levels of teacher expectations present at
their schools. One response was difficult to code:
“Well, these kids just come from the reservation and they haven’t been in schools, so they’ll probably
not going to help my scores in any ... I mean instead of ... I’m a firm believer that kids will do what you
expect them to do and I always think of the same few teachers that all year, they always came to mind
and just ... I don’t know, I’m just kind of almost borderline, just don’t care.”
Probe 12: In question k., what does the phrase “low family involvement” mean to you?
Of the 12 responses to the probe, nine discussed low family involvement in general terms, and two
described how the school currently struggled with low involvement. One administrator had difficulty
responding to the probe:
“For some reason when they get out of elementary we lose them at the middle school level usually,
because in 8th grade they are involved in the sports and the other activities. So I don’t get a lot of family
involvement, but again you need to say, ‘Is that involvement in the school or is it involvement in our
program?’ So that might be... You might want to say something about you know are they involved in the
school or... Because they may be involved in our program, but not in the school. You see what I mean?
They might come to my meetings, but they won’t go to the parent-teacher meetings at the school. So I
don’t know. That question was just a little... You might want to be a little more specific on that one.”
Probe 13: In question l., what does the phrase “inadequate transportation” mean to you?
Six administrators (out of 10 respondents) described the probe items as a lack of transportation to or
from the school. Two administrators noted that inadequate transportation was not an issue for their
school, while two others described it as an issue involving students’ parents.
Probe 14: [Conditional probe: If the participant selects answer choice B, C, or D to question d., then
ask the follow-up question]: Can you give me examples of how teen pregnancies are a problem in
your school?
Of the five administrators who responded, two noted that it is not a current problem but went on to
describe past experiences with teen pregnancy. Three other administrators described how the issue
impacted students’ ability to succeed in school.

Solutions to Improve Item
Respondent: One respondent requested more clarification on the time frame of the question (e.g., this
school year? multiple school years?)

ETS Post-Cognitive Interview Item Recommendations

Page 203

ETS CogLab Report: NIES|2015
Interviewer: Consider rephrasing the question about teen pregnancies to focus on how they impact
schools rather than how they are a "problem.”

Initial KAI Recommendation (Keep/Change/Discard)
Keep: Question seems generally understandable; however, potential changes could include specifying
the time frame to which respondents should refer when answering, and rephrasing “teen pregnancy” to
seem less stigmatic. Additionally, consider whether or not answers are wanted for the whole school,
which could include up to 12th grade, or just for 8th grade students.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS recommends revising this item to clarify that it is specifically asking about AI/AN
students.
This recommendation is the same at grades 4 and 8.

ETS Post-Cognitive Interview Item Recommendations

Page 204

ETS CogLab Report: NIES|2015
School #8

Trend/VC963041

Which types of student and family services are offered at your school on a regular basis? Fill in one oval on
each line.
Offered

Not
Offered

Health services for students such as a school nurse

A

B

Trend/VC963042

Health services for families such as a community clinic

A

B

Trend/VC963043

Social or counseling services for students

A

B

Trend/VC963045

Social or counseling services for families

A

B

Trend/VC963047

Adult education programs for all ages

A

B

Trend/VC963048

Tutoring services for students

A

B

New/VH017944

Career counseling services for students

A

B

New/VH017945

Item
Reading: NA
Understanding: One respondent found the language in question row B confusing.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Eight of the 10 respondents felt that the question tried to identify the types of services or programs
available to students and families. Two administrators limited their responses to row A specifically.
Probe 3: What does the phrase “a regular basis” mean to you?
Of the 10 respondents, nine administrators referred to a recurring schedule or regular time. One
administrator simply referred to things taking place “on average.”
Probe 4: In question f., what does the phrase “tutoring service for students” mean to you?
Of the 11 administrators who responded, five defined tutoring services as the provision of academic
support, five specifically described such services as taking place outside of regular school hours and
three mentioned specific examples of types of tutoring services currently provided.
Probe 5: [Conditional probe: If the participant selects answer choice A for question f., then ask the
follow-up question]: Can you give me examples of the tutoring services offered at your school?
ETS Post-Cognitive Interview Item Recommendations

Page 205

ETS CogLab Report: NIES|2015
Seven of the 11 respondents referred to generic tutoring services that provide students with additional
academic support, and three specifically mentioned the Century 21/21st Century tutoring program.
Probe 6: In question g., what does the phrase “career counseling services for students” mean to you?
Six administrators (out of 11 respondents) referred to guidance or decision-making assistance about
students’ futures, and one spoke of how it informs students of options outside the reservation. Four
administrators described specific programs or services provided by the school.
Probe 7: [Conditional probe: If the participant selects answer choice A for question g., then ask the
follow-up question]: Can you give me examples of the career counseling services offered at your
school?
Of the five responses provided, career counseling was described as career days (four), staff counselors
(four), college visits or partnerships (three), and bringing in outside speakers or programs (four).

Solutions to Improve Item
Respondent: One respondent suggested adjusting the question language to accommodate alternative
terms, such as "school nurse."
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep: As is.

ETS Recommendation (05/02/2014)
Keep this item.
This item performed well in cognitive interviews. No issues were identified that would warrant revising
or dropping this item.

ETS Post-Cognitive Interview Item Recommendations

Page 206

ETS CogLab Report: NIES|2015
School #9

Revised/VH040486

What percentage of your 8th grade students dropped out of school during the last school year?
A. 0 – 2 %
B. 3 – 5 %
C. 6 – 10 %
D. 11 – 20 %
E. 21 – 30 %
F. 31 – 40 %
G. 41 – 50 %
H. More than 50 %

Item
Reading: NA
Understanding: One administrator expressed some confusion over the meaning of the phrase “dropping
out” (i.e., Does it refer to a student leaving school or leaving the entire school system?). Another
administrator questioned the time frame referenced by “last school year” (i.e., Does it refer to a
calendar year or the student’s last year in school?).
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
One administrator (out of nine respondents) noted that they lacked sufficient data to accurately answer
the question. Five administrators felt that the question was asking about dropout rates for the school.
Two thought the question was asking them to define the problem, while one administrator felt that the
question was asking for reasons why students drop out.
Probe 3: Please explain why you selected the answer choice. In other words, how did you calculate
the answer?
While four of the 11 respondents simply stated that student dropouts were not an issue in their schools
(thus, the answer selection would be A), four other respondents conducted a general count of students
still in school. Three administrators, though, noted that they lacked the information to come up with an
accurate calculation.
Probe 4: Do you think the question is easy or difficult to answer? Why is it easy or difficult to answer?
Of the 11 responses to the probe, five respondents found the question easy to answer, five found it
difficult, and one found it in between easy and hard to answer. When describing why, three respondents
ETS Post-Cognitive Interview Item Recommendations

Page 207

ETS CogLab Report: NIES|2015
cited the phrasing of the question, while two noted they lacked the information to calculate an answer.
One respondent explained that the question was easy to answer because student dropouts were not an
issue at the school.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Keep (with reservations): Question generally seems fine as is. However, given the reservations
expressed by multiple respondents regarding the lack of information (affecting their ability to accurately
calculate a figure for their response), there could be concern regarding the reliability of respondents’
answers and the validity of the question item. If the question is meant to assess how many AI/AN
students dropped out, consider adding that in and highlighting it.

ETS Recommendation (05/02/2014)
Revise this item.
Rationale: ETS agrees with KAI that this item should be revised to clarify that it is specifically asking
about AI/AN students.

ETS Post-Cognitive Interview Item Recommendations

Page 208

ETS CogLab Report: NIES|2015
School #10

New/VH040507

To what extent do your school’s 8th grade curricula emphasize preparing American Indian or Alaska
Native students for the following? Fill in one oval on each line.
Not
at all

Small
extent

Moderate
extent

Large
extent

a. Prepare students for life in their community,
tribe, group, pueblo, or village

A

B

C

D

b. Prepare students for the workforce

A

B

C

D

c. Prepare students for high school and college

A

B

C

D

New/VH0
40511
New/VH0
40518
New/VH0
40522

Item
Reading: NA
Understanding: One respondent needed to reread the question item a few times in order to understand
the phrasing.
Changing Answers: NA

Probes
Probe 1: Can you tell me, in your own words, what the question is asking?
Of the nine respondents, six felt the question was asking about how the school generally prepared
students. One respondent felt that the question was asking about teacher capabilities, while another
respondent thought it was asking if the school had high expectations for its students. One respondent
was confused by the question and thought it could be interpreted in multiple ways:
“I don’t know what they mean by “I prepare students for life.” I mean, if we’re preparing students for a
community that has 80% unemployment and no job opportunities, then we’re doing a good job because
a lot of our students don’t have opportunities when they graduate high school. But are we preparing
them to be active participants in their community’s culture? That’s different then … I guess without any
qualifying awards or statements, every single kid is being prepared to be part of their community, but
it’s like, what kind of preparation are they coming with? Are they prepared … Does that make sense? I
would interpret it …”
Probe 3: In question a., what does the phrase “prepare students for life in their community, tribe,
group, pueblo, or village” mean to you?
Three of the 11 respondents felt that the students were prepared for community life, but not tribal life.
Ten administrators felt that the probe item referred to general preparation for the work and behaving
appropriately in the community.

ETS Post-Cognitive Interview Item Recommendations

Page 209

ETS CogLab Report: NIES|2015
Probe 4: [Conditional probe: If the participant selects answer choice B, C, or D for question a., then ask
the follow-up question]: Can you give me examples of how your school’s 8th grade curricula
emphasize preparing American Indian or Alaska Native students for life in their community, tribe,
group, pueblo or village?
Four administrators (out of 11 respondents) noted that their school curricula did not prepare students
for life in their tribe. Six administrators felt that general school curricula and teacher interactions
prepared students, while two administrators discussed how students take part in community service
activities or otherwise engaged with the community. One administrator’s response proved difficult to
code:
“We're going to Reno for about two weeks in March to a summit that I've gone to for about the last five
years. When I read this though, I think in like history or in language arts, are they getting stories that
belong to them or to the American Indian culture? I still think that has a long way to go. That's my goal,
is that why wouldn't we take time to talk about this specific tribe in this area and their history when that
would make them have ownership, be a part of ... validate them.”
Probe 5: What does the word “pueblo” mean to you?
Six administrators (out of nine respondents) defined pueblos in terms of geographic location,
particularly the southwest. Four administrators noted that the term means “village” or “community,”
and three highlighted that the term was specifically Native in origin and reference.
Probe 6: In question b., what does the phrase “prepare students for the workforce” mean to you?
Of the 12 respondents, four defined the probe item as general preparation for the work place, two
provided specific examples of how the school prepared students, and five discussed how student
preparation included development of math, reading, speaking, reasoning, and social skills. One
respondent’s response was difficult to code:
“Well, not you're getting into personal beliefs of mine where I think, I'll just go back to begin with the
end in mind ... I'm getting philosophical in here, but a lot of times we're not thinking about the end
game and what it is that we want them to know and be able to do and I really dislike how sometimes we
don't make anything apply to different situations. I don't know if you saw the visuals out there, did you
see any of those out there?”
Probe 7: [Conditional probe: If the participant selects answer choice B, C, or D for question b., then
ask the follow-up question]: Can you give me examples of how your school’s 8th grade curricula
emphasize preparing American Indian or Alaska Native students for workforce?
Six of the 10 respondents reported that school courses and curricula prepared students for the
workforce. Three respondents described the use of team work or a general discussion of student
preparation when answering. One administrator’s response was difficult to code:
“I work with Fredina Drye-Romero and she is the American Indian Consultant there in Reno. Her mom
actually lives on the reservation. I went to a thing in Vegas the other day and I've gone on five different
American Indian college tours in different states from Clark County with kids checking out colleges. But
as far as this school, this is my first year here and already, like I said in there, it's probably more so than
ETS Post-Cognitive Interview Item Recommendations

Page 210

ETS CogLab Report: NIES|2015
your average principle, we've had the student being in boarding school presentation and he came here
taking the 7th grade US Nevada history teacher.”
Probe 8: In question c., what does the phrase “prepare students for high school and college” mean to
you?
All 11 administrators who responded said that the probe item referred to the academic preparation of
students for graduation.
Probe 9: [Conditional probe: If the participant selects answer choice B, C, or D for question c., then ask
the follow-up question]: Can you give me examples of how your school’s 8th grade curricula
emphasizes preparing American Indian or Alaska Native students for high school and college?
Ten administrators responded to the probe item. Of these, six cited college preparation coursework or
curricula, three cited the use of outside speakers or college prep programs, and one described general
discussions that staff had with students about preparing for high school and college.
Probe 10: Which answer choice would you select for questions a. through c. if the curricula is focused
on high school only?
Two of the 10 respondents found the question confusing. Three administrators reported that their
answer would not change, while four stated that their answer would. One administrator’s response
proved difficult to code:
“I mean, you really have to let them know that you do have high expectations and they've got to do
better than what they're doing in high school.”
Probe 11: Do you think the question is easy or difficult to answer? Why is it easy or difficult to
answer?
Ten administrators responded to the probe item. Of these, five found the question easy to answer,
while four found it difficult. One administrator responded that the question was in between (i.e., both
easy and difficult to answer.) Four respondents explained that the question’s phrasing is specific to high
school rather than middle or elementary school, while one commented on the complex and interrelated
nature of the various elements within the question.

Solutions to Improve Item
Respondent: NA
Interviewer: NA

Initial KAI Recommendation (Keep/Change/Discard)
Change: Several respondents noted that school curricula succeeds in preparing students for life in the
community but fails in preparing them for life in their tribes. Consider separating this into two different
questions, as it examines two different (though admittedly intertwined) aspects of a student life.

ETS Post-Cognitive Interview Item Recommendations

Page 211

ETS CogLab Report: NIES|2015
ETS Recommendation (05/02/2014)
Drop this item.
Rationale: ETS recommends dropping this item as respondents indicated that life in their community is
not necessarily the same as life in their tribes. Respondents indicated that the school curricula may
succeed in preparing students for life in their community, for instance, but not necessarily life in their
tribe. In addition, not all students may have a “tribal life” and may live in a community without other
AI/AN families. Furthermore, sub-items B and C may be more applicable to higher grades, as expressed
by some of the respondents, therefore they may not be appropriate for grade 8.

ETS Post-Cognitive Interview Item Recommendations

Page 212