National Indian Education Study (NIES) 2009 Reports Focus Groups

Appendix G
FOCUS GROUP PROTOCOL
I.

Introduction 
All information pertinent to the background and purpose of this study, the recruitment, contact

and consent processes, and the logistics of the focus groups are provided in the Supporting Statement.
The Focus Group Protocol contains the scripts and guides for the two focus groups:
Focus Group 1: NIES 2009, Part I: Performance of American Indian and Alaska Native Students at
Grades 4 and 8 on NAEP 2009 reading and mathematics Assessments;
and
Focus Group 2: NIES 2009, Part II: the Educational Experiences of American Indian and Alaska
Native Students in Grades 4 and 8 Statistical Analysis Report.

II.

 Focus Group Script and Guide for NIES Part I 

II.a.  Introduction Script 

Hello. My name is [insert name], from Kauffman and Associates, Inc. Kauffman & Associates, Inc., is an
American Indian, woman-owned company based in Spokane, WA, and we have been contracted by the

National Center for Education Statistics (NCES) to conduct a focus group with potential users and/or
  readers of the National Indian Education Study 2009, Part I: Performance of American Indian and Alaska
  Native Students at Grades 4 and 8 on NAEP 2009 Reading and Mathematics Assessments .

  A focus group is a guided discussion in which all participants are encouraged to respond to a series of
  broadly worded questions on a particular topic. For example, today’s topic is about a particular report that

has been recently issued on the academic achievement of American Indian and Alaska Native students. We
are very interested in hearing about what everyone in the group thinks and there are no wrong comments
or responses to comments. My role as moderator is to keep the discussion on focus and to make sure that
nothing is misunderstood. It is not to engage in the discussion.

  The purpose of today’s focus group is to help determine the utility of the report in terms of accessibility and
  functionality of language, layout, and clarity of presentation, and ease of accessing relevant information.
  The goal of this project is to provide feedback to NCES on the content, presentation, and usability of the
  report in order to improve the publication.

Kauffman & Associates, Inc. 

 1 

[Introduction Continued]: We have invited you here today because of your knowledge and experience

with American
Indian/Alaska Native education. We value your opinions and encourage your honest and
  complete feedback on the National Indian Education Study (NIES) 2009 publication on American Indian
th

th

  and Alaska Native 4 and 8 graders’ performance on NAEP, and we are thankful that you’ve agreed to
come to this group to share your thoughts and perspectives. While we encourage your full participation to

help inform and improve upon the utility of the NIES publication, there will be no negative consequences

for choosing not to respond to any questions during the course of this group.

We want to remind you that your answers may be used only for statistical purposes and may not be
  disclosed, or used, in identifiable form for any other purpose except as required by law (P.L. 107-279, Title
  I, Part E, Sec. 183). What is said here today will be aggregated based on the themes and topics we will be
  addressing. No names or identifying information will be associated with responses or appear on any
presentation or report on this focus group. We will be using a recording device to ensure that we preserve a

  complete rendering of what is shared in this group; however, if there are any objections to recording this

group we will rely on note-taking exclusively. Are there any questions? [Respond to any questions.]

II.b.  Informed Consent Script 

You received the Informed Consent form along with the study report when you registered. We invite you to

ask any questions you have regarding the Informed Consent process at this time.
  [Moderator answers any questions].
  If there are no other questions, then to express your written consent to participate in this focus group, we
  ask that you sign the form if you haven’t already, and place it in this manila envelope. You will receive a
copy of this statement for your records, which contains within it the contact information for the persons

responsible for this project, should any questions arise after we leave here today.
[Moderator collects consent forms].

II.c.  Demographic Survey Script 

To begin, we are going to ask each of you to complete this short 8-item demographic survey. This survey
contains only general demographic information that will be compiled to help us to characterize the
responses shared. No responses will be matched to specific demographic information or personal
characteristics.

  Demographic Survey. When participants have completed them, they will
[Distribute NIES Focus Group
place them in a large manila envelope that will be sealed immediately.]

Kauffman & Associates, Inc. 

 2 

II.d.  Focus Group Questions Script 
To get us started, we’d like to have you briefly introduce yourself. I will begin. [Moderator will once
again give her/his name. After the moderator has completed her/his introduction, (s)he will gesture to
the person to her/his immediate left who will then be asked to introduce her or himself. When
introductions have been completed, the note taker will turn on the recorder and the moderator will
begin the focus group. ]
The first couple of questions will ask you about your involvement and use of the NIES 2009, Part I and
your general feelings, thoughts, and reactions to it.
1. Please share something about your understanding of the NIES 2009, Part I and how you might
use the results of this study.
[Prompts will be used to help participants clarify their responses, to encourage further depth, and
to ensure that participants touch on all of the salient points addressed by the Question. To reduce the
burden on the respondent, if they have already addressed issues covered by the prompts, those
prompts will not be asked. Select from the following prompts as needed]:
•
•
•

What is your area of expertise and how did you come to be involved in American Indian
and Alaska Native education?
Please describe how you would use the report to inform the work you do?
In general, please describe the level of contact you have with 4th and 8th grade American
Indian and Alaska Native students, and the differences you note in educational experiences
of American Indian and Alaska Native students relative to other students.

2. In reviewing the NIES 2009 Part I Performance of American Indian and Alaska Native Students
at Grades 4 and 8 on NAEP 2009 Reading and Mathematics Assessments, what stands out as
the study’s most relevant features as a document in terms of utility, purpose, and accessibility?
[Select one or more of the following prompts as needed]:
• What was your initial reaction to reading this report and, after some reflection, what if
anything has altered or reinforced that perception?
• How do the features of the report affect your ability to use the information in the report?
• Describe how the presentation of the material in this report can affect the reader’s ability
and/or interest in using it?
• In what ways, if any, do the organization and presentation of the report enable you to
access the key information easily?
• What makes the report easy or not easy to share with staff and faculty?
• In what ways, if any, does the presentation of the report affect its utility within the
educational setting of a school?

Kauffman & Associates, Inc. 

 3 

3. Please describe in any way you choose, how this report makes you feel about how American
Indian and Alaska Native academic achievement is portrayed and understood.
[Select one or more of the following prompts as needed]:
•
•
•
•
•
•

Describe the report’s effectiveness in conveying and/or presenting its findings to the
reader.
In what ways did the presentation of the report help the reader hone in on areas of
interest?
How do you feel about the manner in which the data were analyzed?
Please share your thoughts on the presentation of the data in relation to being culturally
responsive.
Describe the level of sophistication of the analysis and its appropriateness for academic
planning at educational institutions serving American Indian and Alaska Native students.
Please describe a couple of ways educators could share the results of this report.

The next set of questions will address your feelings and thoughts related to specific aspects of the NIES
2009, Part I.
4. In what ways do you feel the content organization for the NIES 2009, Part I could affect how
the reader might understand or interpret the findings?
[Select one or more of the following prompts as needed]:
• What stands out as the major impact of how the content was presented on your
understanding or interpretation of the material?
• In what way or ways could the content organization of the report affect how its readers
could respond to it?
• What might be the different ways educators, parents, and community members could
interpret the data based on how it was presented throughout the report?
• Describe the effectiveness of the executive summary in providing the highlights of the
NAEP.
• How can you use any of the sections of the report, from the executive summary to the
appendices, to further your understanding of the results of the NAEP?
5. Sometimes color and print can influence a reader’s sense of the attractiveness of a report and
affect the comfort and ease of understanding elements of the report, such as the tables, graphs,
or the reading of the text. How did these factors influence your feelings about the study?
[Select one or more of the following prompts as needed]:
•
•
•
•

What did you think about the cover?
What are some of the aspects of the layout that worked for you?
How useful were the different sections in the report in terms of how they were set up?
What aspects of the layout did not work?

Kauffman & Associates, Inc. 

 4 

• How effective, in terms of readability, were the different fonts and font sizes used
throughout the publication?
• How did the fonts affect your interest in what was being presented?
• What effect did the use of color have on the transmission of information?
• What influence did color have on your ability to differentiate the sections in the study?
• Discuss the layout in terms of accessibility of data to researchers, administrators, educators,
parents, and policy makers.
6. Describe how this report can best be used to help improve American Indian and Alaska Native
reading and mathematics skills.
[Select one or more of the following prompts as needed]:
• What aspects of this report affect its appeal or utility for American Indian and Alaska Native
readers of it?
• Explain how the data presentation (e.g., order, layout, graphs, use of photographs and
examples) can affect the reader’s view of the information.
• Describe the effectiveness of the report in responding to the need for relevant data to
improve curricula for American Indian and Alaska Native serving schools.
• What about this report makes it suitable or unsuitable for dissemination to parents and/or
educators?
• What is the take away message from this study?
7. Describe the study’s accessibility for a diverse audience in terms of vocabulary, technicality,
and detail.
[Select one or more of the following prompts as needed]:
•
•
•
•
•

What do you think are the key factors of this report?
What would be the most salient points of this report for the different populations you
serve?
What would you be most interested in sharing with other educators?
How would you share the information in this study with parents?
In what ways would you use any aspects of the report in discussing the study with
community members?

8. Is there anything else about the NIES 2009, Part I, including its content, presentation, and
conclusions, that we have not discussed that you feel should be addressed?

Kauffman & Associates, Inc. 

 5 

III. 

 Focus Group Script and Guide for NIES Part II 

III.a.  Introduction Script 
  Hello. My name is [insert name], from Kauffman and Associates, Inc. Kauffman & Associates, Inc., is an
  American Indian, woman-owned company based in Spokane, WA, and we have been contracted by the
National Center for Education Statistics (NCES) to conduct a focus group with potential users and/or

readers of the National Indian Education Study 2009, Part II: The Educational Experiences of American

Indian and Alaska Native Students in Grades 4 and 8 Statistical Analysis Report.

  A focus group is a guided discussion in which all participants are encouraged to respond to a series of
  broadly worded questions on a particular topic. For example, today’s topic is about a particular report that
  has been recently issued on the educational experience of American Indian and Alaska Native students.
  We are very interested in hearing about what everyone in the group thinks and there are no wrong
  comments or responses to comments. My role as moderator is to keep the discussion on focus and to make
sure that nothing is misunderstood. It is not to engage in the discussion.

The purpose of today’s focus group is to help determine the utility of the report in terms of accessibility and

functionality of language, layout, and clarity of presentation, and ease of accessing relevant information.
  The goal of this project is to provide feedback to NCES on the content, presentation, and usability of the
  report for end users in order to improve the publication.

We have invited you here today because of your knowledge and experience with American Indian/Alaska

Native  education. We value your opinions and encourage your honest and complete feedback on this
th

National Indian Education Study (NIES) 2009 publication on American Indian and Alaska Native 4 and
th

8 graders’ educational experiences and are thankful that you’ve agreed to come to this group to share
your thoughts and perspectives. While we encourage your full participation to help inform and improve
upon the utility of the NIES publication, there will be no negative consequences for choosing not to respond

  to any questions during the course of this group.

  We want to remind you that your answers may be used only for statistical purposes and may not be
  disclosed, or used, in identifiable form for any other purpose except as required by law (P.L. 107-279, Title
  I, Part E, Sec. 183). What is said here today will be aggregated based on the themes and topics we will be

addressing. No names or identifying information will be associated with responses or appear on any
presentation or report on this focus group. We will be using a recording device to ensure that we preserve a
complete rendering of what is shared in this group; however, if there are any objections to recording this
group we will rely on note-taking exclusively. Are there any questions? [Respond to any questions.]

Kauffman & Associates, Inc. 

 6 

III.b.  Informed Consent Script 

You received the Informed Consent form along with the study report when you registered. We invite you

to ask any questions you have regarding the Informed Consent process at this time.

[Moderator answers any questions].

If there are no other questions, then to express your written consent to participate in this focus group, we

ask that you sign the form if you haven’t already, and place it in this manila envelope. You will receive a
copy of this statement for your records, which contains within it the contact information for the persons
responsible for this project, should any questions arise after we leave here today.
[Moderator collects consent forms].

III.c.  Demographic Survey Script 

To begin, we are going to ask each of you to complete this short 8-item demographic survey. This survey

contains only general demographic information that will be compiled to help us to characterize the

responses shared. No responses will be matched to specific demographic information or personal

characteristics.

[Distribute NIES Focus Group Demographic Survey. When participants have completed them,

they will place them in a large manila envelope that will be sealed immediately.]

III.d. Focus Group Questions Script, NIES Part II
To get us started, we’d like to have you briefly introduce yourself. I will begin. [Moderator will once
again give her/his name. After the moderator has completed her/his introduction, (s)he will gesture to
the person to her/his immediate left who will then be asked to introduce her or himself. When
introductions have been completed, the note taker will turn on the recorder and the moderator will
begin the focus group. ]

The first couple of questions will ask you about your involvement and use of the NIES 2009, Part II and
your general feelings, thoughts, and reactions to it.

1. Please share something about your understanding of the NIES 2009, Part II and how you
might use the results of this study.
[Prompts will be used to help participants clarify their responses, to encourage further depth, and to
ensure that participants touch on all of the salient points addressed by the Question. To reduce the

Kauffman & Associates, Inc. 

 7 

burden on the respondent, if they have already addressed issues covered by the prompts, those
prompts will not be asked. Select from the following prompts as needed]:
What is your area of expertise and how did you come to be involved in American Indian
and Alaska Native education?
• Please describe how you would use the report to inform the work you do?
• In general, please describe the level of contact you have with 4th and 8th grade American
Indian and Alaska Native students, and the differences you note in educational experiences
of American Indian and Alaska Native students relative to other students.
•

2. In reviewing the NIES 2009, Part II Educational Experiences of American Indian and Alaska
Native students at Grades 4 and 8, what stands out as the study’s most salient features as a
document in terms of its utility, purpose, and accessibility?
[Select one or more of the following prompts as needed]:
• What was your initial reaction to reading this report, and, after some reflection, what if
anything has altered or reinforced that perception?
• How do its features/aspects affect your ability to use the information in the report?
• Describe how the presentation of the material in this report can affect the reader’s ability
and/or interest in using it?
• What about the way this report is organized and presented enables you to access key
information easily?
• What makes the report easy or not easy to share with staff and faculty at your school?
• What about the way the report is written or presented can affect its use within the
educational setting of a school?
3. Please describe in any way you choose, how this report makes you feel about how American
Indian and Alaska Native culture and language is integrated into students’ educational
experience.
[Select one or more of the following prompts as needed]:
• Describe the report’s effectiveness in conveying and/or presenting its findings to the
reader.
• In what ways did the presentation of the report help the reader hone in on areas of
interest?
• How do you feel about the manner in which the data were analyzed?
• Please share your thoughts on the presentation of the findings in relation to being clear
about all the factors being addressed and analyzed (e.g., language, culture, school type,
American Indian and Alaska Native student, and teacher density).
• Describe the level of sophistication of the analysis and its appropriateness for academic
planning at educational institutions serving American Indian and Alaska Native students.
• Please describe a couple of ways educators could share the results of this report.
The following topics will address your feelings and thoughts related to specific aspects of the NIES
2009, Part II.
Kauffman & Associates, Inc. 

 8 

4. In what ways do you feel the content organization for the NIES 2009, Part II could affect how
the reader might understand or interpret the findings?
[Select one or more of the following prompts as needed]:
• In what way or ways could the content organization of the report affect how its readers
could respond to it?
• What might be the different ways educators, parents, and community members could
interpret the data based on how it was presented throughout the report?
• Describe the importance of the executive summary in terms of presenting a complete
overview of the results.
• Briefly share how you can or cannot connect the information obtained from students’,
teachers’, and administrators’ questionnaires into a coherent body of information?
• How can you use the sections of the report, from the executive summary to the
appendices, to further your understanding of the results of the data collection?
5. Pictures, graphs, tables, definitions, and examples of questions and responses were used in the
report to explain its findings. Please describe your views and reactions to the use of these
different methods in the report.
[Select one or more of the following prompts as needed]:
•
•
•
•
•
•

What message did you get from the different headlines and highlighted percentages used
throughout the report?
How did examples that were used in the report influence your understanding of the
findings?
Describe the ease of use of the graphs and your ability to access the information contained
in them.
When presented with the information offered in the graphs, what aspect or aspects were
most helpful in understanding the data being presented?
What aspects of the graphs were least helpful in understanding the data being presented?
What did you see as the overall intent of the different photographs presented throughout
the report?

6. Describe how this report can best be used to help improve the educational experience of
American Indian and Alaska Native students.
[Select one or more of the following prompts as needed]:
•
•
•

What aspects of this report affect its appeal or utility, either positively or negatively, for
American Indian and Alaska Native readers?
Explain how the data presentation (e.g., order, layout, graphs, use of photographs and
examples) can affect the reader’s understanding of the information.
Describe the effectiveness of the report to provide an understanding of how American
Indian and Alaska Native students are culturally experiencing school.

Kauffman & Associates, Inc. 

 9 

•

What about this report makes it suitable or unsuitable for dissemination to certain
audiences?
What is the take away message from this study?

7. Describe the study’s accessibility for a diverse audience in terms of vocabulary, technicality,
and detail.
[Select one or more of the following prompts as needed]:
• What do you think are the key factors of this report?
• What would be the most salient points of this report for the different populations you
serve?
• What would you be most interested in sharing with other educators?
• How would you share the information in this study with parents?
• In what ways would you use any aspects of the report in discussions with community
members?
The last question will deal with your thoughts about the NIES 2009, Part II.
8. Is there anything else about the NIES 2009, Part II, including its content, presentation, and
conclusions, that we have not discussed that you feel should be addressed?

IV. 

Conclusion and Compensation  
At the conclusion of the focus group session, participants will be thanked and will be

compensated for their time and effort in the form of a $50 gift card.
 IV.a. Conclusion and Compensation Script
This concludes our questions. We are very thankful for your time and your thoughtful responses. As an
expression of our gratitude, we have a thank you gift card for each of you, and we ask that you sign that
you have received it as you are leaving.

[Note taker records the names of each participant who receives a gift card, and participants sign
to indicate their receipt.]

V. 

Handling of Data 
Focus group data, once transcribed, will be scrubbed of identifiers. All data will ultimately be

electronically entered and password protected with limited access.
Viewing of the raw data will be limited to the following personnel: moderator, note taker Project
Director, and transcriber. Of the staff, only the moderator and note taker will have direct knowledge of
the person who participated in the study. In addition, all members of the focus group will know who
Kauffman & Associates, Inc. 

 10 

their fellow participants are. While it is impossible to ensure that confidentiality will be maintained
among all the members of the focus group, participants will be asked to observe and respect the
confidentiality of all participants.

Kauffman & Associates, Inc. 

 11