Justification -Volume I NIES Reports Focus Groups

National Assessment of Educational Progress

Supporting Statement

2009 National Indian Education Study Reports

Focus Group Studies

8/17/10

OMB# 1800-0011 v.142

Supporting Statement Table of Contents

1. Submittal-Related Information 1

2. Background and Study Rationale 1

3. Study Design and Context 2

3a) Recruitment 2

3b) Focus Group Logistics 3

4. Data Collection Process 3

5. Consultations Outside of the Agency 4

6. Assurance of Confidentiality 5

7. Justification for Sensitive Questions 5

8. Estimate of Hour Burden 5

9. Estimate of Costs for Recruiting and Paying Respondents 6

10. Cost to Federal Government 6

11. Project Schedule 6

12. References 6

Appendix A - Recruitment Letter

Appendix B – Participation Letter

Appendix C – Non-Selection Notification

Appendix D – Registration Letter

Appendix E – Focus Group Consent Form

Appendix F – Demographic Survey

Appendix G – Focus Group Protocol

SUPPORTING STATEMENT

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB #1800-0011). This generic clearance provides for the Department to conduct surveys and other studies regarding customer satisfaction.

2. Background and Study Rationale

The purpose of this focus group study is to evaluate the National Center for Education Statistics’ (NCES) biannual reports, National Indian Education Study - Part I: Performance of American Indian and Alaska Native Students at Grades 4 and 8 on NAEP 2009 Reading and Mathematics Assessments, and National Indian Education Study 2009 - Part II: The Educational Experiences of American Indian and Alaska Native Students in Grades 4 and 8 Statistical Analysis Report, and also to eventually recommend enhancements for future NIES reports. In 2004, the National Indian Education Study (NIES) was created to fulfill the United States’ responsibility for the education of American Indian and Alaska Native (AI/AN) children to meet the academic achievement standards set forth in the Elementary and Secondary Education Act of 2001. The study is sponsored by the Department of Education’s Office of Indian Education. The focus of the NIES is the academic achievement of AI/AN students and their educational experiences. Part I of the NIES gives an in-depth view of the academic performance of 4th and 8th grade AI/AN students on the National Assessment of Educational Progress (NAEP) 2009 reading and mathematics assessments. Part II of the study explores the educational experience of AI/AN students who participated in the NAEP assessments. Whereas Part I was completed by students, Part II contains survey responses from students, teachers, and administrators. To date, the NIES has been administered biannually from 2005 to 2009.

As part of the continuing effort to enhance the NIES reports, NCES will conduct a focus group study on the NIES 2009 reports (parts I and II) that NCES released earlier this year. To ensure that appropriate and relevant feedback is obtained from meaningful sources, participation will be sought from National Indian Education Association members.

The focus groups will seek to discern the knowledge, attitudes, and beliefs of the participants concerning the utility of the presentation of the data (e.g., accessibility and functionality of language), layout and data presentation, clarity of the information being reported, ease of presentation for accessing relevant information, and effectiveness of form to elucidate content (e.g., the role of color in alerting the reader to a change in subject matter or how user-friendly the color scheme is). In this manner, the focus groups will provide needed feedback on the content, presentation, and usability of the reports for end-users for improving existing methods of presentation of data for the varied audiences that use the reports to improve academic performance and the educational experience of AI/AN children in the U.S.. The focus of this activity will include the investigation of specific items (graphs, examples, statistics, etc.), mode of presentation, and knowledge conveyed. A report that summarizes the focus group findings will be produced.

3. Study Design and Context

Two focus groups will be held, one addressing the publication of the NIES 2009 - Part I report and one addressing the publication of the NIES 2009 - Part II report.

The focus group method is a qualitative, group approach to data collection. It involves holding group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue. The questions are purposefully open-ended in nature, incorporating several prompts to encourage a deeper, multifaceted exploration of an issue or topic. The method is useful for going beneath the surface of a response. As a research tool, focus groups are useful for understanding a targeted group’s views on complex social issues such as challenges and issues surrounding cultural issues and education. This data collection method has been frequently used in Indian country with considerable success (see e.g., Devlin, et al., 2006).

To obtain a broad geographical and cultural representation, to minimize the cost of the focus group study, and to reduce the potential burden of cost and time for the participant, NCES decided to hold the two sessions at a location and time that was already scheduled for participants. The National Indian Education Association’s (NIEA) annual meeting, being held in San Diego, CA, from October 7 to 10, 2010, was chosen as an apt site for the focus group study because of its broad-based membership, which includes researchers, educators, administrators, and policy makers. It is also anticipated that many of these individuals will also be AI/AN parents, with children in public or Bureau of Indian Education (BIE) schools.

3a. Recruitment

Kauffman and Associates, Inc. (KAI), will recruit participants and conduct the focus groups for NCES (refer to section 5 for information regarding KAI). A sample of NIEA members who would likely use the information in the reports will be recruited through an e-mail solicitation sent by Wanda Johnson, NIEA Convention Manager (See Appendix A: Recruitment Letter). This e-mail will be sent out by NIEA to 3000 NIEA members and will instruct interested participants to respond to KAI with their relevant contact and organizational affiliation information. Based on experience, a response rate of 2% to a recruitment letter asking for participation of this kind is expected. Respondents who are selected for participation will receive a response e-mail from KAI (See Appendix B: Participation Letter) to thank them for their interest, provide information regarding the purpose, time, and date of the focus group they have been assigned to, and provide the web link to the study report (Part I or Part II) for their review prior to the focus group. Participants will be informed in the Participation Letter that they are expected to spend approximately 60 minutes reviewing the study reports in preparation for their participation in the focus group, and that they will be provided the physical copy of the study report for their focus group at the time of registration at the NIEA conference. When the purposive sampling has been reached, respondents whose participation is not needed will receive a short e-mail letter thanking them for their interest and informing them that their participation will not be needed (See Appendix C: NonSelection Notification).

The ideal focus group size is 12 people. Having a group this size enables discussion and provides for a diversity of ideas to be heard. With more than 12 participants, it is not possible to ensure all views are heard or that a discussion can occur. It also has the potential of limiting the number of questions that can be addressed in the time allotted. Having a group with less than nine individuals limits the number of perspectives that are heard. It is also difficult to have a discussion developed when there are very few participants engaged in the process. It is anticipated that the NIES 2009 - Part I and the NIES 2009 - Part II focus groups will have 12 participants each, for a total of 24 people engaged in the study.

3b. Focus Group Logistics

At the time of registration at the NIEA Conference, participants will receive a letter (See Appendix D: Registration Letter) that outlines the logistics of their focus group participation, and is accompanied by both the Informed Consent Form (See Appendix E: Focus Group Consent Form) and the hard copy of the study report for their focus group. The Registration Letter will ask participants to read and bring the consent form to the focus group. The individual consent form outlines the name of the project and its purpose, the criteria and procedure for participation, including ability to withdraw from participation at any point during the process without negative consequence. It also outlines the risks and benefits of taking part in the study, the process for handling confidential information, any costs or compensation for involvement, and the project director’s contact information. All participants will receive a copy of the signed consent form for their records.

Each of the focus groups will be approximately 90 minutes long. They will take place during the early evening hours from 5:30 to 7:00 p.m., on October 8 and 9, when most conference participants are in attendance. This time schedule will help to ensure that the maximum number of individuals will be available to attend without interfering with participants’ attendance at the conference’s sessions or evening events.

4. Data Collection Process

At the chosen time, the group will convene where the focus group moderator will introduce the process, collect consent forms, collect basic demographic data (See Appendix F: Demographic Survey) and conduct the focus group discussion. The purpose of the demographic survey is to gather basic data with respect to age, gender, tribal and institutional affiliation, and association to AI/AN education (whether as professionals, parents, or both). This data will be used in aggregate form to contextualize the findings of the focus groups. The focus groups will be moderated by an experienced KAI researcher trained in conducting focus groups. This individual has an appreciation and knowledge of how and when to use prompts as well as how to keep a discussion moving and on target. This person is also knowledgeable about how to clarify questions without directing respondents toward a particular response. Because the focus group touches on areas that could be considered sensitive, it is extremely important that the focus group be conducted with sensitivity such as not to present bias or opinion during the session. A member of the research team will take notes during the focus group and a digital recorder will be used to ensure all comments are captured. The moderator will use the focus group guide for each of the two sessions (See Appendix G: Focus Group Protocol). Each focus group session will take approximately 90 minutes to complete.

The project team will employ a systematic approach to the analysis of the qualitative data and the verifiability of the findings (Krueger, 1998). In general, steps will include the execution of a transcript based upon the digital recording; creation of a word processing file for the text; and the analysis of the data using a qualitative software package, NVivo8, the latest edition of the program previously known as NUD*IST. NVivo8 has diverse functionality and great flexibility. Among its many features is its ability to enable the researcher to analyze materials straight from audio files. It also contains controls to playback audio at varying speeds as well as to skip to different sections. Researchers can use NVivo8 to synchronize the scroll of their text to the audio playback. In the analysis of each focus group, the senior researcher will be able to use two types of information. One will be the transcript of the discussion that allows for electronic manipulations. The second will be the actual audio recordings that allow the project team to decipher not just what a participant says but how the participant says it. The analysis will consider verbal and non-verbal reactions of the participants and include words, context, and internal consistency as well as the frequency, extensiveness, intensity, and specificity of comments. To analyze the focus groups, the following steps will be followed:

• Transcription: A professional transcriber will create a written text from the digital recording.

• Preliminary codes: The senior researcher will create a preliminary set of codes.

• Application of the codes: The senior researcher and program evaluator will independently apply the codes to the transcripts.

• Consultation among researchers: Cooperative work between senior researcher and program evaluator will lead to a final cross-coding with differences resolved through dialogue.

• Development of themes: Using NVivo8, senior researcher and program evaluator will create themes.

• Development of Context: Senior researcher and program evaluator will create themes and engage in ongoing dialogue and repeated analysis of data to develop context for the participants’ observations.

• Execution of a draft narrative of findings: Senior researcher will complete a draft of the detailed findings and choose verbatim quotes to illustrate key concepts.

• Review of the draft: Any other staff who took part in the focus groups will have an opportunity to verify and edit the draft.

5. Consultations Outside of the Agency

KAI is an American Indian, woman-owned organization that has worked on issues of education, health, and well being of Native communities for over 20 years. KAI has over six years experience in directly supporting the Office of Indian Education (OIE). With a corporate office located in the heart of Indian country in Spokane, WA, and an office in Silver Spring, MD, KAI currently employs 28 enrolled members of federally recognized tribes. KAI has worked closely with hundreds of tribal communities and thousands of tribal members to address a range of concerns. In addition to our experience with American Indian/Alaska Native education, KAI has worked to address other issues that directly affect education and schools in tribal communities, including family violence, bullying, and youth suicide. This company is known for its culturally appropriate and sensitive work in support of Native communities, and is highly respected by tribal leaders for the services they provide.

Jo Ann Kauffman, founder and president of KAI, is an enrolled member of the Nez Perce tribe and grew up in Seattle, Washington, and on the Nez Perce Reservation in Kamiah, Idaho. She holds a Master of Public Health degree from the University of California at Berkeley. After working in the field of Indian health care for many years, including serving as executive director of the Seattle Indian Health Board, she founded Kauffman & Associates, Inc. in 1990. As the President and CEO of KAI, she has grown the company, remained actively involved in its development, and ensured high quality management support for KAI clients. She has served on numerous national boards and was honored with the “Free Spirit Award” by the Freedom Forum in 1998 for her longtime work as a community activist and advocate for First Amendment issues. Ms. Kauffman was appointed by the governor of the state of Washington to the Board of Trustees of Eastern Washington University, where she currently serves as chair.

6. Assurance of Confidentiality

NCES has policies and procedures that ensure privacy, security, and confidentiality, in compliance with the legislation, Education Sciences Reform Act of 2002 (ESRA), P.L. 107-110, 20 U.S.C. § 9622. Specifically for the NIES project, this ensures that privacy, security, and confidentiality policies and procedures are in compliance with the Privacy Act of 1974 and its amendments, NCES Confidentiality Procedures, and the Department of Education ADP Security Manual.

Participation is voluntary and personally identifiable information will not be maintained for the participants. During the informed consent process, participants will be asked to read and sign the consent form (See Appendix E: Focus Group Consent Form).

After the participant and the moderator sign the consent forms, copies will be made for the participants to have for their records. Only individuals who sign the consent form will be eligible to participate in the focus group.

7. Justification for Sensitive Questions

This study does not involve sensitive questions.

8. Estimate of Hour Burden

Number of Respondents, Frequency of Response, and Burden

This study includes two focus groups. The total number of respondents will be 24 (12 per group).

The estimated total response burden is 125 hours for the NIEA Personnel sending the recruitment letter, the 3000 NIEA Members receiving the recruitment letter, the 60 (2%) of NIEA Members recruited who are expected to respond, and the 24 Focus Group Participants.

Focus Group participants’ total response burden is comprised of .017 hours (one minute) to read the recruitment letter, .083 hours (five minutes) to write a response expressing interest in participation, 1.0 hours to review the NIES report before the focus group and 1.75 hours for the focus group interviews.

Respondents are NIEA members who work with AI/AN children in an educational setting, as researchers, tribal policy makers, and/or parents of AI/AN children.

Table 1. Estimated Burden Hours

Respondent Type	Number of Respondents	Burden Hours Per Respondent	Total Hours
NIEA Personnel Involved in Recruitment	1	3	3
NIEA Members Contacted during Recruitment	3000	.017	51
NIEA Members Responding to Recruitment Letter	60	.083	5
Focus Group Participants	24	2.75	66
Total Respondents	3001		125

9. Estimate of Costs for Recruiting and Paying Respondents

The estimated costs for recruiting and paying respondents are $1,200. All focus group participants will be compensated $50 payment in the form of a gift card. The $50 falls within the focus group incentive range. This amount is proposed to compensate the respondent for the hour to review the NIES report and the 1.75 hours for the focus group interview.

10. Cost to Federal Government

The estimated cost to the government of all recruitment and data collection, analysis, and reporting activities for this study is $16,701

11. Project Schedule

The following table graphically depicts the schedule for efforts related to the implementation of the focus groups.

Activity	Date
Begin coordination with NIEA logistics manager for recruitment process	9/10
Conduct two focus groups (NIES 2009 - Part I and NIES 2009 - Part II) at the 2010 National Indian Education Association Conference	10/7/10 and 10/8/10
Complete transcription of focus groups	10/18/10
Draft and submit a summary report on focus group studies	11/15/10
NCES review of draft summary report	11/22/10
Revise and submit a final summary report on focus group studies	11/30/10

12. References

Devlin, H., Roberts, M., Okaya, A., and Iang, Y. M. (2006). Our lives were healthier before: Focus groups with African American, American Indian, Hispanic/Latino people with diabetes. Health Promotion Practice, 7(1): 47-55.

Krueger, R. (1998). Analyzing & reporting focus group results. Thousand Oaks, CA: Sage Publications.