PPSS_TO24_OMBClearance_PartA_062916

JANUARY 2015

Task Order 24: Study of the Title III Native American and Alaska Native Children in School (NAM) Program

Office of Management and Budget Clearance Request

Part A

Prepared for

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

By

American Institutes for Research

June 29, 2016

Contents

Page

Introduction 1

Supporting Statement for Paperwork Reduction Act Submission 2

Justification (Part A) 2

A1. Circumstances Making Collection of Information Necessary 2

A2. Use of Information 6

A3. Use of Improved Technology to Reduce Burden 6

A4. Efforts to Avoid Duplication of Effort 7

A5. Efforts to Minimize Burden on Small Businesses and Other Small Entities 7

A6. Consequences of Not Collecting the Data 7

A7. Special Circumstances Causing Particular Anomalies in Data Collection 7

A8. Federal Register Announcement and Consultation 8

A9. Payment or Gift to Respondents 9

A10. Assurance of Confidentiality 9

A11. Sensitive Questions 10

A12. Estimated Response Burden 10

A13. Estimate of Annualized Cost for Data Collection Activities 12

A14. Estimate of Annualized Cost to Federal Government 12

A15. Reasons for Changes in Estimated Burden 12

A16. Plans for Tabulation and Publication 12

A17. Display of Expiration Date for OMB Approval 15

A18. Exceptions to Certification for Paperwork Reduction Act Submissions 15

Description of Statistical Methods (Part B) 16

B1. Sampling Design 16

B2. Procedures for Data Collection 16

B3. Methods to Maximize Response Rate 19

B4. Expert Review and Piloting Procedures 20

B5. Individuals and Organizations Involved in the Project 20

References 22

Appendix A. Interview and Focus Group Protocols 26

Appendix B. Notification Letters 84

Appendix C. Study Information Document 89

Appendix D. Institutional Review Board (IRB) Letter of Approval 92

Introduction

The Policy and Program Studies Service (PPSS), within the U.S. Department of Education’s Office of Planning, Evaluation and Policy Development, requests Office of Management and Budget (OMB) clearance for the Study of the Title III Native American and Alaska Native Children in School (NAM) Program.

The NAM program seeks to improve academic outcomes in English for Native American and Alaska Native (NA/AN) students. The NAM program provides funding for activities that support language instruction educational programs, including those that support NA/AN language and culture revitalization, as long as there also is a focus on improving academic English proficiency. Other NAM priorities include promoting family and community engagement and enabling data-based decision making.

The purpose of this study is to describe how current (FY 2011 and FY 2013) grantees have used NAM program funding, including which activities grantees have implemented with the funds, how they prioritize funded activities in relation to other education programming, how they measure student progress, challenges in providing services for this population, and lessons learned from their experiences. Results from this study may help the Department better support grantees.

Clearance is requested for the case study component of the study, including the purpose for the case studies, data collection procedures, and data analysis approach. This submission also includes the clearance request for the data collection instruments and study participant and stakeholder notification letters.

Supporting Statement for Paperwork
Reduction Act Submission

Justification (Part A)

A1. Circumstances Making Collection of Information Necessary

Study Overview

NA/AN students have traditionally scored lower than national averages on many achievement metrics, including measures of reading and English language arts (e.g., DeVoe, Darling-Churchill, & Snyder, 2008; Education Trust, 2013; Nelson, Greenough, & Sage, 2009). For example, during the period from 2005 to 2011, NA/AN fourth- and eighth-grade students consistently achieved lower reading scores than non-NA/AN students on the National Assessment of Educational Progress (NAEP). In 2011, fourth-grade NA/AN students scored 19 points lower, on average, in reading than non-AI/AN students, and eighth-grade NA/AN students scored 13 points lower (NAEP, 2011). In addition, among the 57 percent of high school graduates who took the 2014 ACT, NA/AN students were less likely to meet the college readiness benchmark in English (40 percent of NA/AN students vs. 64 percent of all students) or in reading (25 percent of NA/AN students vs. 44 percent of all students) (ACT, 2014).

The NAM program attempts to address these achievement gaps by providing five-year grants to eligible educational entities that deliver academic English support to NA/AN students. The NAM program gives priority to projects that promote family and community engagement, that employ native language instruction, and that enable data-based decision making to support language instruction. Components of supported programs may include teacher professional development, curriculum development, evaluation and assessment, and technology use. These activities are hypothesized to improve NA/AN academic outcomes by enhancing the educational environment for these students.

This study will examine the implementation of the NAM program at all 22 current (FY 2011 and
FY 2013) grantee sites. The study will provide information about how current grantees are using the NAM funds, with the goals of improving the Department support provided to grantees and of shaping NAM priorities in future funding rounds. Specifically, the study will answer the following questions:

1. How do NAM grantees use NAM funding to support activities intended to increase NA/AN student academic achievement?

2. How do NAM grantees work with partners to provide funded services?

3. How do NAM grantees measure progress and outcomes of funded services?

4. How do NAM grantees address challenges in providing funded services?

5. What are the NAM stakeholders’ perceptions of community participation and student engagement in language instruction and other educational programs?

This study consists of three components: (1) an analysis of current (FY 2011 and FY 2013) grantees’ applications and other extant descriptive data about the grants, (2) semi-structured telephone interviews with current NAM grant coordinators, and (3) in-depth site visits to conduct interviews or focus groups with key stakeholders in the 22 grantee sites.¹ The site visits are the study’s main component.

Study Conceptual Framework

The NAM program has five main funding priorities: (1) academic English support (through teacher professional development, curriculum development, and evaluation and assessment); (2) family engagement; (3) native language instruction; (4) data-based decision making; and (5) using technology to support language instruction. These activities are intended to improve students’ proficiency in academic English. Activities are also intended to improve the educational environment for students by increasing family and community participation and overall student engagement, intermediate outcomes that may, in turn, contribute to improved student academic achievement. The study will examine how grantees implement the NAM program, with a primary focus on the five grant priorities and some attention also to stakeholders’ perceptions of community participation and student engagement. The study’s conceptual framework, as it relates to the NAM priorities and intended outcomes, is shown in Exhibit 1.

Exhibit 1. NAM Study Conceptual Framework

In examining how grantees implement activities associated with the NAM priorities, the study team will draw on previous research supporting these activities as means to improve NA/AN achievement:

1. Academic English support. The NAM program supports the development of academic English through teacher training and professional development, curriculum development, and evaluation and assessment. Many studies have established connections between targeted literacy training and teacher knowledge (Spear-Swerling & Brucker, 2003) and teacher knowledge and instructional effectiveness in the areas of reading and language (Foorman et al., 2006; Lane et al., 2009; Spear-Swerling & Brucker, 2004). The effectiveness of professional development is enhanced through the provision of a high-quality curriculum that can serve as a model for teachers and give them practice with theory and research-based materials and methods (Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).

2. Family engagement. Although parent engagement is recognized as important for student achievement (Jeynes, 2005), teachers in schools with large numbers of NA/AN students have identified a lack of parent engagement as one of the biggest challenges their schools face (Freeman & Fox, 2005). Reasons for lack of parent engagement are complicated but may largely stem from a history of poor educational opportunities and unfriendly school climates for NA/AN students in the U.S. education system (Mackety & Linder-VanBerschot, 2008). Strategies for improving family engagement include instituting open-door policies for families and demonstrating respect for families’ cultural values, especially through culturally appropriate facilitated meetings, activities, and resources.

3. Native language instruction. The inclusion of NA/AN languages (and cultures) in NAM supports for students is crucial, both for those who speak an NA/AN language at home (an increasingly rare phenomenon) and for NA/AN students who speak English at home. For students who speak a non-English language at home, a growing body of research has indicated that first-language instruction promotes gains in English achievement and that bilingualism is an asset (e.g., Cummins, 1979; Dressler & Kamil, 2006; Francis, Lesaux, & August, 2006; Greene, 1997; Rolstad, Mahoney, & Glass, 2005; Willig, 1985). For NA/AN students, regardless of their first language, growing evidence shows that NA/AN language instruction may improve students’ reading and metalinguistic skills (Bacon, Kidd, & Seaberg, 1982; Hirata-Edds, 2011). Descriptive studies suggest that the incorporation of NA/AN languages increases students’ feelings of cultural pride or self-worth (Holm & Holm, 1995), creates more positive learning environments, increases student on-task behavior (Watson-Gegeo, 1989), and results in higher attendance rates (Reyhner, 1989), all factors associated with student engagement in learning. In addition, there is a strong consensus among tribal leaders, policymakers, families, and education leaders that the integration of indigenous language and culture is a key element of educational success (see reviews in Castagno & Brayboy, 2008, and Demmert & Towner, 2003), and NA/AN language and culture transmission is a priority goal across most NA/AN tribes (National Congress of American Indians & National Indian Education Association, 2010).

4. Data-based decision making. In recent years, interest in using data to inform decisions about instruction has increased (Chen, Heritage, & Lee, 2005; Marsh, Pane, & Hamilton, 2006). This approach is a natural result of technological changes, the advent of test-based accountability systems, and the proliferation of quantitative data due to accountability reforms (Tyler, 2011). Some research suggests positive links between data use and student achievement on state assessments, provided there is sufficient support for using the data (Faria et al., 2012), although no research to date examines this link specifically for NA/AN students. The NAM program encourages data-driven decision making as a funding priority.

5. Using technology to support language instruction. The final priority of the NAM program is the use of technology to support language instruction. Some evaluations of technology initiatives have documented positive impacts on student engagement among the general population (Argueta, Huff, Tingen, & Corn, 2011; Bebell & Kay, 2009) and skills related to communication, research, and writing (Bebell & Kay, 2009; Silvernail, 2005; Valiente, 2010). For English learners and other students with low literacy skills, the use of multimedia-enhanced instruction (Silverman & Hines, 2009) and revisiting materials in engaging ways that differ from the initial encounter have proven helpful (Roberts & Neal, 2004). For NA/AN language instruction especially, technology is important for augmenting the language instruction that can be provided by increasingly low numbers of fluent speakers (Hugo, 2014).

In summary, this study’s collection of data from multiple sources, especially interviews and focus groups with NAM stakeholders (the focus of this clearance request), will provide systematic information about the activities in which NAM grantees engage.

A2. Use of Information

The study results will be of immediate interest and significance for the Office of English Language Acquisition (OELA), because the study results will offer detailed information about how the NAM program is implemented in different tribal and educational contexts and what challenges grantees encounter. This information is important to guide how OELA structures and prioritizes future grant opportunities.

The study team will use interview and focus group data collected during program site visits to develop data tables and summaries of the site visit data for presentation to Department stakeholders. The study team also will produce a final publicly available report of cross-site findings, which will include relevant examples of common practices to illustrate findings, as well as descriptions of less common or outlier grant implementation activities. In designing this report, the study team will focus on developing a useful product that addresses multiple audiences, including grant coordinators, tribal education entities, district superintendents, principals, teachers, federal and state policymakers planning for academic interventions for NA/AN students, and researchers conducting studies on this topic. The information will be useful for OELA as the office plans future rounds of the grant. Tribal entities and educators of NA/AN students will be able to take the specific information included in the report to implement similar activities in their own tribes, districts, and schools.

A3. Use of Improved Technology to Reduce Burden

The data collection plans for the case study component of this study reflect sensitivity to issues of efficiency and respondent burden. Before data collection begins, the study team will access extant data available from the OELA website (i.e., grantee applications, notices inviting applications, and project abstracts) as well as information about the grant sites through Internet searches.² The study team will enter information relevant to the study questions into a database that will track information about each site or program. The study team will review this information before each site visit and, as much as possible, reduce the number of questions asked during interviews and focus groups based on what is already known about each site or program.

During the data collection process, the study team will continue to use technology to reduce burden whenever possible. For example:

1. Interviews and focus groups will be audiotaped and then transcribed at a later date to reduce the amount of time that participants will engage in interview activities.

2. A phone number and e-mail address will be provided to study participants, allowing them to contact research staff directly with any questions they may have.

A4. Efforts to Avoid Duplication of Effort

The study team will avoid duplication of effort by using preexisting data (e.g., grant applications, program information available on school or district websites, published program evaluations), whenever possible, to guide data collection. For example, as described previously, the study team will determine whether any of the proposed data collection elements for the site visits can be addressed through preexisting policy or evaluation documents. This step will reduce the number of questions asked in the interviews and focus groups. However, note that systematic data about program implementation are not already available in the detail required for this study.

In addition, the study team has designed the study protocols so that interviewers ask broad questions but systematically track topics of interest, asking detailed follow-up questions only as necessary. In this way, interviewees are not asked to answer questions for which they have already effectively provided a response.

A5. Efforts to Minimize Burden on Small Businesses and Other Small Entities

Some school districts and tribal entities likely to be involved in this study have fewer than 50,000 students and are thus considered small entities. Data collection will be conducted during a wide timeframe and site visits will be scheduled at dates and times that are most convenient for each site.

A6. Consequences of Not Collecting the Data

Failure to collect the data proposed through these interviews and site visits will limit the information available to OELA to guide future NAM grant rounds and to plan support for current and future grantees. In addition, failure to collect the data proposed would prevent the distribution of in-depth information to policymakers and practitioners across the nation about the use of such programs. The absence of this study’s final report could therefore hinder federal, tribal, district, and school stakeholders’ abilities to make careful and informed decisions about NA/AN academic program implementation.

A7. Special Circumstances Causing Particular Anomalies in Data Collection

None of the special circumstances listed in the Paperwork Reduction Act Submission Form General Instructions apply to this data collection.

A8. Federal Register Announcement and Consultation

1. Federal Register Announcement. A 60-day notice to solicit public comments was published in the Federal Register on 3/25/2016 (Volume 81, FR 16154) and no public comments were received.

2. Consultations Outside the Agency.

• Following Department procedures for research with NA/AN participants, the study was presented at a monthly consultation with tribes on January 12, 2016. Participants in the consultation did not offer feedback about the study design.

• A technical working group was consulted as part of this study and will continue to provide conceptual and methodological considerations for the collection, analysis, and reporting of the case study data. The technical working group members provided comments about the study design and data collection instruments in March 2016 and will provide additional feedback about the study’s final report. The study’s technical working group members, listed in Exhibit 2, have considerable expertise in the areas of NA/AN education and research methodologies, NA/AN language and culture revitalization, teacher training, language learning and development, and language education policy. The members demonstrate diverse tribal and geographical representation.

Exhibit 2. Technical Working Group Members

Name	Affiliation
Bryan Brayboy	(Lumbee) Codirector of the Center for Indian Education, Arizona State University
Susan Faircloth	(Coharie Tribe of North Carolina) Director, Indigenous Collaborative on Education, Research and Service, North Carolina State University
Dwight Pickering	(Caddo, Otoe, and Kaw Tribes) Director of American Indian Education Culture/Heritage, Oklahoma State Department of Education
Edward Valandra	(Sicangu Titunwan) Native Studies Sessional Instructor at University of Manitoba
Relyn Strom	(Confederated Tribes and Bands of the Yakama Nation) Principal, Yakama Nation Tribal School

• Per their special relationships with the U.S. government, sovereign tribal nations have legal jurisdiction over all activities, including research activities, that occur within their territories (National Congress of American Indians Policy Research Center & MSU Center for Native Health Partnerships, 2012). Therefore, in tribes involved in the data collection that have formal research approval processes, tribal Institutional Review Boards (IRBs) review the study design and data collection instruments.

A9. Payment or Gift to Respondents

It is respectful and culturally appropriate for non-tribal entities, when working with NA/AN tribes, to share a small gift with each tribal interview or focus group participant as a sign of respect for their culture and recognition of their time and effort to participate in the study. Therefore, each participant will receive a small gift of wild rice in a hand-sewn bag at the conclusion of the interview or focus group.

A10. Assurance of Confidentiality

As researchers, the study team is vitally concerned with maintaining the anonymity and security of the study records. The contractor’s project staff have extensive experience collecting information and maintaining the confidentiality, security, and integrity of interview and focus group data. All members of the study team have obtained their certification on the use of human subjects in research as well as federal security clearances. This training addresses the importance of the confidentiality assurances given to respondents and the sensitive nature of handling data. The team also has worked with the IRB at the contractor organization and at some of the tribes involved in the data collection to seek and receive approval of this study, thereby ensuring that the data collection complies with professional standards and government regulations designed to safeguard research participants.

The following data protection procedures will be in place:

• The study team will protect the confidentiality of the information respondents provide, to the extent provided by law. After the study team collects responses, respondents’ names and the institution's or school's name will be disassociated from the data. Pseudonyms will be used for each grantee. Responses will be used to summarize findings in an aggregate manner (across groups of grantees), or will be used to provide examples of program implementation in a manner that does not associate responses with a specific site or individual. Although participating institutions will be acknowledged in the final report, they will not be identified in the text of any report.

• The study team will protect the identity of individuals from whom we collect data for the study to the extent possible (given the uniqueness of the projects included in the study) and will use it for research purposes only. Respondents’ names will be used for data collection purposes only and will be disassociated from the data prior to analysis.

• Prior to beginning interviews and focus groups, a member of the research team will explain to participants what will be discussed, how the data will be used and stored, and how anonymity will be maintained. Participants will be instructed that they can stop participating at any time. The study’s goals, data collection activities, participation risks and benefits, and uses for the data will be detailed in a consent form that all participants will be provided prior to beginning any data collection activities. Participants will be informed that sites may be named in reports but that individuals will not be named specifically.

• All electronic data will be protected using several methods. The contractor’s internal networks are protected from unauthorized access by defense-in-depth best practices, which incorporate firewalls and intrusion detection and prevention systems. Access to computer systems is password protected, and network passwords must be changed on a regular basis and conform to the contractor’s strong password policies. The networks also are configured so that each user has a tailored set of rights, granted by the network administrator, to files approved for access and stored on the local area network. Access to all electronic data files and workbooks associated with this study will be limited to researchers on the case study data collection and analysis team.

A11. Sensitive Questions

This study will not include the collection of sensitive information. The only data to be collected directly from case study participants will focus on grant implementation and other education policies and practices rather than on individual people. District and school policies and practices are within the public domain (e.g., schools communicate their policies and programs to students and parents in a variety of ways). In this sense, the data are not sensitive in nature.

A12. Estimated Response Burden

It is estimated that the total hour burden for the case study data collection is 504.5 hours. This translates to an estimated cost of $15,093 based on the average hourly wage of participants. This estimate assumes the following data collection activities:

• One-and-one-half hour interviews with 22 grant coordinators

• One-hour interviews with tribal education directors in the 24 tribes in which there are current NAM grantees

• Forty-five minute interviews with superintendents in the 22 districts in which there are current NAM activities

• One-hour focus groups with principals in the 54 schools in which there are NAM activities

• One-hour focus groups with up to 10 program teachers or other program staff at each of the 22 NAM grantee sites

• One-hour focus groups with up to three tribal college instructors at each of the four colleges involved in NAM activities

• One-hour interviews with parent/family coordinators at the 21 NAM sites in which there are family outreach activities

• One-hour focus groups with up to three professional development providers in each of the 16 NAM sites in which there are separate professional development providers

• One-hour interviews with up to two language instruction or curriculum development specialists in each of the 16 NAM sites that have such specialists

• One-hour interviews with up to two people in administrative or evaluation roles in each of the 22 NAM sites

Exhibit 3 summarizes the estimates of respondent burden for interviews and focus groups across all 22 study sites. Each respondent will participate in a single interview or focus group. The estimated burden associated with the data collection at each individual study site is 22 hours, 56 minutes.

Exhibit 3. Estimated Total Hour and Monetary Cost Burden of Case Study Data Collection

Task	Total Sample Size	Estimated Response Rate	Number of Respondents	Time Estimate (in Hours)	Total Hour Burden	BLS Wage Code3	Mean Hourly Rate	Estimated Monetary Cost of Burden
Interviews with grant coordinators	22	100%	22	1.5	33	11-9030	$43	$1,419
Interviews with tribal education directors	24	100%	24	1	24	11-9030	$43	$1,032
Interviews with public school superintendents	22	100%	22	0.75	16.5	11-9030	$43	$710
Focus groups with principals	54	100%	54	1	54	11-9030	$43	$2,322
Focus groups with teachers	220	100%	220	1	220	25-2000	$27	$5,940
Focus groups with tribal college instructors	12	100%	12	1	12	25-1124	$33	$396
Interviews with parent/family coordinators	21	100%	21	1	21	21-1021	$22	$462
Focus groups with professional development providers	48	100%	48	1	48	25-3090	$17	$816
Interviews with language instruction and curriculum development specialists	32	100%	32	1	32	25-3090	$17	$544
Interviews with grant administrators or evaluators	44	100%	44	1	44	11-9151	$33	$1,452
Total for case study data collection	499	—	499	—	504.5	—	—	$15,093
Annualized burden	499	—	499	—	504.5	—	—	$15,093

A13. Estimate of Annualized Cost for Data Collection Activities

There are no additional annualized costs for data collection activities associated with this data collection beyond the total hour burden estimated in item A12.

A14. Estimate of Annualized Cost to Federal Government

The estimated cost to the federal government for the Task Order 24 study, including development of the research plan and data collection instruments as well as data collection, data analysis, report preparation, study management, and technical working group meetings is $999,084 for the 29 months of the study, or approximately $413,414 per year.

A15. Reasons for Changes in Estimated Burden

This is a new data collection.

A16. Plans for Tabulation and Publication

In qualitative research, well-planned, systematic, and transparent data collection and analysis techniques yield reliable, transferable findings (Anfara, Brown, & Mangione, 2002; Creswell, 1998). The study team will establish and adhere to a set of qualitative analytic procedures and standards to limit bias and ensure reliable findings. These procedures are part of a four-step analytic process, illustrated in Exhibit 4, guided by the study questions and conceptual framework. Each step is described in more detail in the text that follows.

Exhibit 4. Overview of Qualitative Analysis Process

Develop Codebook

The case study team will develop a coding framework based on the conceptual framework and linked to constructs underlying the interview and focus group protocols. Codes for analyzing site visit data will be structured so that analysts on the study team can apply more than one code to the same interview passage as applicable.

The case study lead will develop a codebook describing the framework. The codebook will include a complete map of all codes and sub-codes and overall guidance for analysts. For each code, the codebook will indicate alignment to the study questions or other aspects of the conceptual framework, links to specific interview and focus group questions, and notes and examples to guide analysts. The case study lead and another study leader will code three to five transcripts together, noting potential difficulties and areas of confusion arising from the coding structure. They will then either change the codebook to eliminate confusing areas or note in the codebook how analysts should address any confusion.

Code Interview and Document Data

The case study lead will train analysts to ensure that the codes are applied consistently. After the training, each analyst will separately code the same three to five transcripts, using a qualitative data analysis software program (NVivo). This program allows analysts to assign relevant codes to data and then compare coded data across sources. The case study lead will use the software’s inter-rater reliability test feature in addition to random spot checking to determine if the analysts are sufficiently consistent with the master coding (the inter-rater reliability threshold will be a weighted Cronbach’s alpha rating of 95 percent or higher). Analysts who are inconsistent will receive additional training. The case study lead will meet with the analysts at regular check-ins to confirm the consistent application of the codebook. The case study lead also will review a sample of coded transcripts to check for consistency and agreement with the codebook.

During the coding process, analysts will follow the codebook but will also have latitude to create new codes for emergent themes. In some cases, the codebook will prompt analysts to create new sub-codes as they work (e.g., sub-codes within a “challenges” code). At the conclusion of coding, the case study lead will use the software to merge all data files and will then merge similar sub-codes by hand, creating robust lists of emergent sub-codes within codes. For more pervasive emergent themes (e.g., a finding that multiple sites are using a particular instructional technique), the team will meet and agree upon a new code. The analysis team will meet weekly to discuss emergent themes as well as any questions about how to apply the codes and discuss any disputable data source. Any disagreements regarding codes will be resolved by discussion and consensus among coders.

Analyze Within-Case Data

After all transcripts have been coded, analysts will identify the patterns, themes, and categories that are most relevant to the study questions. This process of data reduction involves noting the prevalence of each response, group differences, and associations among data sources. Specifically, analysts will use the software program to query coded data in order to summarize findings for each site. Analyzing within-case data will be a systematic process that relies on coded data and not any one researcher’s or respondent’s perspective on the program or policy. To this end, analysts will review the frequency of coded responses to characterize which themes are common and which themes are outliers, identifying the most common themes. Analysts also will examine differences in responses among respondent types, such as by role (e.g., principal, grant coordinator, tribal education director).

The study team will establish clear standards of evidence in order to draw conclusions. For most topics, analysts will seek convergence of perspectives to draw conclusions (i.e., at least two analysts agree, with no contradictory evidence). However, given the limited number of respondents and cases in this study, there might be topics for which there will be only one knowledgeable respondent (e.g., a district leader may be the only person knowledgeable about the development of the program). The study team will adjust standards of evidence for each study question to account for this possibility before beginning analysis.

Perform Cross-Case Analysis

In the final stage of analysis, the study team will draw on the individual within-case analyses to identify the prevalence of specific practices as well as patterns in relationships among topics across cases, organized by study question. The study team will then create data tables in a spreadsheet that summarize the data. The data tables shall include cross-tabulations if the data warrants them. Examples include:

• Summaries of program activity types (e.g., teacher training, curriculum development, instruction, family outreach, technology usage) and cross-tabulations of program activities by program goals and Department funding priorities

• Summaries of how program activities are implemented

  • Structure and content of teacher training

  • Source and content of new curricula or materials

  • Source and types of student evaluation and assessment

  • Types of family engagement strategies

  • Structure and content of native language instruction (e.g., immersion, dual language instruction)

  • Types of technology and how students access the technology

• Summaries of reported challenges in providing services and cross-tabulations of challenges by primary program activities and reported levels of involvement of stakeholders (e.g., tribal entities, state and local education agencies)

• Summaries of stakeholders’ perceptions of progress and outcomes

Reporting

Data collected for each case or site will be analyzed and included in in a publicly released case study report as well as summarized in a Results in Brief document. The case study report will begin with an introductory section that: (1) features an audience-appropriate overview of the study and (2) outlines common themes that emerged from the data analysis. The report will feature cross-case findings, with relevant examples of common practices to illustrate findings, as well as descriptions of less common or outlier responses. In designing this report, the study team will focus on developing a useful product addressing multiple audiences, including NAM grant directors, federal and state policymakers planning NA/AN education initiatives, and researchers conducting studies on this topic. Technical working group members also will be invited to review and comment on the draft report, and their comments will be incorporated.

The proposed timeline for data collection and reporting activities is shown in Exhibit 5.

Exhibit 5. Timeline for Data Collection Activities and Reporting

Activity	Time Frame
Meet with Department officials	October 2015
Develop study design and data collection instruments	November 2015 to August 2016
Identify technical working group members and conduct technical working group meetings	November 2015 to July 2017
Collect and analyze site visit data	September 2016 to August 2017
Prepare preliminary findings summary and Department briefing	August to September 2017
Prepare final report and Results in Brief document	September 2017 to February 2018

A17. Display of Expiration Date for OMB Approval

All data collection instruments will display the OMB approval expiration date.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions to the certification statement identified in item 19, Certification for Paperwork Reduction Act Submissions, of OMB Form 83-I are requested.

References

ACT (2014). The condition of college and career readiness 2014. Iowa City, IA: Author.

Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative analysis on stage: Making the research process more public. Educational Researcher, 31(7), 28–38.

Argueta, R., Huff, D. J., Tingen, J., & Corn, J. O. (2011). Laptop initiatives: Summary of research across six states. Raleigh, NC: North Carolina State University, Friday Institute for Educational Innovation.

Bacon, H. L., Kidd, G. D., & Seaberg, J. J. (1982). The effectiveness of bilingual instruction with Cherokee Indian students. Journal of American Indian Education, 21(1), 34‒43.

Bebell, D., & Kay, R. (2009). Berkshire Wireless Learning Initiative: Final evaluation report. Chestnut Hill, MA: Technology and Assessment Study Collaborative.

Castagno, A. E., & Brayboy, B. M. J. (2008). Culturally responsive schooling for indigenous youth: A review of the literature. Review of Educational Research, 78(4), 941–993.

Chen, E., Heritage, M., & Lee, J. (2005). Identifying and monitoring students’ learning needs with technology. Journal of Education for Students Placed at Risk, 10(3), 309–332.

Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five designs. Thousand Oaks, CA: Sage.

Cummins, J. (1979). Linguistic interdependence and the educational development of bilingual children. Review of Educational Research, 49(2), 222–251.

Demmert, W. G., & Towner, J. C. (2003). A review of the research literature on the influences of culturally based education on the academic performance of Native American students. Portland, OR: Northwest Regional Educational Laboratory. Retrieved from http://educationnorthwest.org/sites/default/files/cbe.pdf

DeVoe, J. F., Darling-Churchill, K. E., & Snyder, T. D. (2008). Status and trends in the education of American Indians and Alaska Natives: 2008 (NCES 2008-084). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.

Dressler, C., & Kamil, M. I. (2006). First and second language literacy. In D. August & T. Shanahan (Eds.), Developing literacy in second language learners: Report of the National Literacy Panel on Language-Minority Children and Youth (pp. 197–238). Mahwah, NJ: Erlbaum.

Education Trust (2013). The state of education for native students. Washington, DC: Author.

Faria, A. M., Heppen, J., Li, Y., Stachel, S., Jones, W., et al. (2012, Summer). Charting success: Data use and student achievement in urban schools. Washington, DC: Council of the Great City Schools and American Institutes for Research. Retrieved from http://www.cgcs.org/cms/lib/DC00001581/Centricity/Domain/87/Charting_Success.pdf

Foorman, B. R., Schatschneider, C., Eakin, M. N., Fletcher, J. M., Moats, L. C., & Francis, D. J. (2006). The impact of instructional practices in grades 1 and 2 on reading and spelling achievement in high poverty schools. Contemporary Educational Psychology, 31(1), 1–29.

Francis, D. J., Lesaux, N. K., & August, D. (2006). Language of instruction. In D. L. August & T. Shanahan (Eds.), Developing literacy in a second language: Report of the National Literacy Panel (pp. 365–410). Mahwah, NJ: Erlbaum.

Freeman, C., & Fox, M. (2005). Status and trends in the education of American Indians and Alaska Natives (NCES Publication No. 2005-108). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.

Greene, J. P. (1997). A meta-analysis of the Rossell and Baker review of bilingual education research. Bilingual Research Journal, 21(2/3), 1–22.

Hirata-Edds, T. (2011). Influence of second language Cherokee immersion on children’s development of past tense in their first language, English. Language Learning, 61(3), 700‒733.

Holm, A., & Holm, W. (1995). Navajo language education: Retrospective and prospects. Bilingual Research Journal, 19(1), 141–167.

Hugo, R. (2014). Endangered languages, technology and learning: Immediate applications and long-term considerations. In M. C. Jones (Ed.), Endangered languages and new technologies (pp. 95‒112). Cambridge, U.K.: Cambridge University Press.

Jeynes, W. H. (2005). Parental involvement and student achievement: A meta-analysis (Family Involvement Research Digest). Cambridge, MA: Harvard Family Research Project.

Lane, H. B., Hudson, R. F., Leite, W. L., Kosanovich, M. L., Strout, M. T., Fenty, N. S., et al. (2009). Teacher knowledge about reading fluency and indicators of students’ fluency growth in Reading First schools. Reading and Writing Quarterly, 25, 57–86.

Mackety, D. M., & Linder-VanBerschot, J. A. (2008). Examining American Indian perspectives in the Central Region on parent involvement in children’s education (Issues & Answers Report, REL 2008–No. 059). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Central.

Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. Santa Monica, CA: RAND.

National Assessment of Educational Progress (2011). National Indian education study: Summary of national results. Retrieved from http://nces.ed.gov/nationsreportcard/nies/nies_2011/national_sum.aspx

National Congress of American Indians & National Indian Education Association (2010). National Tribal priorities for Indian education. Retrieved from http://www.niea.org/data/files/policy/ncai_niea_joint_priorities_revised_13july2010.pdf

National Congress of American Indians Policy Research Center & MSU Center for Native Health Partnerships (2012). “Walk softly and listen carefully”: Building research relationships with tribal communities. Washington, DC, & Bozeman, MT: Authors.

Nelson, S., Greenough, R., & Sage, N. (2009). Achievement gap patterns of grade 8 American Indian and Alaska Native students in reading and math (Issues & Answers Report, REL 2009 No. 073). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest.

Reyhner, J. (1989). A description of the Rock Point Community School bilingual education program. In J. Reyhner (Ed.), Effective language education practices and Native language survival (pp. 95–106). Choctaw, OK: Native American Language Issues.

Roberts, T., & Neal, H. (2004). Relationships among preschool English language learners’ oral proficiency in English, instructional experience, and literacy development. Contemporary Educational Psychology, 29(3), 283–311.

Rolstad, K., Mahoney, K., & Glass, G. (2005). The big picture: A meta-analysis of program effectiveness research on English language learners. Educational Policy, 19(4), 572–594.

Silverman, R., & Hines, S. (2009). The effects of multimedia-enhanced instruction on the vocabulary of English-language learners and non-English-language learners in pre-kindergarten through second grade. Journal of Educational Psychology, 101(2), 305–314.

Silvernail, D. L. (2005). Does Maine’s middle school laptop program improve learning? A review of evidence to date (Occasional brief). Portland, ME: Center for Education Policy, Applied Research, and Evaluation.

Spear-Swerling, L., & Brucker, P. O. (2003). Teachers’ acquisition of knowledge about English word structure. Annals of Dyslexia, 53, 72–103.

Spear-Swerling, L., & Brucker, P. O. (2004). Preparing novice teachers to develop basic reading and spelling skills in children. Annals of Dyslexia, 54, 332–364.

Tyler, J. H. (2011). If you build it will they come? Teacher use of student performance data on a web-based tool (Working paper 17486. NBER Working Paper Series). Cambridge, MA: National Bureau of Economic Research.

Valiente, O. (2010). 1–1 in education: Current practice, international comparative research evidence, and policy implications (OECD Education Working Paper No. 44). Paris, France: Organisation for Economic Co-operation and Development.

Watson-Gegeo, K. A. (1989). The Hawaiian language immersion program: Classroom discourse and children’s development of communicative competence. Paper presented at the Annual Meeting of the National Council of Teachers of English, November 18, Baltimore, MD (ED321561).

Willig, A. C. (1985). A meta-analysis of selected studies on the effectiveness of bilingual education. Review of Educational Research, 55, 269–267.

Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007–No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest.

1 There were 25 grants awarded under the NAM program in the FY 2011 and FY 2013 grant rounds, including 13 grants in FY 2011 and 12 grants in FY 2013. Two sites received grants in both rounds, and another site received two grants in 2013, for a total of 22 grantee sites. Grantees included public schools and school districts, NA/AN tribes, tribal schools and colleges, and Bureau of Indian Education schools in 10 states. The average grant award was $237,000 per year in 2011–14. See http://www2.ed.gov/programs/naancs/index.html for additional information.

2 Examples of internet searches might include demographic information about the schools or colleges in which the grantees operate; information on state or district laws or initiatives that might affect curriculum and instruction for Native American students (such as a state adding Native American education as a required subject in all schools); and any available information on federal budget cuts or appropriations in the tribal and Bureau of Indian Education schools affiliated with some of the grantees.

3 Wage information was collected from the Bureau of Labor Statistics (BLS).

6290_06/16