10-11-07 memo

REL Northwest—October 11, 2007

Response to OMB Questions:

An Experimental Study of The Project CRISS Reading Program

On Ninth Grade Reading Achievement in Rural High Schools

1. What is the rationale for targeting rural schools only?

The Northwest region is a large geographic area that is sparsely populated with many small rural districts and relatively few large urban centers. Our focus on rural high schools is consistent with NWREL’s statutory responsibility to address rural education issues and the needs of rural students. Poverty levels can be high in many of our rural areas. We are focusing this study on relatively small high schools in rural areas in order to represent high-need schools in our region that often have fewer available programs and resources to help students with reading comprehension difficulties, primarily because they do not have large central offices and support personnel. In addition, we would not want to combine schools from different settings within the region (e.g., rural and urban and suburban) because of the contextual and demographic differences that could result in differential effects of Project CRISS. It is reasonable to expect that implementing Project CRISS across many teachers in a large departmentalized urban high school is more difficult than in smaller rural schools with more teacher collaboration. Resource limitations do not allow for a larger sample of schools that could be blocked by locale and compared. Therefore, our focus is only on rural schools.

2. What is the anticipated SES and racial and ethnic composition of the study population in the target schools? How will you ensure that your sample consists of the "high need schools" with limited resources, high poverty rate, and ethnic minorities?

We have collected data on the basic characteristics of our universe of schools. In terms of racial composition, rural high schools in our region are predominantly white. There are some small communities that have seen an influx of Hispanic students over the past decade, and there are pockets of Native American students on or near Indian reservations across our four-state study region. However, the actual demographic data reveal that minority students still comprise a relatively small percentage of rural high school students and tend to be concentrated in a small number of communities and high schools. Poverty is prevalent in these schools but to varying degrees. These data are summarized below.

For high schools in the region meeting our rurality and size criteria, the median school percentage of students in each race category is: Hispanic=3.5 percent; Alaskan Native or American Indian=2.0 percent; White= 89.5 percent. These are very skewed distributions. Only 7 percent of the rural high schools have sizable Native American populations and 9 percent have sizeable Hispanic populations, measured as one-quarter or more of students in the minority racial group. Poverty, on the other hand, shows a more normal distribution. Measured as the district poverty rate for school age children, the Northwest rural schools range from one percent of families with school age children in poverty to 38 percent, with mean=15.57; median=15.50; sd=6.15.

The intent of the study is to focus on all rural high schools and in our sampling process we hope to capture a range of schools defined by SES. We have a limited universe of schools (less than 200) that meet our criteria and a relatively large desired sample (66 schools). Schools must agree to all conditions before being admitted to the study. We will categorize schools that voluntarily agree to the study by State (Washington, Oregon, Idaho, Montana) and District Poverty (high, low using the median poverty rate as the cutoff), and randomly select equal numbers of treatment and control schools from each cell in the two-way stratification table. This process will ensure balanced experimental and control groups by state and poverty level.

The poverty index, with its wide and normal distribution, is an important variable to look at as an indicator of high need. Because our poverty index is taken from census data, these are communities that are likely to have fewer resources because of economic conditions. We believe that the high poverty schools will be equally if not more likely to want to participate in the study given low entering ninth grade reading skills among disadvantaged children. In any case, our sampling process will ensure balance across treatment and control groups along the poverty index. We will compare the volunteer participants to the universe of schools along our poverty index (and other demographic factors) to determine how well our volunteer sample is representative of rural and high need schools in the region.

3. What is the cost of the CRISS intervention and how likely is it that a high need school with limited resources will be able to purchase it?

Project CRISS services will be provided to each participating school as part of the study. We believe that Project CRISS is a viable model to test for rural schools because of its cost, potential for impact, and building of capacity within a small district. The extent to which a typical rural or high-need school can afford such a model is discussed below.

Based on data received from the developer, the two-year Project CRISS intervention costs are in the range of $18,000 to $25,000 per school depending on the number of faculty, with a limit of 30 teachers per school. These costs are spread out over two years to cover Project CRISS trainer time and travel, materials for teachers, consultations between trainings, and costs for a school Local Facilitator to obtain certification to become a trainer/coach for his/her school or district. We believe this is a reasonable cost and compares favorably in price to other intensive, two-year professional development programs that can serve between 10 and 30 teachers with ongoing assistance. Project CRISS also builds local capacity and continuity through the training and certification of the Local Facilitator who can continue to assist the school without the need for follow-up, for-fee Project CRISS services. Rural schools with limited resources would be able to access school improvement funds from their state departments or other school improvement grant programs to adopt Project CRISS.

 

4. Given that CRISS is a "long standing Montana based program," how will you ensure that the control schools are truly controls, especially in Montana?

Project CRISS was developed in the small district of Kalispell, Montana and is still very active in Kalispell and some of the surrounding high schools. In the recruitment and selection process, we will exclude schools that (a) are presently using Project CRISS as a school wide intervention or by significant numbers of teachers, or (b) that had CRISS training during the past five years. We have also confirmed from the Project CRISS developers that most of their work over the past five years is out of state and region in large, populous Midwestern and eastern states that in some cases have endorsed Project CRISS as a desired statewide model. While some schools may still have a few teachers who are familiar with or have been trained in Project CRISS, there is little chance they will be contaminated by doing “a little bit” of CRISS since there is a high threshold of full teacher commitment to gain participation in the study. We will also monitor implementation and professional development activities in both treatment and control schools.

 

5. The supporting statement says that you will not use statistical sampling of schools, so what procedure will you use if you have interest from more than eligible 66 schools?

Given that the total universe of schools that meet our criteria (four states, rural as defined by NCES codes, between 250-1,000 students) is just under 200 schools, we do not anticipate having this problem. We are hoping to achieve the 66 schools based on about a one-third positive response rate. Experience from other RCT studies of educational intervention indicates that this is a realistic expected response rate. We feel there will be fairly high interest because improving adolescent literacy is a strong regional need and because schools will receive the Project CRISS intervention plus a small payment for data collection services. Nevertheless, there are many reasons why schools may choose to decline, such as the presence of other reading programs or lack of teacher support. If we achieve a slightly higher than one-third participation rate, we can increase the sample size in a small increment to the extent that funds allow. In the unlikely event that we obtain a considerably higher response rate than anticipated, we will need to admit schools on a first come-first serve basis. In our recruitment, we will be dealing with all states simultaneously over a period of time to contact schools, present Project CRISS and the study, and gain support. This should reduce any bias by state or poverty level in the unlikely event that we end up with too many interested schools.

6. Is the "student outcome data" described on page 3 the state assessment test for purposes of NCLB? If not, why isn't the state assessment test being used to evaluate this program?

Because we are working across four states to achieve a sufficient overall sample size, we will not use state assessment data because each state uses a different test. In most of these states they are locally developed criterion reference tests. The four state tests do have a common achievement metric of “percent of students meeting standard” for reading/language arts in selective grades. While this is a highly relevant measure since it is the basis for measuring adequate yearly progress (AYP) under NCLB, this is not a reliable or valid measure of student reading comprehension for our purposes. Each state defines different reading standards and sets different benchmarks, some relatively “easy” and some “tough.” Further, our states typically measure reading benchmarks in tenth grade while our study focuses on ninth grade as an important transition year Therefore, we selected the Stanford Diagnostic Reading Test-4^th Edition as a common, reliable, and valid test of reading comprehension across all study schools in the ninth grade. This long-standing test focuses on basic reading comprehension and will still have relevance to helping high schools achieve NCLB goals, even if it is not the state test.  

7. Please clarify whether ED is requesting an exemption from the incentives policy. If so, please provide a rationale for the exemption, particularly with regards to the payment of a test coordinator.

We have revised Section A, Part 9 so that no payment will be necessary to schools for data collection. We will use NWREL evaluation staff plus external consultants if needed to accomplish the testing in the schools. This will ensure a higher quality and more complete administration and avoid compensating school staff for data collection, who also have a vested interest in the outcome in treatment schools. A revised Section A (see Part 9 on Payments and Gifts) and Section B (see Part 1 and the subsection on Data Collection under Part 2) are submitted to reflect this change.  

8. Please cite the statute (e.g. on consent forms, recruitment letters) when providing assurance of confidentiality.

Recruitment letters and consent forms will be so revised. Appendix K is resubmitted for the revised consent/recruitment forms.

9. Please ensure that the OMB #, expiration date, and PRA blurb are included on all instruments.

All instruments will be so revised. Appendices D-H are resubmitted with these changes.

4