Justification For Items In The Teacher, Local Facilitator And Principal Questionnaires

APPENDIX I

JUSTIFICATION FOR ITEMS IN THE TEACHER,

LOCAL FACILITATOR, AND PRINCIPAL

QUESTIONNAIRES

NWREL Experimental Study of Project CRISS

LOCAL FACILITATOR LOG QUESTIONNAIRE (APPENDIX D)

The Project CRISS Local Facilitator in the school is a key person in implementing the program. The LF provides close support to teachers in between visits by the CRISS certified trainer. Gathering the information below on a monthly basis will provide precise implementation measures within a time frame in which the LF can remember and provide reliable estimates of her/his activity.

Items 1–4 are specific behavioral items that reflect the expectations of what a LF should do to support teachers, based on the Project CRISS materials. Item 5 provides a measure of how much time the LF has spent on these activities, an important consideration for a high-quality implementation. These results can be compared to the general expectations of the developer that an LF needs about 5 hours per week or more, on average, to be an effective facilitator of the process. Items 6 and 7 check if the LF has access to the long distance support she needs from the CRISS certified trainer and how frequently they communicate with each other. Close support from a national trainer is a key implementation factor.

Finally, the Background Questionnaire provides an index of the teaching and coaching experiences of the LF, a factor that could affect the LFs ability to perform her/his expected duties. All of these behavioral and descriptive items will help describe the fidelity of implementation in sufficient detail to judge implementation success. In the event that there are no positive results on student outcomes, these data will help determine whether a program implementation failure or theory failure occurred.

TEACHER QUESTIONNAIRES (APPENDIX E, F)

In the Treatment School version, teachers are asked in item 1 if they participated in CRISS workshops as a measure of implementation fidelity. Item 2 in the Treatment version is virtually the same question as item 1 in the Control version asking about any and all professional development related to adolescent literacy that could affect the student impact measure. It is necessary to document all of the professional development related to literacy in both types of schools to adequately describe the treatment and counterfactual conditions. Even in the treatment schools, CRISS may not be the only program operating to improve reading comprehension.

Items 3–6 in the Treatment version ask about specific activities that should occur in between training events in Project CRISS schools. Asking these questions will help confirm the more detailed implementation data gathered from the LF log, and will help determine the level of active participation of the ninth-grade teachers. Parallel questions are included in the Control version as items 2–5 to determine whether or not the control schools are mimicking Project CRISS through activities of a literacy coach.

Finally, both versions end with several background characteristics (items 7–10 in the Treatment version, items 6–9 in the Control version), which are proxy measures for general teacher quality and experience. This will allow us to ensure that our random draw did not unexpectedly result in teachers with substantially different levels of education and experience in the treatment and control schools.

PRINCIPAL QUESTIONNAIRES (APPENDIX G, H)

The Project CRISS materials discuss roles for the principal in the Project CRISS intervention: (a) participate in Project CRISS training so she/he understands the underlying learning principles and student strategies in order to encourage and support teachers; and (b) use a principal “walk through” protocol that is provided in a principal guidebook and that the CRISS certified trainer demonstrates to the principal on the second visit. Items 1, 3, and 4 in the Treatment School version deal with these specific principal behaviors. For item 4, we would hope to see the principals mention the Project CRISS rubric as an index of full implementation. Item 2 asks about any further professional development activities that might help the principal be a better Project CRISS instructional leader.

In the Control School version, item 1 asks principals about any teacher professional development to determine if there is any contamination from a CRISS-like intervention (parallel to item 1 in the Treatment version). We also ask principals about their own professional development and “walk throughs” in items 2–4, parallel to the same items in the Treatment version.

Finally, items 5–9 in both versions ask basic background characteristics, which are proxy measures for principal experience and quality. This will allow us to ensure that our random draw did not unexpectedly result in principals with substantially different levels of education and experience in the treatment and control schools.

3