OMB – Education Q&As on 200710-1850-001:
The Effects of a Hybrid Secondary School Course in Algebra 1 on Teacher Practices, Classroom Quality and Adolescent Learning
Part A, p.10 – Are the weekly distance sessions for teachers structured as formal PD instruction, or are they less formal gatherings for teachers to get their questions answered? Or something else?
Response: These are formal professional development sessions. They are designed to deepen understanding and encourage application of pedagogical practices that were introduced during the summer sessions. Each session has a topic area or focus and may require a short reading or introduce an instructional tool for classroom use. The building of collaborative professional relationships among participating teachers is also encouraged through these sessions. Teachers discuss challenges and share solutions to problems they face, or things they learned as they use the hybrid approach in their classrooms. (Note: topics change monthly, and teachers are required to attend these sessions only once per month).
Part A, response 4 – What will ED use for the post-test? Will it be an existing test or is ED developing a new post-test for this study?
Response: ED will use an existing test. Our plan is to use the Educational Testing Service assessment for Algebra I.
Did ED consider whether parental consent is needed for the post-test?
Response: Yes. A summary of the protocol has been reviewed by an IRB. Parental consent is not required.
Confidentiality notice – Please add ESRA pledge language to all information collections.
Response: The following ESRA statement was added to all information collections.
Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, "Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law.”
PRA notice – Please add the PRA notice to all information collections.
Response: The following PRA statement was added to all information collections.
Paperwork Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is xxx-xxx. The time required to complete this information collection is estimated to average 12 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4700. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Sandra Garcia, The Institute for Education Sciences, U.S. Department of Education, 555 New Jersey Avenue ,N.W., Suite 506C, Washington, D.C. 20208.
Encouraging control teacher commitment
Part A, response 9 – Did ED consider using an MOU for control schools and teachers instead of a dinner requiring some travel?
Exhibit A (“Professional Development Teachers in the Control Group Can Expect”) – Does ED see any utility in describing more about what the control teachers will receive (i.e., more than just hearing about their responsibilities)?
Response: (a) Yes. This decision was carefully considered. We purposely chose dinners to give members of control schools a personal connection to the project. The dinner meeting will not only review roles and expectations for participants, but also emphasize the importance of the project and their role in it. Making this personal connection acknowledges the value we assign to their participation. Dinner locations will be chosen to minimize travel time for participants.
(b) Thank you for pointing out this inconsistency in the description of the study. We changed the language to read, “What Control Teachers Can Expect.” (Please see Exhibit A).
OMW Follow-up: Is attendance of the dinner session optional for control teachers? This should be made clear in the brochure, which currently states that teachers “will” attend the session. If a control teacher opts not to attend the session, how will the information be conveyed to them?
Response: Brochure language for “What Teachers in the Control Group can Expect” has been changed to the following:
Regionally-located sessions will be offered to teachers in the control group in the fall of school year 2008 that will provide a study overview, a description of their responsibilities during the study, and discuss opportunities for receiving the intervention at the conclusion of the study. Alternative sessions will be offered for teachers unable to attend the initial meetings.
Exhibit A (FAQs #2) – What does it mean for control sites to have “first access” to the intervention? What will the cost be of the eventual package?
Response: Annual enrollment in the program is likely to be limited due to the cost to the state of using Master teachers. In that case, KDE will give enrollment priority to the 30 control schools by having an initial registration period open only to them. Control schools will be able to enroll in the program at this point, if they wish. At the end of this initial registration period, any remaining slots will be opened by KDE to non-study schools. The cost to schools is not clear, as the Kentucky Department of Education has not priced it.
OMB Follow-up: Please use language in the brochure similar to that included in your passback response. The way it is written now, the use of the word “access” implies almost automatic ability to have the intervention
KDE will give enrollment priority to the control schools by having an initial registration period open only to them. Control schools will be able to enroll in the program at this point, if they wish. At the end of this initial registration period, any remaining slots will be opened by KDE to non-study schools.
Exhibit A – Why ask the teachers the same question as the tech coordinators about Internet access?
Response: Good point. We dropped the question from the teacher application. (Exhibit A).
Exhibit A – Did ED consider asking whether the teachers are certified to teach algebra?
Response: Teachers in KY are not specifically certified to teach Algebra. The questions: Are you certified to teach mathematics, Are you certified to teach the grade level you are assigned, and, If not, describe your certification, together cover the certification issue.
Generally, we question whether requiring so much information in the study application will depress the number of applicants. Specifically, have you considered the impacts on the response rate of:
Requiring signatures from a principal and superintendent on the same physical form?
Requiring detailed teacher information (starting with “Are you currently teaching Algebra I?”) at the application stage? Did ED consider collecting this more detailed information at the teacher survey (HTQ & CTQ) stage?
Response: We considered each of these issues carefully when developing the applications.
The advantage of having the signatures for the principal and superintendent on the same form is that it ensures a coordinated response and increases the likelihood of communication between the school and the district leaders about participation in the study. At the same time, the use of a single form does not impose much additional burden for applicants because school and district buildings are often in very close proximity in rural districts. Further, administrative staff frequently travel between school and district offices so it would be very easy to obtain both signatures. And, forms can be faxed between offices when inter-office travel is inconvenient. However, to further reduce any burden, we will offer the option to principals and superintendents to return two separate forms if that would be better for them.
(b) Including this set of questions on teacher application forms serves two important purposes, (1) establishing eligibility for participation (the teacher instructs Algebra I and has access to the required technology) and (2) providing important background information that will help master teachers tailor instruction to the needs of individual teachers and identify potential barriers to success. Neither of these goals can be achieved if the questions are placed in the teacher survey.
We agree with OMB that one must be cautious in the application process to avoid unnecessary burden that could depress participation. With that in mind, the set of teacher questions is designed to be answered quickly (most questions use a “check the box” response), adding very little time (about a minute) to the application process.
Also regarding the application process, did ED consider using an MOU instead of requiring applications?
Response: The application process serves as the MOU and contains most of the same information we would have put into an MOU.
Letter from KDE to Eligible Districts and Schools & Follow-up Email – Would ED consider adjusting references to a “federally funded” study/initiative to a “federal research study” or “Department of Education sponsored study” or something along those lines?
Response: We changed the language in the letter and e-mail to a “federal study”. (Exhibit A)
Exhibit D (HTQ & CTQ) – Would ED consider doing some additional pre-testing on the questionnaires? We have a few questions below that might be resolved through pre-testing.
Response:
The HTQ and CTQ (Exhibits D & E) were revised based on OMB recommendations listed in questions 13 a – d and field-test results.
Specific responses to questions 13 a-d are below.
The revised HTQ and CTQ were field-tested by practicing Algebra I teachers to obtain feedback regarding item clarity and understandability.
A summary of the field test follows responses to Question 13.
HTQ & CTQ – What is the baseline teachers should use of questions 5 and 11?
Response: Items 5 and 11 have been rewritten to remove the need for a baseline reference:
Item 5 - Original: Use of my district’s Algebra I curriculum increases student interest and engagement.
Item 5 - Revised: Student interest and engagement is high when I use the Hybrid Algebra I Approach [treatment]. …my district’s Algebra I curriculum [control].
Item 11 - Original: Use my district’s Algebra I curriculum increases the amount of academically focused class time.
Item 11 - Revised: The amount of academically focused class time is high when I use the Hybrid Algebra I Approach [treatment]. …my district’s Algebra I curriculum [control].
HTQ – What is the purpose of question 9? We are wondering teachers in the treatment group will provide meaningful responses.
Response: Even though treatment teachers will have agreed to implement and will have been trained to implement the “Hybrid Algebra I Approach”, routine implementation of the approach may vary across teachers. The purpose of Item 9, “I routinely use the Hybrid Algebra I Approach.” is to capture possible variations in routine use of the approach. This information will be useful in examining aspects of implementation fidelity.
HTQ & CTQ – How should teachers add up the number of PD activities for graduate coursework? One course? One project within a course?
Response: Items for the Math-Related Professional Development (PD) have been rewritten to address the concern of recording number of PD activities. See below and attached Revised HTQ
Original: Math-Related Professional Development (PD)
Indicate the number of math-related PD activities you completed during the past 12 months. |
None |
1 |
2 |
3 |
More than 3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Not at all |
Some |
A lot |
Did not complete math PD |
|
|
|
|
|
|
|
Revised: Math-Related Professional Development (PD)
To what degree has participation in the following math-related PD/graduate work (completed during the past 12 months) changed the way you teach Algebra I? |
Not at all |
Some |
A lot |
Did not complete math PD/course(s) |
Workshops |
|
|
|
|
Extended (non-graduate school) PD programs |
|
|
|
|
Graduate coursework |
|
|
|
|
HTQ & CTQ – q.31-35. Would ED consider providing more space for these responses?
Response: Additional space has been added. Please see attached HTQ and CTQ
December 2007 Field Test Results
The revised HTQ and CTQ were field tested by four practicing Algebra I teachers and the following changes have been made to the instruments to address their feedback.
1) Feedback: “By curriculum [control] or Hybrid Algebra I Approach [treatment], do you mean the textbook that the teacher uses, the scope and sequence, or something different?”
Response: The following “Note” was added to the instrument directions:
Treatment:
Note: Hybrid Algebra I Approach is defined as student use of KVHS online Algebra I student courseware for two days per week or 40% of the instructional time combined with routine teacher use of research-based strategies and resources provided in professional development.
Control:
Note: “My district’s Algebra I curriculum” is defined as the district-approved scope and sequence for Algebra I.
2) Feedback: “Question 1: This question asks about concepts. Do you also want to know about mechanics? If so, that should be another question. Otherwise, you will probably get mixed responses, some referring to the conceptual learning and some to the mechanical learning. I personally think my text does mechanics well and concepts terribly.”
Response: The following item has been added to address Algebra I mechanics:
Item 16. Use of the Hybrid Algebra I Approach is effective for helping students learn key algebraic mechanics.
3) Feedback: “Question 6: Age appropriate? As in the non-math content in examples used, or mathematically age appropriate? My immediate response to this question is “Uh, I guess so”
Response: Item 6 has been reworded to address this concern. See below.
Item 6 – Original: The KVHS Algebra I curriculum is at an age-appropriate level for most of my students.
Item 6 – Revised: The difficulty level of the KVHS Algebra I student courseware is appropriate for most of my students.
4) Feedback: “Question 7: This question uses the word meaningful, which could be interpreted to mean something different from person to person.”
Response: Item 7 has been reworded to address this concern. See below.
Item 7 – Original: I can meaningfully implement the Hybrid Algebra I Approach.
Item 7 – Revised: I can implement the Hybrid Algebra I Approach according to the recommended guidelines [as stated in the “Note” included with the Directions above.].
5) Feedback: “Question 12 and 13: How is adequate defined? Your definition may be different from the teachers’. This information might be targeted by asking how often they need to look something up or how often they don’t know the answer to a question. You could also define ‘adequate’ in parenthesis.”
Response: Items 12 and 13 have been reworded to address these concerns. See below.
Item 12 – Original: My algebra skills are adequate to conduct classes that implement the Hybrid Algebra I Approach.
Item 12 – Revised: I have the essential Algebra I knowledge and skills needed to conduct classes that implement the Hybrid Algebra I Approach.
Item 13 – Original: My technology skills are adequate to conduct classes that implement the Hybrid Algebra I Approach.
Item 13 – Revised: I have the essential technology knowledge and skills needed to conduct classes that implement the Hybrid Algebra I Approach.
| File Type | application/msword |
| Author | DLowther |
| Last Modified By | Bridget Dooling |
| File Modified | 2008-01-10 |
| File Created | 2008-01-10 |