RELMidwest_5.1.7_TFTEfficacyStudy_Supporting Statement Part A_revised61124_CLEAN

The U.S. Department of Education (ED), through its Institute of Education Sciences (IES), requests clearance for the recruitment materials and data collection protocols under the Office of Management and Budget (OMB) clearance agreement (OMB Number 1850-0986) for activities related to the Regional Educational Laboratory (REL) Midwest Program under contract 91990022C0011.

Computational skills with fractions underpin advanced mathematics (Booth & Newton, 2012), are essential for success in high school mathematics, and are a prerequisite for college-level mathematics courses (Siegler & Lortie-Forgues, 2015). Unfortunately, student difficulty with fractions is well documented (Barbieri et al., 2020; Liu, 2018; Siegler & Lortie-Forgues, 2015). Even after studying fractions and related topics for several years, U.S. students often lack a conceptual understanding of fractions (Siegler et al., 2010). These fraction difficulties are widespread and critical to address because “early fraction knowledge strongly predicts later mathematics knowledge even after children’s IQ, reading comprehension, working memory, whole-number arithmetic knowledge, race, ethnicity, and parental education and income are statistically controlled” (Fazio et al., 2016, p. 1).

Difficulties with fractions-related content are not confined to students; teachers often have difficulties as well. Teachers often struggle with fraction computation (Harvey, 2012), and many practicing and preservice teachers have considerable difficulty with fraction operations, including multiplication and division (Tekin-Sitrava, 2020; Whitehead & Walkowiak, 2017). In a recent study, only 42 percent of prospective teachers who attempted to solve equations with fractions solved the equations correctly (Jones et al., 2020). Although teachers’ work with students is at the heart of student learning, administrators also are essential in building systemic approaches to improving teaching and learning and in providing the appropriate supports for teacher success (Park et al., 2019). Therefore, administrators need to be prepared to set standards, identify needs, and provide the appropriate supports if teachers are to be effective.

To address these needs, REL Midwest is developing a toolkit (the Teaching Fractions Toolkit) that supports teachers to enact evidence-based practices summarized in Developing Effective Fractions Instruction for Kindergarten Through 8th Grade (Siegler et al., 2010). Drawing on the recommendations and implementation steps outlined in the practice guide, the toolkit will address teacher understanding of fraction computation, rates, and ratios, as well as implications for classroom practices related to fractions content for grade 6 teachers. REL Midwest is developing the toolkit in collaboration with district partners in Illinois.

ED, in consultation with the American Institutes for Research® (AIR®), which is ED’s contractor for REL Midwest, is planning a two-part evaluation of the toolkit in 40 Illinois public schools across 6–10 school districts. The evaluation will consist of an impact study and an implementation study. OMB approval is being requested for recruitment and a multimode data collection and analysis of a group of schools, students, and staff members in these Illinois public schools.

Description of the Teaching Fractions Toolkit

Box 1: Recommendations in the Developing Effective Fractions Instruction for Kindergarten Through 8th Grade practice guide

1. Build on students’ informal understanding of sharing and proportionality to develop initial fraction concepts.

2. Help students recognize that fractions are numbers and that they expand the number system beyond whole numbers.

3. Help students understand why procedures for computations with fractions make sense.

4. Develop students’ conceptual understanding of strategies for solving ratio, rate, and proportion problems before exposing them to cross-multiplication as a procedure to use to solve such problems.

5. Professional development programs should place a high priority on improving teachers’ understanding of fractions and of how to teach them.

The Teaching Fractions Toolkit is based on and supports implementation of five evidence-based recommendations in the What Works Clearinghouse practice guide Developing Effective Fractions Instruction for Kindergarten Through 8th Grade (Siegler et al., 2010). The practice guide recommendations (see Box 1) are based on rigorous research for improving K–8 students’ understanding of fractions, with the expectation that general education teachers, mathematics specialists and coaches, special educators, and administrators will use these resources to improve their teaching of fractions.

This toolkit includes two types of supports: teacher supports and institutionalizing supports for administrators and mathematics leaders who support mathematics teachers.

The primary audience for the teacher supports is grade 6 mathematics teachers in general education classrooms. Teacher supports include six teacher professional development (PD) modules. Each professional development module consists of two synchronous sessions led by a PD facilitator, separated by approximately three hours of asynchronous assignments in the interim between sessions. In each module, teachers engage in individual and collaborative PD activities, including exploration of mathematics tasks, student work analysis, lesson planning, the use of formative assessment items, and reflection on classroom practice, all of which will support teachers’ understanding related to the implementation steps for practice guide Recommendations 2–4 as well as how to mitigate possible roadblocks identified for Recommendations 2–4. The toolkit also includes associated resources to support engagement in PD in each module, including mathematics tasks, interactive applets, protocols for student work analysis and planning, videos, student artifacts, readings, and reflection prompts. The toolkit includes a teacher reflection tool to assess initial and developing classroom practices aligned with the practice guide recommendations and questions to inform lesson planning and reflection. The teacher PD is designed so that it can be used with in-person meetings or fully online for all activities (synchronous and asynchronous). The modules and materials will be designed with flexibility so that local facilitators and teachers will be able to implement all or part of the PD in an in-person environment if they choose to do that. The guidance for facilitators will make suggestions about how to lead teacher discussions either in person, if feasible, or via videoconference using whatever videoconference platform the district employs.

The primary audience for the institutionalizing supports is administrators and mathematics leaders (principals, assistant superintendents, curriculum directors, mathematics coaches, and teacher leaders) who support teachers of mathematics. Institutionalizing supports include:

Three videos—one to introduce the toolkit and two to introduce what the practice guide recommendations look like in practice
Two leader handouts—one summarizing the practice guide recommendations and one outlining the progression of fraction content represented in the practice guide
A tool for administrators and leaders to assess district conditions to support fractions instruction
Facilitation guides for school leaders to lead professional development for grade 6 teachers

The institutionalizing supports will bolster the understanding of administrators and mathematics leaders of the importance of the mathematics content embodied in the practice guide recommendations; inform them about the research basis for teacher practices included in the recommendations; guide decisions about supporting teachers to enact the recommendations; and support leaders such as mathematics coaches or other PD providers to lead the PD that is part of the teacher supports.

All materials that users need in order to implement the teacher PD and other toolkit activities and supports are included in the toolkit and will be accessible in one central online location (https://ies.ed.gov/ncee/rel/Midwest/Toolkit) with a clear and user-friendly linked menu on the landing page. The toolkit development team will work with the IES website contractor to get the online platform ready prior to the start of the evaluation so that participating educators will be able to access all toolkit materials online. The landing page will have a brief overview of the toolkit resources, environment, and overarching goals plus sections for institutionalizing supports and teacher supports. The teacher supports section includes the six PD modules. Each module will include a participant workbook, a facilitator guide, and two slides decks (one for each synchronous meeting). Modules will be linked for easy cross-movement and include a navigation menu to the module overview, learning objectives, individual learning activities, a link to resources and tools for that module, the teacher practice monitoring tool, support tips, a glossary of terms and acronyms, and references. All resources will be navigable with a screen reader. When clicked, links will appear in a new tab or window so that the user remains connected to the module. Videos and animations will be captioned with audio available in transcripts to ensure accessibility and Section 508 compliance. Templates, checklists, and tools will be provided in HTML, PDF, and editable document formats. The modules will include links to some interactive GeoGebra applets for use by teachers and their students when working on mathematics tasks. These applets will be developed in the open-source GeoGebra website and made available to teachers through links from the Teaching Fractions Toolkit website and modules.

A1. Circumstances Necessitating the Data Collection

As part of the REL solicitation request (Solicitation #91990020R0032), IES required each applicant to develop at least one research-based toolkit to support educators’ use of evidence-based practices and to conduct an independent efficacy and implementation evaluation of the toolkit.

Per the solicitation:

IES is invested in developing practitioner-friendly toolkits to help educators use evidence-based practices in classrooms – from preschool through postsecondary settings. Some of the best evidence available is consolidated in the WWC Practice Guides, in which researchers and practitioners review the evidence from the most rigorous studies available, develop recommendations for practice, and create action steps for how to use the recommended practices. To help get this evidence into the hands of stakeholders, RELs shall partner with educators and postsecondary instructors (if relevant) to develop one toolkit based on an assigned WWC Practice Guide, which shall include all materials necessary for effective implementation.

The toolkit contains the following two components: (1) institutionalizing supports for administrators and mathematics leaders who support mathematics teachers and (2) teacher professional development modules and associated resources that build teachers’ knowledge and practices, including diagnostic and monitoring tools—formative assessment probes for use with students and a teacher practice monitoring tool for reflecting on practice. The solicitation also states that RELs must evaluate the efficacy and implementation of the professional development resources in the finished toolkit. According to the solicitation, “(t)he evaluation shall examine changes in teacher practice and may also include measures of teacher knowledge and/or teacher self-efficacy.”

The purpose of this data collection will be to measure the efficacy and implementation of the REL Midwest-developed toolkit designed to improve teacher self-efficacy and practices for fraction computation and rate and ratio instruction, as well as student learning outcomes in grade 6 mathematics. The study will address the following research questions (RQs):

What is the impact of the toolkit on grade 6 teachers’ self-efficacy and teaching practices for fraction computation and rate and ratio instruction compared to the business-as-usual condition?
What is the impact of the toolkit on grade 6 students’ performance in solving fraction computation and rate and ratio problems compared to the business-as-usual condition?
How did the professional development supports and resources available to grade 6 math teachers differ in treatment and control schools?
To what extent is the toolkit implemented with fidelity within each participating school and overall across all participating schools?
To what extent is the fidelity of implementation associated with teachers’ self-efficacy and practices and students’ performance on solving fraction computation and rate and ratio problems?
What contextual factors support or hinder the adoption and implementation of the toolkit?
To what extent do the participating teachers and school leaders perceive the toolkit as usable, useful, and feasible to implement? What aspects of the toolkit do they perceive could be improved?

The toolkit evaluation will produce a report for district and school leaders who are considering strategies to improve fraction teaching and learning in grade 6. The report will be designed to help them decide whether and how to use the toolkit to help them implement the practice guide recommendations. The findings from the evaluation also will inform further refinement of the toolkit.

A2. Purpose and Use of the Data

Data collected for this evaluation will be used to examine the implementation of the toolkit in participating schools and the toolkit’s efficacy in improving teacher self-efficacy and practices for fraction computation and rate and ratio instruction, as well as student learning outcomes in grade 6 mathematics.

REL Midwest will conduct the evaluation during the 2024/25 school year. This study will not be replicated under this contract, as it ends in 2027; in other words, this is not a recurring data collection. The evaluation team aims to recruit 40 schools in Illinois so that the study will be powered to detect effects of the toolkit on student learning and teacher practice outcomes that are of statistical and practical significance and are comparable in magnitude to those effects reported in previous studies of similar interventions. The evaluation will employ an experimental design in which schools that have expressed interest in using the toolkit and have agreed to participate in the evaluation will be randomly assigned within blocks to treatment condition (toolkit) or business as usual (control) in summer 2024. Each district with multiple schools participating in the study will serve as its own randomization block. Schools from districts in which only one school is participating in the study will be grouped into blocks based on school locale and prior-year school performance. In schools assigned to the toolkit group, grade 6 teachers and their administrators will be invited to use the toolkit materials with the guidance of a local facilitator. In control schools, grade 6 teachers and administrators will not have access to the toolkit until after the study. Both groups will be asked to participate in study data collection, which will include teacher surveys, teacher classroom observations, teacher interviews, school and district leader and facilitator interviews, and an assessment of students’ abilities to solve fraction computation and rate and ratio problems. The study will also collect administrative data, including background information about students and teachers and student scores from state math assessment in prior year (when students were in grade 5).

including the data source, key measures, type of data collection (primary or secondary), respondent, condition (treatment or control group), time of data collection, and RQs addressed.

Table 1. Overview of data sources, measures, and RQs

Data source	Measure	Data collection type	Target sample	Condition	Dates for acquiring data	RQ
Teacher surveys	Teacher self-efficacy; teacher experience with implementation	Primary	120 teachers	Treatment and control	September 2024; May 2025	RQ1, RQ3, RQ5, RQ6, RQ7
Classroom observations	Teacher practice	Primary	120 teachers	Treatment and control	January–May 2025	RQ1, RQ5
Student assessment	Student ability to solve fraction computation and rate and ratio problems	Primary	2,400 students	Treatment and control	May 2025	RQ2, RQ5
PD implementation checklist	Implementation fidelity	Secondary (implementation team)	8 facilitators	Treatment	May 2025	RQ4, RQ5
Administrator implementation checklist	Implementation fidelity	Primary	20 school administrators (principals or designees)	Treatment	May 2025	RQ4, RQ5
PD records and artifacts	Implementation fidelity	Secondary (implementation team)	60 teachers; 8 facilitators	Treatment	May 2025	RQ4, RQ5
Teacher interviews	Teacher experience with implementation	Primary	12 teachers	Treatment	April–May 2025	RQ6, RQ7
School leader and facilitator interviews	School leader and facilitator experience with implementation	Primary	6 leaders (facilitator or principal)	Treatment	April–May 2025	RQ6, RQ7
District administrative data	Student characteristics, student state assessment scores from prior year (2023/24), teacher characteristics, and student–teacher links (rosters)	Secondary (district)	10 districts	Treatment and control	Fall 2024	RQ1, RQ2, RQ5
School characteristics	School characteristics	Secondary (publicly available)	40 schools	Treatment and control	January 2024	RQ1, RQ2, RQ5

Note: PD = professional development.

REL Midwest will safeguard all data collected for this evaluation through protocols approved by the contractor’s federally approved Institutional Review Board, including adherence to Family Educational Rights and Privacy Act (FERPA) regulations. REL Midwest will analyze the data collected through this evaluation using statistical models that are preapproved by IES via an independent peer review process. The contractor (AIR) will then summarize the impact and implementation findings in a report, and that report will undergo review for quality and relevance by IES’s external review contractor. Once the report has undergone IES review, findings will be published through IES for educators and education researchers alike.

Finally, once data are analyzed and summarized, REL Midwest will sanitize the data files of any information that can be linked to students, teachers, schools, or districts. Data files will then be submitted to IES and made available to other researchers as restricted-use files.

Data Collection Activities for Which Clearance Is Requested as Part of This Package

We describe below the purposes for each recruitment and data collection activity for which OMB approval is being sought. These activities involve burden on respondents and are therefore the basis for seeking clearance.

District and school recruitment. The evaluation team will first identify Illinois districts with at least one school serving grade 6 students and will collect publicly available data on relevant district and school characteristics (e.g., school locale, enrollment, student demographic, pupil–teacher ratio, financial information such as expenditure per student) from the Illinois State Board of Education (ISBE) website and from the Common Core of Data from the National Center for Education Statistics (NCES). The evaluation team will then work with partners at ISBE and leverage existing relationships with Illinois districts to help widely distribute information about the study to districts across the state (see appendix A1). Districts that are interested in participating in the study will be asked to complete an online form to indicate their interest and provide information to the evaluation team to help determine their eligibility for the study (see appendix A2). Districts will be eligible to participate if they serve students in grade 6, are willing to participate in a randomized controlled trial with delayed implementation for control schools and are not already providing professional development in grade 6 math instruction that is of the same type and level of intensity as that being provided by the toolkit. The evaluation team will schedule initial virtual informational meetings with districts that have expressed interest to confirm their interest and eligibility and to answer any questions district leaders may have. At this meeting, the evaluation team will inform district leaders about the roles, responsibilities, and benefits of the study. If district leaders are interested in participating, the evaluation team will ask for their help contacting schools and their ideas for how the study might be a fit for their schools. The team will follow up with one-on-one meetings with school leaders, if requested, to answer questions and confirm their interest. Researchers on the team will ask school districts to sign a memorandum of understanding, indicating that they understand the intervention and the study and that schools will participate in the study regardless of the condition to which they are assigned. The evaluation team will obtain approval from the Institutional Review Board at AIR and research approvals or data agreements from participating districts for the evaluation activities.

Teacher recruitment. Upon district agreement, the team will reach out to school principals and offer to schedule a school-specific information meeting to provide information directly to teachers and to hear their thoughts. Teachers who teach at least one regular grade 6 math class will be eligible to participate in the study. For the proposed evaluation, a regular grade 6 math class refers to a class that is designated by the school as a general education class and that teaches the district’s middle-track grade 6 math curriculum. This definition excludes advanced classes, such as gifted and talented programs and accelerated classes, as well as remedial classes and self-contained special education classes. The evaluation team will prepare a study information sheet for teachers and distribute it to teachers before the meeting (see appendix A3). The team will remain flexible and adaptive in the face of emerging recruitment experiences (e.g., by extending the information session to address any immediate concerns of teachers). The evaluation team will collect consent forms from all eligible teachers in participating schools in late summer 2024, after randomization of schools and prior to the start of the 2024/25 school year (see appendix A4 for a copy of the consent form). Only those teachers who have consented will participate in data collection for the evaluation.

Teacher surveys. To assess the impact of the toolkit on teachers’ self-efficacy for fractions instruction, the evaluation team will administer an online survey to participating teachers in treatment and control schools in fall of 2024 (baseline survey) and spring 2025 (post survey). The team will use measures with established reliability and validity, including a measure of self-efficacy related to using and teaching with visual representations in math (DePiper et al., 2019), a measure of self-efficacy for pedagogy in math (McGee & Wang, 2014), and a measure of self-efficacy for teaching math content (McGee & Wang, 2014). The visual representations self-efficacy measure (α = 0.96), developed and validated by researchers at Education Development Center (DePiper et al., 2019), includes 10 items that ask teachers to indicate how confident they are in using visual representations in math and in teaching with visual representations in math. The self-efficacy for pedagogy in math measure (α = 0.86), developed and validated by McGee and Wang (2014), contains 7 items that ask teachers to rate their level of self-efficacy for key pedagogical strategies and practices (for example, motivating students, using a variety of assessments, implementing alternative teaching strategies). The self-efficacy for teaching math content measure (α = 0.86), also developed and validated by McGee and Wang (2014), contains 15 items that ask teachers to rate their level of self-efficacy for teaching math content specific to elementary school. With input from math content experts, the evaluation team made changes to the wording of 6 items and replaced another 2 items to make the measure align better with grade 6 math content. In addition to the teacher self-efficacy measures, the teacher survey will include items to collect information about professional development trainings that teachers attend during the 2023/24 and 2024/25 school years, implementation of the toolkit (for treatment teachers only), and teacher background characteristics. The survey draws items from the PD characteristics scales used in the analysis of service contrast in the Middle School Mathematics Professional Development Study by Garet et al. (2010; α = 0.69–0.90) and the 2015 Grade 8 NAEP Mathematics Teacher Questionnaire. We expect that each teacher survey will take 20 to 30 minutes to complete (see appendix B for a copy of the teacher survey questionnaire and appendix C for sample email invitation and follow-up emails).

Interviews with teachers. The evaluation team also will conduct virtual interviews with a purposeful sample of 12 teachers in spring 2025. The evaluation team will first identify six schools in which to conduct interviews. Schools will be selected based on project staff initial rating of teacher uptake and engagement—ratings will include three levels: low, average, and high teacher uptake and engagement. Interview schools will be selected randomly by these rating levels. In each school, two teachers, randomly selected from a list of willing teachers, will be interviewed. Interviews will be recorded and transcribed with permission. Each interview will take about 45 minutes to complete (see appendix D for a copy of the teacher interview protocol).

Administrator implementation checklist. To assess implementation fidelity, the evaluation team will ask administrators in treatment schools to complete a short online checklist in spring 2025 that asks about their use of institutionalizing supports. The checklist should take no more than 30 minutes to complete (a copy of the checklist is provided in appendix E).

District administrative data. The evaluation team will collect district administrative records regarding teacher and student background characteristics to describe the study sample, to test baseline equivalence, and to include as covariates in the impact analysis. Teacher background characteristics will include teacher total years of teaching experience, highest degree, and certification status. The evaluation team will also request teacher email addresses in order to email teachers the invitation to complete the surveys and to schedule interviews and classroom observations. Student background characteristics will include student characteristics (race/ethnicity, gender, multilingual learner student status, special education status, and eligibility for the National School Lunch Program) and student scores on the state math assessment from spring 2024 (when students were in grade 5). Masked student identifiers will be requested to allow the evaluation team to link administrative data over time and across multiple district sources. Student–teacher links (classroom rosters) will be requested. District staff will submit information electronically using secure file transfer procedures. We expect it will take each district a total of 20 hours of staff time to prepare and submit the data.

The following activities will not involve burden on respondents or are not subject to Paperwork Reduction Act per OMB guidance (5 C.F.R. 1320.3(h)). They are listed here to provide a complete picture of the study.

Classroom observations. To examine the toolkit’s impact on classroom instruction, the evaluation team will conduct in-person classroom observations in treatment and control schools in winter/spring 2025, using a protocol adapted from the classroom observation protocol used in the Middle School Mathematics Professional Development Impact Study sponsored by IES (Garet et al., 2010). The observation will focus on three validated classroom practice measures from the Garet et al. protocol that are related to the goals of the toolkit and that align with recommendations in the WWC fraction practice guide, including: Teacher focuses on mathematical reasoning (α = 0.62), teacher elicits student thinking (α = 0.70), and teacher uses representations (α = 0.81). Observations will be conducted by the evaluation team members from AIR, ED’s contractor for REL Midwest, who are trained on the observation instrument. Each teacher will be observed for a single class period. OMB approval is not required for in-person classroom observations because facts or opinions obtained through direct observation by an employee or agent of a federal agency is not to subject to the Paperwork Reduction Act per OMB guidance (5 C.F.R. 1320.3(h)(3)).

Student assessment. The evaluation team will administer a 30-minute assessment to examine the impact of the toolkit on students’ abilities to solve fraction computation and rate and ratio problems in spring 2025. The evaluation team has constructed a customized test by drawing on items from existing state standardized tests (released items or practice test) that correspond to the standards of focus in the toolkit and that measure students’ uses of problem-solving strategies emphasized in the WWC practice guide. State assessment items have undergone analysis for validity and reliability, as well as review to remove bias, ensuring item functioning. The assessment will be administered online, but the evaluation team will support paper-and-pencil administration if a participating school is not able to administer the online assessment. The evaluation team will work with schools to offer appropriate accommodations that students typically use during classroom instruction and classroom assessments. OMB approval is not sought for the student assessment because examinations designed to test aptitude, abilities, or knowledge is not to subject to the Paperwork Reduction Act per OMB guidance (5 C.F.R. 1320.3(h)(7)).

Professional development records and artifacts. The evaluation team will collect training records (e.g., teacher professional development session attendance records) and artifacts (e.g., teacher reflection plans) from facilitators and teachers in treatment schools. These data provide evidence that toolkit activities were completed as intended and complement the toolkit platform statistics. OMB approval is not sought for collection of professional development records and artifacts because teachers and facilitators will be completing and submitting these materials to the implementation team during the implementation of the professional development. The implementation team will transfer these records to the evaluation team for analysis.

Facilitator implementation checklist. To assess implementation fidelity, the implementation team will create an implementation checklist that identifies key activities for each synchronous session and will ask the facilitator to complete the checklist for each session, as well as identify toolkit resources used for planning and implementing the professional development. OMB approval is not sought for this data collection activity because facilitators will be completing and submitting the checklist to the implementation team during and as part of the implementation of the professional development. The implementation team will transfer the completed checklists to the evaluation team for analysis.

Interviews with school leader and facilitator. The evaluation team will conduct interviews with a purposeful sample of six school leaders, including facilitators, to gather data on their experiences with the implementation of the toolkit. Interviews will be recorded and transcribed with permission. Because interviews will be conducted with fewer than 10 individuals, OMB approval is not sought for this data collection activity.

Data on school characteristics. The evaluation team will collect data on school characteristics—including school size, locale, student racial and ethnic composition, school financial data, and prior school performance—from the ISBE website and the Common Core of Data. OMB clearance is not requested for this data collection activity because the evaluation team will collect these data from publicly available sources.

A3. Use of Technology to Reduce Burden

The data collection plan was designed to obtain reliable information in an efficient way that minimizes respondent burden, and technology will be used to reduce burden for all the data collections for which we are seeking clearance.

Teacher surveys will be administered online using an online platform that will allow respondents to complete the survey at a time and place that are convenient for each respondent. The evaluation team will email to study participants a link to the online surveys. To reduce the burden on respondents, the software is flexible and allows survey respondents to participate using a multitude of devices, such as computers and Smart phones, and to switch between devices while completing the survey.

Administrators also will complete the implementation checklist through a secure online platform that allows them to complete the checklist at a time and place that are convenient for each administrator.

The teacher survey, administrator implementation checklist, and student assessment will be created and administered using Jotform integrated with an Airtable base. Forms created with Jotform are Level A and Level AA compliant with Web Content Accessibility Guidelines (WCAG) 2.1 standards. The Form Accessibility setting of the Jotform allows users to create Section 508 compliant forms. Documentation of Jotform’ accessibility can be found at this link: https://cdn.jotfor.ms/assets/pdf/JotForm_VPAT_v.1.0.pdf. REL Midwest will use no question types that are not 508 compliant. IES will confirm that all electronic data collection materials being used in the study are 508 compliant before they are administered to participants.

Teacher interviews will be conducted virtually (via phone or an online platform such as Microsoft Teams meetings) to reduce the writing burden for teachers. This mode of data collection is appropriate for the conversational exchange necessary to obtain answers to the open-ended questions and allows probing for more detail than a self-administered survey can provide.

For the district administrative data, we will reduce burden by gathering the data electronically rather than in hard copy. We will provide clear instructions regarding the data requested and methods of transmitting the data securely.

The evaluation team will record participant interviews (with permission) so that participants’ responses can be accurately transcribed and analyzed with a focus on efficiency (recording to be destroyed after transcription).

A telephone number to a staffed help desk and an e-mail address will be available during the data collection process to permit respondents to contact REL Midwest with questions or requests for assistance. These procedures are designed to minimize the survey burden on respondents.

A4. Efforts to Avoid Duplication of Effort

Throughout the evaluation, efforts will be made to minimize and reduce the burden on respondents. Wherever possible, the evaluation team will rely on secondary data sources to reduce burden on district and school personnel. By collecting administrative records on teachers, the evaluation team will eliminate the need for several items on the teacher survey and thereby avoid redundancy. The evaluation team will collect school-level characteristics, such as size, locale, and student composition, from publicly available sources and thereby reduce the burden on district and school personnel. The evaluation team will use data that are collected by the implementation team during the professional development to measure fidelity of implementation, thus reducing the burden on teachers and facilitators. The primary data collection that is part of this study includes only information that is not available from other sources.

A5. Methods to Minimizing Burden on Small Entities

The evaluation team will aim to recruit districts from diverse settings in terms of geographic locale and district size. We expect one or more of the school districts that decide to participate in the evaluation will be small (i.e., have a population of fewer than 50,000 students). The evaluation team has developed the data collection plan based on this assumption.

The use of administrative records will reduce the burden on school educators by ensuring that only the minimum amount of original data is requested from districts to meet the objectives of this study. To avoid placing extra burden associated with travel on small entities, the implementation events are conducted separately in each district. The evaluation team will work closely with districts to ensure the most efficient processes are established for data collection to minimize burden on the part of district and school personnel. Further, whenever possible, data will be collected through electronic means—online—to reduce the length of time it takes respondents to respond.

A6. Consequences of Not Collecting Data

The Education Sciences Reform Act of 2002 (ESRA) states that the central mission and primary function of the RELs is to support applied research and provide technical assistance to state and local education agencies within their region (ESRA, Part D, section 174[f]). If the proposed data were not collected, REL Midwest would not be fulfilling its central mission to serve the states in the region and provide support for evidence-based research. The systematic collection and analysis of the data described above is required to accomplish the goals of the research project approved by IES. Participation in all data collection activities is voluntary. Information for site recruitment will be collected using the process described in response to question A2. This is a one-time study (i.e., not recurring); therefore, periodicity is not addressed.

A7. Special Circumstances

There are no special circumstances involved with this data collection. Data collected will be conducted in a manner consistent with the guidelines in 5 CFR 1320.5.

A8. Federal Register Announcement and Consultations Outside the Agency

A 60-day Federal Register Notice was published on [DATE TO BE COMPLETED BY IES]. A 30-day notice was published on [DATE TO BE COMPLETED BY IES].

ED has consulted with the following groups regarding the availability of data, the soundness of the evaluation design for addressing evaluation questions, and the clarity of measures:

The REL Midwest contractor that proposed this evaluation based on the needs identified within the Midwest region.
A subject matter expert Dr. Joshua Polanin, who is a principal researcher at AIR and an expert in research design and evaluation, has provided feedback on the evaluation methodology and outcome measures.
The REL Midwest contractor has consulted with former educators within its organization about teacher surveys and interviews. These former educators reviewed the survey and interview questions for clarity of wording, for “loadedness” of questions (i.e., whether questions are written to elicit only one type of response), and appropriateness of response options.
ED also has contracted with another organization to review technical aspects of project plans and reports submitted by REL contractors. This external peer review contractor examines the rigor of the evaluation design, the analytic approach for determining impact and fidelity of implementation, and the degree to which findings address the evaluation questions and conclusions are supported by the data. The evaluation plan has been reviewed by the external review contractor and was approved by IES in March 2023.
The REL Midwest contractor has submitted the study for review by AIR’s Institutional Review Board (IRB) and the study is anticipated to be approved by IRB in June 2023.

A9. Payments or Gifts

The study team has proposed incentives for the surveys and interviews to partially offset respondent time and effort in completing the survey or interview. Incentives are proposed because high response rates are needed to make the survey findings reliable, and we are aware that teachers are the targets of numerous requests to complete surveys on a wide variety of topics from state and district offices, independent researchers, and ED.

The evaluation team proposes to provide teachers a $50 gift certificate and administrators a $50 gift certificate upon completion of the interview. These amounts were determined based on average hourly wages for teachers and administrators, which were calculated based on the most current estimates of yearly salaries (currently from 2021) in the Bureau of Labor Statistics (BLS) Occupational Outlook Handbook. Per IES guidance, “the minimum incentive, no matter how much time the research activity requires, is one full hour of wages”. Teachers completing the survey will receive a gift card of $50 for completing each of two 30-minute surveys (baseline survey before the intervention and follow-up survey after the intervention) for a total of $60 per person. Incentives will be distributed electronically (i.e., a link to a gift card) after respondents complete the data collection instruments.

No incentives will be given to principals for completing the administrator implementation checklist or to district staff for completing the collection of district administrative records.

A10. Assurances of Confidentiality

AIR, ED’s contractor for REL Midwest, will be following the policies and procedures required by ESRA, Title I, Part E, Section 183, which requires “All collection, maintenance, use, and wide dissemination of data by the Institute” to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, FERPA, and the Protection of Pupil Rights Amendment.

In addition, for student information, ESRA states “The Director shall ensure that all individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act.”

Subsection (c) of section 183 referenced above requires the Director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.”

Subsection (d) of section 183 prohibits disclosure of individually identifiable information as well as making the publishing or communicating of individually identifiable information by employees or staff a felony.

AIR, ED’s contractor for REL Midwest, will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information about respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team. All members of the study team have obtained their certification on the use of human subjects in research; REL Midwest staff have also obtained federal security clearances. All institution-level identifiable information will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required.

REL Midwest will request unique identifiers (IDs) for each student, as well IDs for the mathematics teacher(s) associated with each student. To minimize access to personally identifiable information, REL Midwest will request that districts send a separate crosswalk file that links teacher names to IDs. Both the implementation and the evaluation teams will maintain data security by allowing access to shared folders only to team members who need it. REL Midwest also will create a process whereby the project director must approve all requests for access to these folders (and maintain a list of individuals who have access).

In addition to these safeguards, AIR, ED’s contractor for REL Midwest, routinely employs the following to carry out privacy assurances with respect to study data:

All AIR employees sign a privacy pledge emphasizing its importance and describing their obligation.
Identifying information is maintained on separate forms and files, which are linked only by sample identification number.
Access to hard-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.
Computer data files are protected with passwords, and access is limited to specific users.
Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use.

A11. Justification for Sensitive Questions

There are no personally sensitive questions in this data collection. Teachers completing the survey will be asked questions to measure their self-efficacy in teaching math to students in grade 6, about the type of professional development activities they are enrolled in, their opinions about the quality of the professional learnings offered by the toolkit, and information about their academic and teaching background characteristics (years of experience, certification, undergraduate and graduate major). REL Midwest will not collect data on teacher gender, race, or ethnicity, either through survey or in collection of administrative data. Teachers completing the interview will be asked about their experiences with implementing the toolkit and their opinions about the toolkit. School administrators completing the checklist will be asked about their use of institutionalizing supports during implementation of the toolkit.

A12. Estimates of Hours Burden

For this clearance package, the evaluation team has calculated hours of burden for three components: recruitment activities, primary data collected from study participants, and extant data provided by the districts. The evaluation team has calculated hours of burden for two recruitment activities: districts filling in an interest and eligibility form and teachers reviewing and completing a teacher consent form. The evaluation team has calculated hours of burden for six primary data collection activities: a teacher survey, classroom observations, a student assessment, an administrator implementation checklist, interviews with teachers, and interviews with school leaders. Table 2 shows the hourly burden overall and for each activity. The total burden associated with activities included in this clearance package is 374 hours, with an annualized burden of 187 hours over two years. Based on average hourly wages for participants, the total burden amounts to an estimated total monetary cost of $11,485 and an annualized cost of $5,743. The annualized number of responses is 206, for a total of 412 response across two years.

Table 2. Estimated annual burden and respondent costs

Recruitment and data collection activity	Target sample Size	Estimated response Rate	Number of respondents	Number of administrations	Total number of responses	Average burden hours per response	Total burden hours	Estimated respondent average hourly wage	Estimated monetary cost of burden
Recruitment
District recruitment (completing the eligibility form)	50	20%	10	1	10	0.5	5	$50	$250
Teacher recruitment (completing the consent form)	158	85%	134	1	134	0.25	33.5	$30	$1,005
Primary data collection
Teacher survey	134	85%	114	2	228	0.5	14	$30	$3,420
Classroom observations*	134	85%	114	1	114	0	0	$0	$0
Student assessment*	2,400	85%	2,040	1	2,040	0	0	$0	$0
Administrator implementation checklist	20	90%	18	1	18	0.5	9	$50	$450
Interviews–teachers	12	100%	12	1	12	1	12	$30	$360
Interviews–school leaders*	6	100%	6	1	6	1	6	$50	$300
District administrative data collection
Administrative data on teachers and students	10	100%	10	1	10	20	200	$30	$6,000
Totals			298		412		374		$11,485
Annualized totals			149		206		187		$5,743

* Indicates activities for which OMB clearance is not being sought and that are not included in the estimated total burden hours or estimated monetary cost.

A13. Estimates of Cost Burden to Respondents

There are no additional respondent costs associated with the data collection for this study other than the hour burden accounted for in item 12. The collection of qualitative and quantitative data proposed in this package is a one-time series of data collection activities. There are no plans for follow-up studies or other recurring data collections outside of what is being proposed in this package.

A14. Annualized Cost to the Federal Government

The total cost to the federal government for developing, fielding, and analyzing the evaluation over all five years is approximately $1,185,000, and the estimated annualized cost to the federal government for each year of the study is approximately $237,000 (over five years).

Funding includes staff time for REL Midwest staff to recruit participants and collect, clean, and analyze data from the study. Also included are costs incurred by REL Midwest staff related to study preparation and submission of the study information to IES (from proposed research design through reporting of results).

A15. Reasons for Program Changes and Adjustments

This is a new study.

A16. Plans for Tabulation and Publication of Results

All results for REL rigorous studies will be made available to the public through peer-reviewed evaluation reports that are published by IES.

The analyses will be carried out using hierarchical linear modeling, where appropriate, to take into account nesting (e.g., the nesting of students and teachers within schools) and will incorporate covariates measured at baseline to maximize precision. To avoid potential selection bias, the impact analyses will employ an intent-to-treat approach, in which all students and teachers in all randomly assigned schools during the 2024/25 school year are included in the analyses, whether or not the teachers actually participated in the toolkit professional development or participated to the full extent expected. This approach is explained further in Part B.

Findings that are answers to the research questions will be presented regardless of whether they are statistically significant. The main report will focus on findings about the student achievement measure and the domain averages for teacher self-efficacy and teacher practice and on findings about service contrast. Findings from the exploratory analyses (analyses by individual characteristics, analysis of individual measures within the teacher self-efficacy and teacher practice domains) will be presented in appendixes. The main report will describe key findings with text and a simple table or a graph (e.g., a bar chart). The full detailed findings will appear as tables in an appendix.

No responses or data will be reported for individual staff members, students, or schools. Reported data will contain no fewer than four cases per reported table cell to protect confidentiality and mask individually identifiable data.

After the study report is finalized, the evaluation team will prepare restricted-use data files in accordance with NCES standards. These files will contain all the primary survey data collected for the study with all personal identifiers removed. Thorough documentation will be provided for each data file, including a detailed codebook and explanations of the unit of observation, weights, and methods for handling missing data. All restricted use files are required to be reviewed by IES’ Disclosure Review Board. The Disclosure Review Board (DRB) comprised of members from each NCES Division, representatives from IES’ Statistical Standards Program, and a member from each of the Institute of Education Sciences (IES) Centers. The DRB will review disclosure risk analyses conducted by the REL contractor to ensure that data released do not disclose the identity of any individual respondent. The DRB approves the procedures used to remove direct identifiers from restricted-use data files. IES restricted-use data sets require a user’s license that is applied for through the same process as NCES restricted-use data sets. Even the evaluation team would be required to obtain a restricted-use license to conduct any work with the data beyond the original evaluation.

District administrative data collected in this evaluation, however, will not be included in a restricted-use data file. Instead, the study team will prepare an alternate file, with the relevant documentation needed to replicate the analysis or answer additional research questions in the event that our district partners were to make administrative data available for such an analysis. In a .txt format, the alternate file will describe the data requested from participating districts and instructions on how to obtain those data, as well as the steps the study team took to merge and clean the data, create variables, and run the analyses.

The timeline for the activities in this project, including data collection, analyses, and reporting are in Table 3.

Table 3. Project Timeline

Activity/milestone	01	02	03	04	05	06	07	08	09	10	11	12
	2022
Submit evaluation proposal.											l
Submit data management plan.											l
Develop and refine evaluation instruments.			l	l	l	l	l	l	l	l	l	l
	2023
Develop and refine evaluation instruments (continued).	l	l	l
Receive notice of institutional review board approval or exemption.			l
Submit Office of Management and Budget clearance package (after first round of REL peer review of the evaluation proposal is complete).				l
Recruit districts, schools, and teachers.											l	l
	2024
Recruit districts, schools, and teachers (continued).	l	l	l	l	l
Establish data-sharing agreement.					l	l
Implement intervention.									l	l	l	l
Collect data.									l	l	l	l
	2025
Implement intervention (continued).	l	l	l
Collect data (continued).	l	l	l	l	l
Analyze data.					l	l	l	l
Draft report.							l	l	l
Submit summative report.									l
Implement intervention (delayed treatment).									l	l	l	l
	2026
Implement intervention (delayed treatment; continued).	l	l	l

A17. Approval not to Display the Expiration Date for OMB Approval

The Institute of Education Sciences is not requesting a waiver for the display of the OMB approval number and expiration date.

A18. Exception to the Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act
(5 CFR 1320.9).

References

Barbieri, C. A., Rodrigues, J., Dyson, N., & Jordan, N. C. (2020). Improving fraction understanding in sixth graders with mathematics difficulties: Effects of a number line approach combined with cognitive learning strategies. Journal of Educational Psychology, 112(3), 628–648. http://eric.ed.gov/?ID=EJ1247111

Booth, J. L., & Newton, K. J. (2012). Fractions: Could they really be the gatekeeper’s doorman? Contemporary Educational Psychology, 37(4), 247–253. http://eric.ed.gov/?ID=EJ977998

DePiper, J. N., Nikula, J., & Louie, J. (2019). Shifts in self-efficacy for teaching English learners: Emergent findings from mathematics teacher professional development. In S. Otten, A. G. Candela, Z. de Araujo, C. Haines, & C. Munter (Eds.), Proceedings of the forty-first annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (pp. 547–551). University of Missouri.Fazio, L. K., Kennedy, C. A., & Siegler, R. S. (2016). Improving children’s knowledge of fraction magnitudes. PLoS ONE, 11(10), Article e0165243. https://doi.org/10.1371/journal.pone.0165243

Garet, M. S., Wayne, A., Stancavage, F., Taylor, J., Walters, K., Song, M., Brown, S., Hurlburt, S., Zhu, P., Sepanik, S., & Doolittle, F. (2010). Middle school mathematics professional development impact study: Findings after the first year of implementation (NCEE 2010-4009). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://eric.ed.gov/?id=ED509306 Harvey, R. (2012). Stretching student teachers’ understanding of fractions. Mathematics Education Research Journal, 24, 493–511. https://eric.ed.gov/?id=EJ984997

Jones, D. L., Zientek, L. R., Sharon, V. V., & Swarthout, M. B. (2020). Solving equations with fractions: An analysis of prospective teachers’ solution pathways and errors. School Science & Mathematics, 120(4), 232–243. https://doi.org/10.1111/ssm.12402Liu, Y. (2018). Fraction magnitude understanding and its unique role in predicting general mathematics achievement at two early stages of fraction instruction. British Journal of Educational Psychology, 88(3), 345–362. https://doi.org/10.1111/bjep.12182

McGee, J. R., & Wang, C. (2014). Validity-supporting evidence of the Self-efficacy for Teaching Mathematics Instrument. Journal of Psychoeducational Assessment, 32(5), 390–403. https://eric.ed.gov/?ID=EJ1030705 Park, J. H., Lee, I. H., & Cooc, N. (2019). The role of school-level mechanisms: How principal support, professional learning communities, collective responsibility, and group-level teacher expectations affect student achievement. Educational Administration Quarterly, 55(5), 742–780. https://eric.ed.gov/?id=EJ1232698

National Assessment of Educational Progress. (2015). Mathematics Teacher Questionnaire: 2015 Grade 8. https://nces.ed.gov/nationsreportcard/subject/about/pdf/bgq/teacher/2015_bq_teacher_g08_m.pdf

Siegler, R., Carpenter, T., Fennell, F., Geary, D., Lewis, J., Okamoto, Y., Thompson, L., & Wray, J. (2010). Developing effective fractions instruction for kindergarten through 8th grade: A practice guide (NCEE 2010-4039). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://eric.ed.gov/?id=ED512043

Siegler, S. F., & Lortie-Forgues, H. (2015). Conceptual knowledge of fraction arithmetic. Journal of Educational Psychology, 107(3), 909–918. https://doi.org/10.1037/edu0000025

Tekin-Sitrava, R. (2020). Middle school mathematics teachers’ reasoning about students’ nonstandard strategies: Division of fractions. International Journal for Mathematics Teaching and Learning, 21(1), 77–96.

Whitehead, A. N., & Walkowiak, T. A. (2017). Preservice elementary teachers’ understanding of operations for fraction multiplication and division. International Journal for Mathematics Teaching & Learning, 18(3), 293–317. https://eric.ed.gov/?id=EJ1164169

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Handout
Subject	Handout
Author	Sorensen, Diane
File Modified	0000-00-00
File Created	2024-07-21

RELMidwest_5.1.7_TFTEfficacyStudy_Supporting Statement Part A_revised61124_CLEAN