Justification

Regional Educational Laboratory, Southwest

Usability, Feasibility, and Acceptability Study of the Leading Early Childhood Achievement and Development (LEAD) Intervention

OMB# 1850-0952 v.9

Volume I

Supporting Statement

Submitted by:

National Center for Education Evaluation (NCEE)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

February 2024

Attachments

Attachment I. School Leader and Teacher interview protocols

Table of Contents

1) Submittal-Related Information 3

2) Background 3

3) Design and Context 4

4) Recruitment and Data Collection 4

5) Estimated Respondent Burden 5

6) Estimate of Costs for Recruiting and Paying Respondents 5

7) Cost to federal government 5

8) Assurance of Confidentiality 5

9) Justification for Sensitive Questions 6

10) Project Schedule 6

1. Submittal-Related Information

The following material is being submitted under the National Center for Education Evaluation (NCEE) generic clearance agreement (OMB# 1850-0952), which provides NCEE the capability to collect preliminary or exploratory information to aid in study design by: (1) fielding brief, quick turnaround surveys, extracting test case administrative data, administering interviews, or conducting “mini-experiments” in advance of a study for the purpose of determining feasibility, a random assignment or comparison group strategy, or a data collection approach most suitable for a potential or planned evaluation; and (2) developing, testing, and improving its survey and assessment instruments, methodologies, and study dissemination strategies.

This request is to conduct interviews with teachers and school leaders participating in the Leading Early Childhood Achievement and Development (LEAD) professional learning intervention field test. These interviews will be used to understand school leader and teacher perspectives on the usability, feasibility, and acceptability of the LEAD intervention in order to inform design and refinements to the LEAD intervention. Interviews will begin in February 2024.

2. Background

The Oklahoma State Department of Education (OSDE) has an interest in learning more about variation in quality, as reflected in teachers’ use of research-based early learning practices, specifically play-based learning and high-quality interactions, in PreK and kindergarten classrooms. Play-based learning is meaningful, actively engaging, joyful, iterative, and socially interactive (Zosh et al., 2017). Although specific statewide measures of instructional quality are not available, other data support the state’s concerns. For example, only 9 percent of ECE facilities participating in Oklahoma’s Reaching for the Stars Quality Rating and Improvement System demonstrate the highest level of program quality. Also, disparities in early reading skills between students with different socioeconomic and racial/ethnic backgrounds in Oklahoma (OSDE, 2022) suggest disparities in access to quality ECE learning experiences.

OSDE would like to address these disparities by increasing the use of research-based early learning practices in PreK and kindergarten classrooms. OSDE is actively working to increase the use of research-based early learning practices through multiple initiatives, but additional support for school leaders and teachers is still needed. OSDE has limited optional online modules available to educators on research-based early learning practices on OSDE Connect (https://osdeconnect.pdx.catalog.canvaslms.com/), the state’s repository for online professional learning courses for Oklahoma educators. OSDE does not currently offer or track any training for school leaders that focuses on preK, kindergarten, and play-based learning or high-quality interactions. This need is substantiated by staff in OSDE’s Department of Teacher and Leadership Effectiveness, which receives requests from school leaders for professional development and support to understand and implement research-based best practices in early childhood education teaching and learning. With staffing issues and other constraints in the department, OSDE cannot meet these requests for professional development. Without this ECE foundational knowledge, school leaders are not equipped to be instructional leaders who demonstrate effective instructional leadership that can build a collaborative and reflective school environment for teachers.

OSDE, Anadarko Public Schools, Shawnee Public Schools, and the Regional Educational Laboratory (REL) Southwest formed the Leading Early Childhood Achievement and Development (LEAD) partnership in Oklahoma to respond to this need. REL Southwest will collaborate with these partners to codevelop, iteratively refine, and test a professional learning intervention (referred to as the LEAD intervention). The LEAD intervention is designed to complement and fully support the use of Oklahoma’s Academic Standards (OAS) and district-selected curricula. The intervention aims to build school leaders’ (for example, principal, instructional coach, another designated school leader) and PreK and kindergarten teachers’ knowledge of research-based early learning practices—play-based instructional practices and high-quality interactions (short-term outcomes). In addition, the LEAD intervention will develop school leaders’ instructional leadership skills and provide resources to support PreK and kindergarten teachers’ implementation of research-based practices (medium-term outcomes). In time, these practices are expected to lead to better academic outcomes for students (long-term outcome).

REL Southwest is developing the LEAD intervention through an approved Training, Coaching, and Technical Support (TCTS) project. To increase the likelihood that the LEAD intervention will achieve its intended outcomes, the partnership will study the usability, feasibility, and acceptability of the core components of the LEAD intervention. The proposed study includes two iterations of piloting the LEAD intervention across two cohorts: cohort 1 during the 2023/24 school year and cohort 2 during the 2024/25 school year. Usability, feasibility, and acceptability testing can yield practical recommendations that can be applied to the structure, design, and redesign of the LEAD intervention. Piloting the intervention across two cohorts, rather than just one, will allow the TCTS team to iteratively test and refine the usability, feasibility, and acceptability of the intervention.

The study team will conduct the same research activities with both cohorts. The study will provide the LEAD development team with critical, timely information about participants’ perceptions of what makes (or does not make) LEAD a usable, feasible, and acceptable professional learning opportunity. The study is the initial step or phase in testing a high-quality professional learning intervention that is responsive to the local context and limitations. The findings from the study will inform an efficacy trial in a future applied research project during the 2025/26 school year.

3. Design and Context

The interviews for which this clearance is requested will be used to understand teacher and school leader perceptions of LEAD’s usability, feasibility, and accessibility. REL Southwest will use the findings generated from these interviews to inform and develop hypotheses about optimal delivery of LEAD for different contexts. Hypotheses will be related to dosages, modalities (in-person versus virtual versus asynchronous modules), and other LEAD components that will be efficacious under different school and district conditions (such as different resource constraints).

For each cohort, we will conduct interviews at two time points (after completing module 1 and when participants are close to completing module 4) with four school leaders and eight teachers. For each cohort, the study team will interview the same school leaders and teachers at the two time points. The interviews will be conducted via video conference and will be audio and/or video recorded so that researchers can reference recordings to supplement written notes taken during the interview.

The study team developed a school leader and teacher interview protocol for each time point, for a total of four interview protocols. The protocols focus on understanding school leader and teacher perspectives on the usability, feasibility, and acceptability of the LEAD intervention. Usability questions ask about the materials and supports that school leaders received for the intervention (for example, To what extent were the LEAD materials easy to understand? How easy or difficult was it to access and use the materials?). Feasibility questions ask about the convenience and practicality of the intervention activities (for example, To what extent were the school leader PLC sessions scheduled at a time that was convenient to attend? What would make it more convenient? What school or community contextual factors supported or hindered participation in the LEAD intervention? How could the intervention be adapted to address feasibility concerns?). Acceptability questions ask about how relevant the LEAD intervention is to student needs in their school and if the LEAD intervention is contributing to changes in school and classroom practices at their school.

We will pilot test the school leader interview protocols with two school leaders and the teacher interview protocols with two teachers to ensure that the language in the protocols is relevant and understandable to participants, confirm that questions generate the type of information necessary to answer the study research questions, and determine if the protocols can be implemented in 45-minutes.

4. Recruitment and Data Collection

REL Southwest partners (such as district administrators) will identify LEAD participants and the study team will be responsible for helping to select LEAD participants to interview, obtaining their agreement to be interviewed, conducting the interviews, compiling interview responses, and summarizing findings. Interviewers will ask questions in an open-ended manner using the protocols in Attachment I, while allowing for new questions or ideas to be brought up during the interview as a result of the interviewee’s responses.

For each cohort, the study team will conduct 45-minute semi-structured interviews with up to four school leaders (for example, one from each participating school) and up to eight teachers (for example, one PreK and one kindergarten teacher from each participating school). If more than one school leader is participating from each school, we will select the school leader who is most involved in professional development of PreK and kindergarten teachers to interview (that is, we will interview the school leader who leads the community of practice for PreK and kindergarten teachers). One PreK and one kindergarten teacher will be randomly selected from each school to participate in the interviews from among the participating teachers. Interviews will occur at two time points: after completing the first module and when participants are close to completing module 4.

The study includes two iterations of piloting the LEAD intervention across two cohorts: cohort 1 will include up to four schools during the 2023/24 school year, and cohort 2 will include a new cohort of up to four schools during the 2024/25 school year. Piloting the intervention across two cohorts of schools, rather than just one, will allow the TCTS team to test the usability, feasibility, and acceptability of the initial version of LEAD intervention and any refinements to educators in different settings and context. Cohort 1 includes schools from two districts (Anadarko Public Schools and Shawnee Public Schools). Two additional districts (these districts have not yet been identified) will participate as Cohort 2. Cohort 1 will begin in January 2024 and end in April 2024 and Cohort 2 will begin in September 2024 and end in March 2025.

For school leader and teacher interviews, interviewers will audio record the interviews, take field notes, and work with a transcriptionist to transcribe the audio recordings. The study team will review transcription files to check their accuracy. The files will then be uploaded to NVivo for coding and analysis. The study team will develop a codebook with codes aligned to research questions and descriptions and examples of each code. We will use a combination of inductive and deductive approaches to coding. We will start with a small set of deductive codes aligned to the study’s research questions and then inductively add new codes. Using this approach, we will be able to focus our analysis on the study’s priority constructs and at the same time allow for unanticipated themes to emerge. The coding process will systematically examine the data to identify prevalent patterns and themes in school leaders’ and teachers’ experiences with the LEAD intervention. The process will incorporate key findings from the analysis of qualitative data into formative feedback for the LEAD development team.

Two study team members will code the qualitative data for this study. Training for the coders will consist of reviewing the coding structure and if necessary, agreeing on any additional elaboration or explanation of the codes. In addition, coders will double code approximately 10 percent of the interview data. For example, if we have a total of 24 interviews across the two time points for Cohort 1, coders will independently double code two interviews, one randomly selected school leader interview and one randomly selected teacher interview. The coders will calculate interrater reliability on the double-coded data. Interrater reliability will be calculated using Cohen’s kappa statistic, which accounts for the possibility of agreement by chance and provides an assessment of the level of calibration between each coder (Cohen, 1968). To advance to coding the remaining data independently, the coders must first reach 0.60 for Cohen’s kappa, the minimum threshold for interrater reliability. If coders do not reach this minimum threshold using the initial sample of 10 percent of the qualitative data collected, they will review the coding structure and then double code up to an additional 10 percent, discuss their codes, and calculate interrater reliability again. Once the coders complete the initial double coding and calculate interrater reliability, they will discuss the data that they double coded, review any discrepancies in their codes, agree on the final code, and make any necessary updates to the coding structure based on their discussion.

5. Estimated Respondent Burden

Table 1 details the annual (school year) respondent burden for the interviews. The estimates assume that it will take 15 minutes to schedule interviews, that interviews will take 45 minutes, and that each respondent will be interviewed twice in each school year.

Table 1. Estimated Annual Participant Burden

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Teacher interviews	8	16	60	16
School leader interviews	4	8	60	8
Study Total	12	24	1,440	24

Note: Numbers have been rounded to whole numbers.

Table 2 details the respondent burden for the interviews across both cohorts of the study. As in Table 1, the estimates assume that it will take 15 minutes to schedule interviews, that interviews will take 30 minutes, and that each respondent will be interviewed twice.

Table 2. Estimated Total Participant Burden, Across Two Cohorts

Activity	Number of Respondents	Number of Responses	Minutes per respondent	Total burden hours
Teacher interviews	16	32	60	32
School leader interviews	8	16	60	16
Study Total	24	48	2,880	48

6. Estimate of Costs for Recruiting and Paying Respondents

Consistent with IES guidance on incentives, each school leader will receive a $50 electronic gift card after completing each interview as a thank you and each teacher will receive a $30 electronic gift card after completing each interview as a thank you.

7. Cost to federal government

The total cost to federal government for conducting the interviews will be $133,102. This cost includes scheduling interviews, conducting interviews, and analysis and reporting.

8. Assurance of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Respondents will be informed that their participation is voluntary and that all information they provide will not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). At the start of the interview, this information will be conveyed verbally to respondents by the interviewer.

The REL Southwest research team will follow confidentiality and data protection protocols in place at the American Institutes for Research (AIR). They will submit and maintain a data governance plan. All data will be housed on a secure drive, and access to this drive will be restricted to staff assigned to the project. Any data obtained for this study will be used only for descriptive analyses.

9. Justification for Sensitive Questions

The interview protocols do not include questions that might be considered sensitive.

10. Project Schedule

Cohort 1 will begin implementing the LEAD intervention in January 2024 and end in April 2024. Cohort 2 will begin the implementation in September 2024 and end in March 2025. Data collection through interviews will begin in February 2024 and will conclude in May 2025.