NonSub Change - CCL Case Study Dec 2023

To: Kelsi Feltz

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Nina Philipsen

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: November 29, 2023

Subject: Non-Substantive Change Request – Culture of Continuous Learning Project: Case Study (OMB #0970-0605)

This memo requests approval of non-substantive changes to the approved information collection, Culture of Continuous Learning Project: Case Study (OMB #0970-0605).

Background

On March 17, 2023, we received OMB approval to conduct a descriptive case study for the Culture of Continuous Learning Project: Case Study (OMB #0970-0605). The purpose of this project is to document the factors that contribute to the feasibility of implementing the Breakthrough Series Collaborative (BSC) quality improvement methodology in Head Start and child care centers. The OMB-approved information collection includes: (1) implementation materials (e.g., worksheets); (2) key informant interviews; (3) focus groups with key staff, teachers, and parents; (4) surveys with administrators, teachers, staff, and parents; and (5) classroom observations.

As the team prepares for the data collection, they have identified several proposed updates to improve the effort. These include:

1. Modifying the consent language for instruments associated with data archiving to also include language about the project’s recently approved National Institute of Health (NIH) Certificate of Confidentiality

2. Eliminating the observations instrument (Instrument 18) and updating the estimated respondent burden table accordingly.

3. Adding 7 questions to the pre-post survey as an alternative to observations (Instrument 17b)

4. Eliminating questions, rearranging text, and adding survey directions and one question to the BSC Application Questionnaire (Instrument 1)

Consent language updates.

The purpose of this modification request is to add important language to the consent forms associated with data archiving for the project, as well as language about the project’s recently approved NIH Certificate of Confidentiality. This certificate means that the research team cannot release or use information that may identify participants in any action unless they say it is okay.

Additionally, we have finalized our data archiving plan which includes the possibility to make some of the data available for other researchers to use under restricted use settings. As such, the team needs to modify the consent forms of certain instruments (listed in the Overview of Requested Changes section below) to reflect this change. It should be noted that the change to the consent forms is being made before Time 1 data collection starts so will not result in needing to go back to participants to inform them of this change. Here is the language we are adding to the consent form.

• “Data files from the research component of this project may be stored in a trusted online location where they could be made available to other researchers who commit to keeping the data secure. Any personal information that could identify you will be removed before files are shared with other researchers. No research findings will include any personal information. This research is also covered by a Certificate of Confidentiality from the National Institutes of Health. This means that researchers cannot release or use information that may identify you in any way unless you say it is ok.”

Classroom observation updates.

As part of the original study design, the CCL team initially proposed collecting classroom observational data to answer key research questions related to the project’s theory of change. Our goal was to measure both change over time within classrooms and the potential spread of practices. We proposed conducting classroom observations using the Swivl recording system through which a 2.5-hour recording would be made of a teacher’s classroom, focused on the teacher’s movements, at three timepoints. Our team estimated the administrative tasks associated with these observations (i.e., time needed to schedule and coordinate the observations) would take teachers approximately 20 minutes (0.33 hours) at each timepoint (see Table 1 for updated burden estimates).

At the time of our initial OMB submission, we were in the process of deciding which exact coding systems to use for the observational data. Two measures (the A-TSRS and CLASS) were created for research purposes, to measure high-quality social and emotional (SEL) classroom practices, and to capture evidence of sensitivity to changes in classroom practices over time. These tools, however, are not fully aligned with the Pyramid Model that is used within the project’s BSC Change Framework. Two measures (the TPOT and the TPITOS) that are aligned with the Pyramid Model were used in CCL Phase I (OMB #0970-0507) but were found to be unsuccessfully operationalized as research constructs that could detect change over time in classrooms. Thus, upon investigating the available options, the team is concerned about developing an observational protocol that would fit the needs of the current project.

Additional considerations about collecting observational data. In addition to the lack of clear measures to select as observational tools, the team discussed the prospect of conducting observations in ECE programs in a post-pandemic context. At first glance, the use of tools such as the Swivl video recording system would minimize the presence of an observer spending significant amounts of time in classrooms and was identified as a way to reduce the potential burden on teachers. However, video recordings still require a person to be on-site to help with set up and solve any issues that arise during the observation.

Our team is aware of ECE centers that have cited challenges with visitors as they navigate staff shortages and other financial stressors; in Child Trends’ recent work on a state evaluation project involving center-based ECE recruitment, center directors reported that they are not as flexible as they were in the past to having an observer in the classroom. During a 2023 OPRE methods convening about changes in research protocols as a result of the COVID-19 pandemic, members of our team heard from other researchers who described their experiences with low participation rates in OPRE-funded work that included observations and other time-intensive research requests. Our team is aware of the importance of maintaining strong relationships with the current project’s participants, and we are concerned that research requests involving classroom observations will be perceived as insensitive.

Proposed alternative approach. To address the need to collect information about SEL practice in classrooms while minimizing data collection burden on programs, our team proposes adding a set of seven questions that ask teacher participants to assess how frequently specific behaviors happen in their classrooms. The alternative use of survey items would provide true baseline data for these classroom practices; this option would allow us to document pre-test practices at the classroom level, potentially adding to our understanding of the changes made in SEL classroom practices among ECE programs that participate in the BSC—a key goal of the current study.

BSC Application Questionnaire Updates.

The CCL team cut questions from the BSC Application Questionnaire (see Instrument 1) as they were identified as not necessary to fulfill project needs (i.e., inform BSC team selection decisions). Other text was rearranged in the instrument to streamline the application. Lastly, the team added survey instructions (e.g., Please click ‘Save & Continue Later’ before pausing work on this application!) and one question asking respondents to identify the state their child care center is located in—a question that is crucial for team selection decisions and data analysis.

Overview of Requested Changes

We are seeking a non-substantive change request to address these issues and ensure the data collection is of maximum utility for the government. While the proposal to eliminate conducting observations from the study and updating the survey instruments changes burden, we do not believe these modifications increase burden on participants.

Changes across instruments. We have updated the relevant instruments, Supporting Statement A (SSA), and Supporting Statement B (SSB) to reflect language added to the consents regarding the project’s data archiving and NIH Certificate of Confidentiality. Language in the following documents have been updated.

• Instrument 1: BSC Application Questionnaire

• Instrument 11: BSC Implementation Staff and Faculty Focus Groups

• Instrument 12: BSC Implementation Staff and Faculty Background Survey

• Instrument 13: Key Informant Interviews with BSC Center Administrators Discussion Guide

• Instrument 14: BSC Teachers and Support Staff Focus Group Discussion Guide

• Instrument 15: BSC Parent Focus Group Discussion Guide

• Instrument 16: Individual BSC Teams Focus Group Discussion Guide

• Instruments 17a-dii: Pre-post Surveys with Administrators, Teachers, Staff, and Parents

• SSA & SSB

Change to Instrument 1 only. To streamline the BSC application questionnaire and make it simpler and more straightforward for respondents to complete, the team has removed several questions about past participation in professional development and quality improvement activities. We estimate that these changes will reduce the time to respond from 1.5 hours to 1.25 hours.

Change to Instrument 18 only. For the reasons described above, the team proposes removing the classroom observations from this data collection.

Change to Instrument 17b only. Understanding changes in teacher practices within ECE classrooms is a key goal of the study so as an alternative to the originally proposed observations, the team proposes adding seven questions to the teacher survey. This will provide a better understanding of teachers’ experiences in the classroom; the new questions address the concepts of “Nurturing, responsive and supportive relationships and environments” and “Teaching social skills and emotional competencies.” The CCL team piloted the revised survey and found the additional seven questions did not increase burden.

The proposed questions include:

1. Number of lesson plans you developed in the past week that explicitly focused on social and emotional learning: _______

2. Number of times in the past week you supported a child in regulating their behavior: ___

3. Number of times in the past week you supported a child in developing a pro-social behavioral skill: ______

4. Number of times in the past week you supported a child in understanding their or other’s emotions: ____

5. Number of times in the past week you perceived challenging behaviors or a behavioral disruption among children in your classroom: ___

6. Number of times in the past week you used a negative tone of voice in the classroom: __

7. Number of days in the past week you identified no behavioral disruptions or challenging behaviors during transitions between activities: ___

Changes to Supporting Statements.

Supporting Statement A (SSA) and Supporting Statement B (SSB) have been updated to reflect the aforementioned changes, including:

• adding language regarding data archiving and the NIH Certificate of Confidentiality,

• reducing the estimated burden by updating the response time to Instrument 1 from 1.5 hours to 1.25 hours, and

• removing the burden (48 total hours) associated with observations (Instrument 18).

Time Sensitivities

Recruitment for data collection is planned to begin shortly. As such, we appreciate a prompt response.

4