NonSub Change - CCL Case Study July 2023

To: Kelsi Feltz

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Nina Philipsen

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: July, 27 2023

Subject: NonSubstantive Change Request – Culture of Continuous Learning Project: Case Study (OMB #0970-0605)

This memo requests approval of nonsubstantive changes to the approved information collection, Culture of Continuous Learning Project: Case Study (OMB #0970-0605).

Background

On March 17, 2023, we received OMB approval to conduct a descriptive case study for the Culture of Continuous Learning Project: Case Study (OMB #0970-0605). The purpose of this project is to document the factors that contribute to the feasibility of implementing the Breakthrough Series Collaborative (BSC) quality improvement methodology in Head Start and child care centers. The OMB-approved information collection includes: (1) implementation materials (e.g., worksheets); (2) key informant interviews; (3) focus groups with key staff, teachers, and parents; (4) surveys with administrators, teachers, staff, and parents; and (5) classroom observations.

As the team prepared for the data collection, three concerns arose about our initial assumptions when preparing the request package:

1. whether the proposed token of appreciation structure would encourage sufficient response rates,

2. whether the amount proposed for tokens of appreciation aligned with the level of effort requested from participants, and

3. whether we were collecting sufficient information about teacher’s and families’ experiences with their child care center.

Additionally, we noted that participants should be informed that they can decline to respond to any question.

Tokens of appreciation structure. After the initial request was approved by OMB, our team became aware of new research that supported the claim that offering prepaid tokens of appreciation before taking a survey, followed by an additional token of appreciation distributed post-survey, may be more effective than other methods for increasing response rates and addressing non-response bias. For example, in the Assessing the Implementation and Cost of High Quality Early Care and Education (ICHQ) project (OMB #0970-0499), Albanese, Edwards, Weiss, and Gonzalez (2022) found that giving a portion of the total token amount prior to survey completion, followed by providing the rest of the full amount upon completion, increased the response rate among child care staff by 20 percentage points compared to giving the full amount upon survey completion.

To meet the current study’s goals, it is vital we secure participants with a range of background characteristics and personal circumstances to capture a variety of possible perspectives on the BSC experience in the current study. Additionally, maintaining engagement and participation longitudinally (i.e., completing multiple requests over the course of the BSC) is crucial to the study design; a lack of consistent engagement from a variety of participants (regardless of income level) would reduce the overall quality and utility of the data collection efforts. As such, we believe amending the token of appreciation structure to allow for a pre- and post- survey token amount will contribute to our plan to have high response rates that represent the range of experiences and perspectives needed to meet our study’s goals.

Alignment between tokens of appreciation amount and level of effort. In reviewing recent research (e.g., findings from FACES [OMB #0970-0151] and LAUNCH [OMB #0970-0373]), the project team determined that the initial level of appreciation we proposed for individuals who participate in focus groups, interviews, and surveys was not aligned with the level of effort requested. FACES (OMB #0970-0151) for example, offered $35 tokens of appreciation in their 2006 and 2009 cohorts to parents/guardians who completed baseline information forms and reports about their children participating in the study. These tokens of appreciation were reduced to $15 in FACES 2014-2018, which resulted in a drop in response rates from 93.1% to 77.5%, and differential response rates across different demographic groups. The sample for the Project LAUNCH Cross-Site Evaluation (OMB #0970-0373) included preschool and ECE settings and did not initially offer tokens of appreciation to parents completing a 30-minute web-based survey. Early results indicated that respondents were not representative of their communities; individuals with low incomes and those who did not have full-time employment were underrepresented. OMB approved a $25 token of appreciation after data collection had started, which improved both the completion rate and representativeness of responses (LaFauve et al., 2018). Taken together, these findings indicate that offering a lower token of appreciation reduces response rates and yields samples that are not fully representative. In order to meet the study’s goals of securing participants from a wide range of backgrounds and personal circumstances, we propose increasing the total dollar amount offered for the 60-90-minute focus groups and interviews, as well as the total dollar amount for surveys.

More information on teacher’s and families’ experiences with child care. Understanding families’ experiences with their child’s child care center is a key goal of the study. As such, the team proposes adding one additional measure to Instruments 17a-dii; upon review of the measures, the team identified an area where we could add one additional measure to provide a better understanding of families’ experiences. Specifically, we believe that the Parent Voice essential scale from the Essential 0-5 Survey measure would provide a more comprehensive understanding of families’ experiences with their child’s child care center—an understanding that is vital to the study’s goals. The study team piloted the instrument with less than nine individuals to ensure the addition of this scale will not increase respondent burden beyond what OMB has already approved. Lastly, we added two minor questions to the teacher survey to clarify the number of years respondents have worked in the ECE field and their future professional plans.

Overview of Requested Changes

We are seeking slight modifications to address these issues and ensure the data collection is of maximum utility for the government. None of these modifications increase burden on participants.

Changes across instruments. To address the proposed changes to the tokens of appreciation structure (see table below and to update survey consent language to note that respondents can decline to respond to any question, we have updated several instruments and the Supporting Statement A (SSA). Language in the following documents now allow for a pre- and post-survey token of appreciation, an increase in total tokens of appreciation, and provide notice to participants that they may decline to respond to any questions.

• Instrument 13: Key Informant Interviews with BSC Center Administrators Discussion Guide

• Instrument 14: BSC Teachers and Support Staff Focus Group Discussion Guide

• Instrument 15: BSC Parent Focus Group Discussion Guide

• Instrument 16: Individual BSC Teams Focus Group Discussion Guide

• Instruments 17a-dii: Pre-post Surveys with Administrators, Teachers, Staff, and Parents

• SSA

Instrument	Avg. Burden per response (in hours)	Previous token of appreciation per response	Prepaid token of appreciation per response	Post-activity token of appreciation per response	Total new token of appreciation per response
Key Informant Interviews BSC Center Administrators	1	$10.00	$0	$50	$50.00
BSC Teachers and Support Staff Focus Groups	1.5	$10.00	$0	$50	$50.00
BSC Parent Focus Groups	1.5	$10.00	$0	$50	$50.00
Individual BSC Teams Focus Groups	1.5	$10.00	$0	$50	$50.00
Administrator Surveys	0.5	$20.00	$5	$20	$25.00
Teacher Surveys	0.5	$20.00	$5	$20	$25
Other Center Staff Surveys	0.5	$20.00	$5	$20	$25.00
Non-BSC Parent Surveys	0.25	$20.00 lottery ($20 token of appreciation given to up to 5 respondents at each of the 2 timepoints)	$0	$25	$25.00 lottery ($25.00 token of appreciation given up to 20 respondents at each of the 2 timepoints)
BSC Parent Surveys	0.5	$20.00	$5	$20	$25.00

Change to Instrument 1 only. Instrument 1: BSC Selection Application Questionnaire. To meet the study’s goals and highlight the voluntary nature of completing the application, the study team updated the consent language by removing the statement that an applicant can skip any question or stop answering questions at any time. This change is necessary as our team needs a fully completed application to determine whether a center is a good fit for the study.

Change to Instrument 17a-dii only. We have added the Parent Voice essential scale from the Essential 0-5 Survey measure to help capture families’ experiences with their child’s child care center. We believe these items are more aligned with the topics covered by the BSC than those in the Family and Provider/Teacher Relationship Quality (FPTRQ) Measure Short Form originally included in the Survey Instrument. We also deleted the word, “preschool” in reference to preschool teachers. This minor wording change was necessary to fully capture the range of teachers’ classroom age groups within a child care setting. Lastly, we have cut some of the previously OMB approved scales (such as the FPTRQ Measure Short Form referenced above) and items from the surveys to ensure they stay within our OMB-approved burden estimates, best align with the study’s draft evaluation logic model, and have no duplicitous constructs.

Time Sensitivities

Recruitment for data collection is planned to begin shortly. As such, we appreciate a prompt response.