NonSub: Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase 1

To: Office of Information and Regulatory Affairs (OIRA); Office of Management and Budget (OMB)

From: Office of Planning, Research and Evaluation (OPRE); Administration for Children and Families (ACF)

Date: June 22, 2016

Subject: Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase I (OMB 0970-0355)

The Assessing the Implementation and Cost of High Quality Early Care and Education (ECE-ICHQ) information collection request was submitted and approved under the ACF/OPRE generic clearance for pre-testing (0970-0355). We are writing to request the following non-substantive changes:

• An increased level of appreciation for participating programs

• Updated recruitment materials to better reflect the study activities and mention a token of appreciation for participating in the study

• Revised data collection instruments to increase efficiency and ease of use and offer more flexibility to respondents in how information is provided to the study team

Justification for increased level of appreciation

The project team conducted a small pilot to test the data collection process and tools. Sites were pursued for the pilot based on purposive selection and to limit travel costs; the sites were not located in the three states identified for Phase 1. At the same time, the team reached out to three individuals from the states identified to be part of Phase 1 to seek assistance in identifying centers for data collection. The following experiences suggest an increased level of appreciation will be necessary to gather the information needed for this important first phase of the project.

• The current $100 token of appreciation was not viewed favorably relative to the time center staff were asked to commit to the data collection. All three programs approached for the pilot and one state representative contacted as part of Phase I mentioned this disconnect. Among the three sites approached for the pilot, one program participated, one declined, and one initially agreed and then wanted to back out. The team worked with the last program to conduct data collection on a piece-meal basis to get as much completed as they were willing to do. This piece-meal effort was not time- or resource-efficient. Of the states contacted to be part of Phase 1 data collection, one declined. One concern expressed by the state representative was that the level of appreciation of $100 was not sufficient based on what the team would be asking centers to do.

• There is little direct, short-term benefit to encourage participation from a broad range of centers. The project relies on achieving variability across the types of programs that participate in Phase 1—large and small, serving different ages of children, and ranging in types of funding sources and auspices. Programs that receive public funding (such as Head Start) may perceive a greater responsibility to participate in research, particularly when the research is specific to the program (for example, the Head Start Family and Child Experiences Survey (FACES) [0970-0151]). Most centers targeted to achieve the desired variability in center types and characteristics for Phase 1 may have no particular sense of responsibility to participate in this work.

Based on these experiences, the project team determined that our originally proposed level of appreciation for centers was not well-aligned with the level of effort requested from the center. While this data collection effort under the generic clearance is still considered low burden for individuals, in order to adequately thank centers for their time, we suggest increasing the level of appreciation. We looked to other projects receiving generic clearance approval with either a significant recruitment or staff-level participation effort to identify a level of appreciation that may better support the project goals. The project team found the FACES Redesign Pilot study and the Assessing Early Childhood Teachers’ Use of Child Progress Monitoring to Individualize Teaching Practices study (EDIT) as the closest, recent comparable data collection efforts. Both of these projects, using similar levels of appreciation, were able to meet their center recruitment goals and obtain over 80% response rates for data collections at the center level.

Using information from these two projects, the ECE-ICHQ study team recommends two revisions to the appreciation (see Table 1): (1) increasing the amount of the center-level appreciation to $350 to achieve success in recruiting centers with the desired variability of characteristics, and (2) adding an individual-level appreciation of $10 for each respondent completing the time-use survey to support completion by the number and range of staff needed within each center.

Table 1. Level of burden and appreciation per center for the ECE-ICHQ, FACES Redesign Pilot, and EDIT studies

	ECE-ICHQ
Respondent type	Type and level of burden1	Level of appreciation
		(current)	(proposed)
Directors, other administrators or managers	17.75 hours per center to participate in recruitment and data collection by gathering information, working on cost workbook and other self-administered tools, and participating in telephone and in-person interviews	$100 per center	$350 per center
	30 minute survey and 15 minute cognitive interview debrief with an average of two administrators per center	Small giveaway item valued at $2 per respondent	$10 per respondent
Teachers	30 minute survey and 15 minute cognitive interview debrief with an average of 11 teachers per center	Small giveaway item valued at $2 per respondent	$10 per respondent
	FACES Redesign Pilot
Respondent type	Type and level of burden	Level of appreciation
Directors, other administrators or managers	40 minutes per Head Start grantee for recruiting two centers and completing classroom selection forms. Supporting on-site presence of data collection team to conduct 30 child assessments per center.	$200 per grantee program
Parents	10-minute survey on-line or by telephone	$15 per respondent
	EDIT
Respondent type	Type and level of burden	Level of appreciation
Directors	10-minute recruitment call and supporting on-site presence of data collection team to conduct observation in one classroom and interview the teacher	$50 per center
Teachers	3.08 hours per teacher to assemble documents for review, record videos, complete an SAQ, and participate in interview	$75 per respondent
	20-minute debriefing discussion by telephone	$20 per respondent

Results from additional test sites. Given the challenges in recruitment and data collection experienced with the three pilot sites, OPRE elected to pursue four additional pilot sites to assess whether changes to the level of appreciation and improvements to the data collection process and tools would lead to greater success. The study team was successful in recruiting four sites within four weeks by offering the $350 center-level token of appreciation. The study team administered the time-use survey to three staff in only one of the four centers (to remain under nine respondents total from the pilot and this one additional center). The two teachers completed and returned the surveys and participated in the debriefing discussion within the day; they were pleased with the $10 token of appreciation. The survey from the center director has not yet been received. The amount of time to recruit and collect data from the final four centers in the pilot was substantially shorter than the first three centers who were only offered $100 appreciation.

Revisions to recruitment materials

The initial email to center directors more explicitly describes the study and its purpose. It also introduces the proposed $350 token of appreciation for participating in the study.

The center recruitment call script has been updated to include a mention of the $350 gift card that the study team proposes to give participating centers. The call script has also been updated to more accurately describe the study activities that will take place via the phone and during the site visit. Additionally the recruitment call is intended to gather initial center information to help with sampling so some questions were taken from the self-administered questionnaire (SAQ).

The frequently asked questions document has been updated to include information on: why the study team is interested in this specific location and why the center should participate. The section regarding how the centers were chosen has been updated to reflect a shorter site visit and increased token of appreciation.

Revisions to data collection materials

The data collection instruments have been revised to increase efficiency and ease of use and offer more flexibility to respondents in how information is provided to the study team. The revisions to the data collection tools do not add new substantive content for data collection; question sections or items have been reorganized within and across tools and wording changes have been made to improve clarity. The revised set of materials does not change the approved burden estimates. Below, we summarize changes made for each instrument.

The SAQ has been eliminated and the questions from that tool have been distributed to other tools to increase efficiency of data collection and improve the quality of responses. Through the pilot, the team learned that centers were not able to set aside time to complete the SAQ on their own but they could readily answer the questions when asked as part of other data collection efforts. Questions on center characteristics were moved into an initial data collection interview that will occur immediately after recruitment and before the visit. Some questions on staffing, training, planning, and other family services were moved into a pre-visit implementation call to gather information that will help the team prepare for the site visit. Questions that require additional guidance from or discussion with the interviewer were integrated into the implementation interview protocols. Five items from the SAQ are maintained in worksheets for completion by center staff on their own.

The implementation interview has been parsed into five modules for ease in data collection.

The cost workbook has been revised to allow respondents to provide information in ways that map more easily to their existing records. For example, information about staff salaries and benefits can be provided for individual employees, groups of employees (e.g. by type), or a total number for all employees. We removed questions that pilot respondents indicated they could not easily answer at an individual staff level and instead ask a more general question about variation among staff. We added cost categories to better distinguish between the types of costs reported.

The time-use survey has been revised to simplify several items and utilize a more salient time referent for these items. For example, we broke one long item with over a dozen response options into several items, each with fewer responses. These items now ask respondents to report the percent of time they spent during their most recent work day, where previously respondents were asked to report the number of hours in a typical work week. These changes mean respondents no longer need to check their work across questions to make sure the hours reported are consistent. We have also simplified and streamlined the section about classes taken in pursuit of a degree or credential, from a large table into a series of four very brief questions.

1 These burden estimates are from our original submission of OMB Section A , Table A.5. The estimate of 17.75 hours for directors, other administrators, and managers is inclusive of: Center Recruitment and Engagement Script Part 1 (.33 hours, 1 response per respondent), Center Recruitment and Engagement Script Part 2 (.42 hours, 1 response per respondent), SAQ (3.5 hours, 1 response per respondent), SAQ Cognitive Interview Protocol (.5 hours, 1 response per respondent), Implementation Interview Protocol (2.5 hours, 1 response per 2 respondents), Cost Workbook (6.5 hours, 1 response per respondent), and Cost Interview (1.5 hours, 1 response per respondent). The estimates of the 30 minute survey and 15 minute debrief for directors, other administrators, managers and teachers is the time use survey.

3