supporting statement part A
Data Collections for Culture of Continuous Learning Project: A Breakthrough Series Collaborative for Improving Child Care and Head Start Quality
OMB Information Collection Request
New Collection
Supporting Statement
Part A
September 2017
Passback: March 2018
Submitted By:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer: Ivelisse Martinez-Beck
CONTENTS
A1. Necessity for the data collection 4
Study background 4
Legal or administrative requirements that necessitate the collection 5
A2. Purpose of survey and data collection procedures 5
Overview of purpose and approach 5
Research questions 5
Study design 6
Universe of data collection efforts 6
A3. Improved information technology to reduce burden 14
A4. Efforts to identify duplication 15
A5. Involvement of small organizations 15
A6. Consequences of less frequent data collection 15
A7. Special circumstances 15
A8. Federal Register notice and consultation 15
Federal Register notice 15
Consultation with experts outside of the study 16
A9. Incentives for respondents 16
A10. Privacy of respondents 18
A11. Sensitive questions 19
A12. Estimation of information collection burden 19
Burden hours 19
Total annual cost 21
A14. Estimate of cost to the federal government 21
A15. Change in burden 21
A16. Plan and time schedule for information collection, tabulation, and publication 21
Analysis plan 21
Time schedule and publications 22
A17. Reasons not to display OMB expiration date 23
A18. Exceptions to certification for Paperwork Reduction Act submissions 23
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Child and Families (ACF) at the U.S. Department of Health and Human Services seeks approval to collect information as part of the Culture of Continuous Learning (CCL) Project: A Breakthrough Series Collaborative for Improving Child Care and Head Start Quality. The purpose of the study is to investigate the feasibility of supporting children’s social and emotional learning using a continuous quality improvement methodology called the Breakthrough Series Collaborative (BSC).
CCL project staff will gather information throughout implementation of the Breakthrough Series Collaborative to assess the feasibility and potential effectiveness of the model. Data collection will incorporate a variety of local perspectives, including those of leaders in the ECE system, program directors, teachers, and parents. Research questions will address issues such as what makes an ECE program ready to participate in an intensive quality improvement process; what support ECE program staff need to collect and use data; what changes can be documented in organizational culture as a result of participation; what conditions promote learning within and across organizations; and what adaptations to the Breakthrough Series Collaborative model would enable it to work best within the context of ECE.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
The Feasibility Study aims to address two sets of questions: (1) questions regarding the feasibility of implementing a BSC focused on social and emotional learning within both child care and Head Start settings, and (2) questions regarding the participants’ experience of initial changes in individual beliefs, practices, and organizational culture as a result of participating in the BSC.
To assess the feasibility of the BSC, the research team will take a case study approach using observational measures, surveys, focus groups, interviews, and secondary measures. After obtaining OMB approval, the surveys and observations will be administered to participants of the core BSC team, as well as a few additional staff members, at each of the eight sites selected to participate in the BSC. Administrative data from the BSC implementation and participating programs will also be gathered and reviewed. After completion of the BSC, the surveys and observation will be re-administered to participants, and a focus group of core BSC team members will be convened.
The data collection and reporting activities to be conducted as part of the CCL Project feasibility study seek to address the following research questions:
What does “successful participation” in the BSC look like?
What characteristics support “successful participation” in the BSC in child care and Head Start settings?
To what extent are participants’ experiences similar or different across Core BSC teams?
Were there model adaptations, and if so, were they delivered as expected?
What BSC elements are most helpful to Core BSC Teams? What BSC elements were most challenging?
To what extent does the 12-month timeframe promote or impede progress? And successful participation?
What are the costs of implementing the BSC in early care and education settings?
In what contexts or under what conditions do participants in the BSC experience initial changes in individual beliefs, knowledge, and/or practices?
In what contexts or under what conditions do participants in the BSC experience initial changes in organizational culture?
How and to what extent do quality improvements spread within ECE programs and systems, beyond just the participants in the Core BSC team?
Strengths and limitations. One notable limitation of the study design is the small sample size. However, in order to strengthen the design, we will use a case study design. A case study design is well suited to yield rich description of BSC implementation in early care and education settings and the extent to which it is a feasible approach for promoting changes in practice at multiple levels of the organization that ultimately support children’s social and emotional learning.
The implementation of a BSC in early care and education settings in one community is considered a single case study in which the BSC serves as the case and participating Head Start and child care programs are sites within the case. [For comparison, a multi-case study would compare different types of quality improvement approaches, perhaps contrasting the approaches in different communities, cities, states, or countries (Yin, 2014).] However, the CCL project is an instance of a particular case study design – one in which multiple units of analysis are embedded within a single case study, known as an embedded case study design (Yin, 2014). Because of the interest in understanding BSC implementation in Head Start and child care programs, the case study will include embedded cross-site comparisons of Head Start versus child care programs within the single case study design. Cross-site comparisons will not be limited to comparing Head Start programs to child care programs, however; we will explore if there are any patterns to the conditions or characteristics of programs (including but beyond program type) that support successful engagement in BSC activities. For example, we may also be interested in looking at size of program as a condition that matters in the support of successful engagement in BSC activities (see sampling section below). Nevertheless, we will also aim to avoid a pitfall of embedded case study designs – that of having the original phenomenon of interest (in this case, the BSC) become the context rather than the focus of the study (Yin, 2014, p. 56).
Clearance is requested for the following data collection activities for the CCL Project Feasibility Study. Table A.1 provides a crosswalk between the study instruments and the specific research questions each is designed to address. Each of the instruments are described in detail following the table.
Research Questions |
BSC Selection Questionnaire |
Team Building Activities |
Data Collection Planning WS |
PDSA Planning Form & Tracker |
Discussion Forum Postings |
Learning Session Day 1 Evaluation |
Learning Session Overall Evaluation |
Action Planning Form |
TPOT/ TPITOS |
Pre-/Post- Survey |
ECWES |
Self-Report of BSC Activities |
Core BSC Team Focus Group |
1. What does “successful participation” in the BSC look like? |
|
x |
x |
x |
|
|
|
x |
|
|
|
|
|
2. What characteristics support “successful participation” in the BSC in child care and Head Start settings? |
x |
x |
x |
x |
x |
|
|
x |
|
x |
|
|
|
3. To what extent are participants’ experiences similar or different across Core BSC teams? |
x |
x |
x |
|
|
|
|
|
|
|
|
x |
|
4. Did the procedures and training used to support Core BSC Team Members work well? |
x |
x |
x |
|
|
x |
x |
x |
|
|
|
x |
|
5. What BSC elements are most helpful to Core BSC Teams? What BSC elements were most challenging? |
|
|
|
|
x |
|
|
|
|
|
|
x |
x |
6. To what extent does the 12-month timeframe promote or impede progress? And successful participation? |
|
|
|
|
|
|
|
|
|
|
|
x |
x |
7. What are the costs of implementing the BSC in early care and education settings? |
|
|
|
|
|
|
|
|
|
|
|
|
x |
8. In what contexts or under what conditions do participants in the BSC experience initial changes in individual beliefs, knowledge, and/or practices? |
x |
|
|
x |
x |
x |
x |
|
x |
x |
|
x |
|
9. In what contexts or under what conditions do participants in the BSC experience initial changes in organizational culture? |
|
|
|
|
x |
|
|
|
|
x |
x |
|
|
10. How and to what extent do quality improvements spread within ECE programs and systems, beyond just the participants in the Core BSC team? |
|
|
|
|
|
|
|
|
x |
|
x |
|
x |
BSC Selection Questionnaire (Attachment A). The selection packet provides information to the CCL BSC Implementation Team and assists in the mutual selection process. The packet contains a series of questions that cover several topical areas, including: Population served (demographics, numbers and ages), number of classrooms per program/site, staff (demographics, numbers and ages), programs/services offered by the program/site, HR/professional development/training plan, organizational capacity for improvement (organizational readiness for change, capacity for change, communication systems, etc.), and current strengths and challenges in the designated content area (in this case, social-emotional learning). The Feasibility Study Team will use this information to build a descriptive portrait of the programs that engage in the CCL BSC process. Some characteristics of the programs gathered at this early stage in the BSC process may help to identify for whom and under what circumstances the BSC is successful in ECE settings. The selection packet is completed by the director of each program.
Pre-Work Assignment: Team Building Activities (Attachment B). The pre-work assignment of the BSC process helps set the stage for the work to be done during the Learning Sessions. Information gathered through group work helps the Core BSC Team (1) to build shared goals and purpose and (2) evaluate their organization’s performance and identify strengths and weaknesses to guide their improvement work. The first assignment is to develop a team name and motto. The second assignment is to complete a written questionnaire to evaluate the strengths and weaknesses of the program based on the Primary Drivers within the SEL Driver Diagram. The Feasibility Study Team will use the written documentation from the Pre-Work Assignments to describe the BSC process for the implementation and outcomes portions of the evaluation. The full Core BSC Team from each site will be invited to complete the Pre-Work Assignments.
Pre-Work Assignment: Data Collection Planning Worksheet (Attachment C). The pre-work assignment of the BSC process helps set the stage for the work to be done during the Learning Sessions. Information gathered through group work helps the Core BSC Team develop a plan for data gathering. The assignment is to develop a written plan for how to collect the metrics associated with the SEL Driver Diagram. The programs are collecting this data for their own use as part of their participation in the BSC, but are given flexibility to collect and maintain the data in ways that are appropriate and sustainable for their individual programs. The data also will be used as administrative data for the feasibility study at a later point in the project. The Feasibility Study Team will use the written documentation to describe the BSC process. The full Core BSC Team from each site will be invited to complete the Pre-Work Assignments.
Plan, Do, Study, Act Planning Form & Tracker (Attachment D). The Plan, Do, Study, Act (PDSA) Tracker helps teams identify and track the specific small changes they test based on the secondary drivers (concrete strategies). The form allows them to keep track of the lessons learned from each PDSA cycle. These forms enable teams to monitor how their tests of change align with the driver diagram (aims, primary drivers, secondary drivers), and how they relate to the associated metrics. The Feasibility Study Team will conduct secondary analysis of the PDSA Planning Form & Trackers for each Core BSC Team to answer questions about “successful participation” in the BSC and describe the types of activities addressed in the BSC focused on improving SEL. Members of the Core BSC Team from each site will develop and/or add to their PDSA Planning Form & Tracker weekly.
Discussion Forum Prompts (Attachment E). An online discussion forum will be created for Core BSC Teams to share information with one another, and to problem-solve around improvement practices and organizational capacity for improvement. Although participation the Discussion Forum is voluntary, those Core BSC Team Members who do participate may be characterized as “more engaged” in the BSC process. The Feasibility Study Team will conduct secondary analysis of the Discussion Forum postings to determine whether participation in such online discussions promotes more successful outcomes of the BSC for participants and to identify the most common themes and topics discussed to identify some of the challenges that Core BSC Teams faced while implementing the BSC, as well as self-identified change in practice. Every member of the Core BSC Team is encouraged to participate in the Discussion Forum.
Learning Session Day 1 Evaluation (Attachment F). A written questionnaire is used to gather feedback on Core BSC Team Members’ experiences for each day of the Learning Session. The Day 1 Evaluation collects Information on what was most and least helpful, any burning questions participants have, and any other matter that the participants want to bring up. The Feasibility Study Team will analyze this information secondarily to determine whether the training and support provided to the Core BSC Teams worked well and whether the Implementation Team was able to be responsive to the Core BSC Teams. Every member of the Core BSC Team will be asked to complete the questionnaire.
Learning Session Overall Evaluation (Attachment G). A written questionnaire is used to gather feedback on Core BSC Team Members’ experiences for each day of the Learning Session. The Overall Evaluation collects information on participants’ general perceptions of the learning session and self-reflections of social and emotional teaching practices and improvement. The Feasibility Study Team will analyze this information secondarily to determine whether the training and support provided to the Core BSC Teams worked well and whether the Implementation Team was able to be responsive to the Core BSC Teams. Every member of the Core BSC Team will be asked to complete the questionnaire.
Action Planning Form (Attachment H). This form is used by the Core BSC Teams at the end of each Learning Session to help plan for each Action Period. Team members identify their key priorities for this action period, which PDSAs they will be testing or spreading, how they will measure their success, who will be responsible, and their timelines and process for tracking progress. The Action Planning Forms help “roll up” the multiple PDSAs they plan to test (as documented on the individual PDSA Planning Forms & Trackers) into a single team workplan that can help guide the team’s Action Period work more generally. The Feasibility Study Team will request copies of all Action Planning Forms to help document the focus of improvement for Core BSC Teams. This information will be used to document the goals set by sites for improvement and, when analyzed along with subsequent PDSA worksheet information, may identify which individuals and sites were most successful in carrying out their Action Plan. Each Core BSC Team completes a single Action Planning Form.
Classroom Observations.
Teaching Pyramid Observation Tool (TPOT) (Attachment I). In order to examine how preschool teachers and practitioners change their practices around social and emotional learning, the feasibility study team will use the Teaching Pyramid Observation Tool as a pre-/post- observation measure of the classroom environment. The TPOT is an on-site observation tool and has three subscales about how well preschool teachers implement the Pyramid Model. The first subscale is called Key Practices. Examples of questions include “Teachers engage in supportive conversations with children” and “Teacher acknowledges the children’s communication to him or her.” The second subscale is Red Flags. An example question is “Transitions are more chaotic than not.” The third subscale is Responses to Challenging Behavior. Examples in this subscale are “Teacher responds to children by stating the expected behavior in positive terms (i.e. what to do) or providing instruction in an acceptable behavior.” The TPOT is administered only in classrooms that serve children between the ages of two and five years old. The observation takes approximately two hours to fully complete. At the end of the observation, the observer conducts a 20-minute interview with the lead teacher. The purpose of the interview is to capture any subscale components that were not able to be observed. For example, if no children exhibited challenging behaviors during the observation, the observer can ask teachers how they respond to challenging behaviors in the interview. If a teacher is unavailable to complete the interview immediately after the observation, the observer will make arrangements to complete the interview over the phone as soon as possible.
Teaching Pyramid Infant-Toddler Observation Scale (TPITOS) (Attachment J). In order to examine how infant and toddler teachers and practitioners change their practices around social and emotional learning, the feasibility study team will use the Teaching Pyramid Infant-Toddler Observation Scale as a pre-/post- observation measure of the classroom environment. The TPITOS is an on-site observation tool about how well preschool teachers implement the Pyramid Model. It involves an observation of an infant-toddler classroom and involves three main elements: (1) observing for red flags; (2) observing specific routines (i.e., free play, feeding/mealtime, structured group activity) and assessing child engagement; and (3) observing specific routines (i.e., free play, feeding/mealtime, physical care routine, and structured group activity) and rating behavioral and environmental items. That TPITOS is administered only in classrooms that serve infants and toddlers. The observation takes approximately two hours to fully complete. At the end of the observation, the observer conducts a 20-minute interview with the lead teacher. The purpose of the interview is to capture any teaching practices that were not able to be observed. For example, if no children exhibited challenging behaviors during the observation, the observer can ask teachers how they respond to challenging behaviors in the interview. Additionally, during the interview, the observer will ask teachers about things like their collaboration with other teachers and strategies for communicating with parents and families. If a teacher is unavailable to complete the interview immediately after the observation, the observer will make arrangements to complete the interview over the phone as soon as possible.
The TPOT and TPITOS will be administered by a trained and reliable observer (who is blind to whether the classroom teacher(s) is/are a member of the Core BSC Team or not) before the first Learning Session. The TPOT and TPITOS will be administered a second time – post-BSC – after the fourth Learning Session. We intend to administer the TPOT or TPITOS in up to 4 classrooms at a large site (2 classrooms with teachers who are on the Core BSC Team and 1 or 2 classrooms with teachers who are not on the Core BSC Team) and in 2 or 3 classrooms at a small site (1 or 2 classrooms with teachers who are on the Core BSC Team and 1 classroom with teachers not on the Core BSC Team). Whether classrooms receive a TPOT or TPITOS observation will depend on the ages of the children they serve.
Early Childhood Work Environment Survey (ECWES) (Attachment K). The Feasibility Study Team will use the Early Childhood Work Environment Survey, a tool that measures the organizational climate of early childhood programs. It was developed specifically for early care and education settings. The 15-minute survey includes questions about staff’s perceptions about various organizational practices. There are ten dimensions in the survey that will help the feasibility study team better understand the collective perceptions of staff at the entire ECE program. Specifically, the tool will help us decipher what things are going well in the program and better isolate areas that need to be strengthened. Example items about work attitude include asking the respondent to indicate if they “intend to work here at least two more years,” and “I sometimes feel trapped in my job.” The survey will be available through a web-portal hosted by New Horizons (the owner of the ECWES tool); participating staff will receive a link with login information and instructions for completing the survey. The ECWES will be completed with all staff (administrative, teaching, and support staff) at the ECE program who work more than 10 hours per week and who have consented to the Feasibility Study. For the most valid summary site profile, the scale should be completed with a minimum of five people per site. After all staff at the site who have consented to the Feasibility Study complete the ECWES or after 30 days (whichever comes first), an electronic Work Environment Profile summarizing the aggregate results will be available for the Feasibility Study Team to review.
Pre-/Post- Survey (Attachment L). As part of the feasibility study data collection, the Feasibility Study Team will develop and field a survey with staff at the early care and education sites who participate in the CCL project (both Core BSC Team members and non-Core BSC Team members). The survey will be web-based, although a paper/pencil option will be available if preferred by individual respondents. The purpose of the survey is to answer key implementation and outcomes research questions that cannot be answered through secondary measures review nor observation only. We plan to subsume some standardized, proprietary measures into the online survey so that from a staff person’s perspective, they are only completing one survey. These measures include background information from the Early Childhood Work Environment Survey, the Early Childhood Job Satisfaction Survey, the Psychological Safety survey, the Maslach Burnout Inventory, teacher and director self-efficacy items, and the Perceived Problems Questionnaire.
This survey will be administered before the first Learning Session and after the fourth Learning Session. The survey will take a total of 40 minutes to complete. The sample will include everyone on the Core BSC Team, plus additional program staff who consent to the CCL Feasibility Study, which may include up to two additional teachers and support staff for small sites and up to four additional teachers and support staff for large sites. Below is a list of the separate tools making up the full online survey.
Background Information. All respondents to the online survey will be asked about education level (highest level of education attained and the degree field), age, sex, race and ethnicity, and income (total household income level). These questions are from the Early Childhood Work Environment Survey and will be administered again in this pre-/post-survey so that responses can be captured by the Feasibility Study Team rather than aggregated at the site-level.
Early Childhood Job Satisfaction Survey (ECJSS). The feasibility study team will use the Early Childhood Job Satisfaction Survey to measure the discrepancy between existing and ideal working conditions as perceived by the employee. The tool measures five unique facets of the working environment. Conditions in the five areas may vary and contribute to an employee’s feelings of fulfillment in one area while feeling discouraged or discontent in other areas. This will measure in more detail what is contributing to an employee’s overall satisfaction or dissatisfaction with their job. Respondents are asked to rate 50 items using a Likert scale about the extent to which they agree or disagree with different aspects of their job. Example items include “I feel encouraged and supported by my colleagues” and “My supervisor respects my work.” In part II of the survey, respondents rate five aspects of their position using a Likert scale as being ideal or not ideal. Example items include “Relationship with co-workers” and “Relationship with supervisor.” In part III of the survey, respondents are asked to check three job characteristics they value most. Examples include, “Colleagues-working with people I like,” and “Security-the assurance that my position is secure.” Lastly, in part IV, respondents are asked to write what are their top two satisfactions and frustrations with their job. Each completed survey will result in a job satisfaction profile for staff in each of the five measured facets, as well as a metric about the employee’s congruence or incongruence with their ideal job, and qualitative information about each staff’s occupational values, and primary job satisfactions and frustrations.
Psychological Safety Survey. The feasibility study team will use a measure examining staff’s feelings of psychological safety and learning behavior in work teams. It measures the psychological safety or trust staff have in each other not to gain personal advantage at someone else’s expense. Example items include, “If you make a mistake at this center, it is often held against you,” and “Teachers at this center feel it is safe to take a risk (e.g. trying something new in the classroom).”
Maslach Burnout Inventory Educators Survey – Emotional Exhaustion subscale. The feasibility study team will use the Emotional Exhaustion subscale of the Maslach Burnout Inventory, a questionnaire to assess how staff members feel about their job and their reactions to work. The reason we are using items in this scale is to capture job burnout to better understand factors relating to successful participation in the BSC. The Maslach Burnout Inventory has three subscales in all: Emotional Exhaustion, Personal Accomplishments, and Depersonalization. To decrease burden on participants, the Feasibility Study team decided to only use the Emotional Exhaustion subscale, as it has the strongest psychometric properties of all the subscales (alpha = .90; Iwanicki & Schwab, 1981). Participants are asked to rate how often items are true for them. Sample items include “I feel emotionally drained from my work” and “Working with people all day is really a strain for me.”
Perceived Problems. The feasibility study team will use the Perceived Problems questionnaire, a checklist about teachers’ perceptions of day-to-day problems preschool teachers experience. The reason we are using selected items from this scale is to identify teachers’ areas of concern in social and emotional behavior in their classrooms. The full questionnaire includes 45 yes/no items, however for the purpose of the feasibility study, we propose administering only 10 items that pertain to social and emotional learning and development. Sample items include, “I have a problem getting children to do what I ask them to do,” and “I have a problem controlling the noise or energy level in the room.”
Beliefs and Background Questionnaire. Only teachers will be asked questions about professional development, including topic area of trainings attended in the past two years and the number of continuing education credits received within the past two years in the pre-/post-survey. There will also be a section in the survey about quality improvement initiatives the teacher has engaged in within the past two years and the teacher’s perception about how useful they were. Finally, this survey will capture teachers’ beliefs about social and emotional learning.
Teachers’ Sense of Efficacy Scale: CCL Adaptation. The feasibility study team will use an adaptation of the Teachers’ Sense of Efficacy Scale to assess self-efficacy in parents and teachers. The scale has been adapted for early childhood teachers. This scale will be used to better understand factors relating to successful participation in the BSC. Sample items include “How much can you do to make positive changes in your care setting?” and “How much can you do to calm a child who is upset?”
Directors’ Efficacy Scale from the Directors’ Role Perception Survey. The feasibility study team will use sections from the Directors’ Role Perception Survey to address self-efficacy in directors. Only subsections from the Directors’ Role Perception Survey will be used. This scale will be used to better understand factors relating to successful participation in the BSC. Topics addressed in this scale focus on leadership and management, including managing staff, using data to promote continuous quality improvement, and implementing developmentally-appropriate curriculum.
Self-report of BSC Activities (Attachment M). In the post-survey only, we will ask all participants who are teachers (both lead and assistant) to self-report on their participation in BSC elements (e.g., attending a Learning Session, training, receiving coaching from a BSC Implementation Team Member or Faculty, doing a PDSA, etc.). This is to understand the extent to which others outside of the Core BSC Team may have participated in BSC activities, and also, for those who are Core BSC Team Members, whether they report high levels of engagement in the BSC. Additionally, this section of the survey will ask participants about the experience of their core BSC team’s participation in the BSC (e.g., engaging in Learning Session interactions, soliciting ideas and feedback, using PDSAs, etc.). This is to capture the team learning climate, both within teams and between teams. The team learning climate will be captured using Nembhard’s intra-organizational and inter-organizational leadership activity items.
Core BSC Team Focus Group Topic Guide (Attachment N). As part of the feasibility study data collection, the feasibility study team will develop and field a focus group protocol with staff at the early care and education sites who participate in the Core BSC Teams. The purpose of the focus group is to answer key implementation and outcomes research questions that cannot be answered through secondary measures review nor observation only. Topics that will be covered by the focus group protocol include, but are not limited to: BSC elements that were beneficial and challenging for BSC Team Members, strengths and limitations of a 12-month timeline for the BSC, and perceptions and experiences with “spread”. Honoraria will be provided for participation in the focus group. This focus group will occur after the fourth Learning Sessions. Up to six members of each site’s Core BSC Team will participate in a focus group. For larger programs that may have larger Core BSC Teams, the focus group participants will be randomly selected from the full team.
Table A.2.2 Summary of timing, sample, and goal of study instruments
Instrument |
Timinga |
Sample |
Overall goal of instrument |
|
Fall 2017 |
1 individual from up to 18 sites applying to participate in the BSC
|
To assess readiness and interest in participating in the BSC |
|
Fall 2017/Winter 2018
|
6 individuals from each of the 8 sites participating in the BSC
|
To engage the team in developing a shared goal and purpose for their work together in the BSC
|
|
Fall 2017/Winter 2018
|
2 individuals from each of the 8 sites participating in the BSC
|
To establish a concrete plan for how to collect the metrics data for the BSC |
|
Winter 2018-Spring 2019
|
6 individuals from each of the 8 sites participating in the BSC
|
To help teams focus their plans for improvement by identifying a plan for each step of the process and identifying how it connects with the drivers from the change framework
|
|
Winter 2018-Spring 2019
|
6 individuals from each of the 8 sites participating in the BSC
|
To create a forum for ongoing sharing of ideas and collaborative problem-solving for improving practices and organizational capacity
|
|
Winter 2018-Spring 2019
|
6 individuals from each of the 8 sites participating in the BSC
|
To help team members reflect on their experience and provide feedback to the implementation team that can be used to improve the BSC
|
|
Winter 2018-Spring 2019
|
6 individuals from each of the 8 sites participating in the BSC
|
To help team members reflect on their experience and provide feedback to the implementation team that can be used to improve the BSC |
|
Winter 2018-Spring 2019 |
6 individuals from each of the 8 sites participating in the BSC |
To help teams identify key priorities for the upcoming Action Period |
|
Winter 2018 & Spring 2019 |
2 teachers from each of the 8 core BSC teams, plus an additional 2 teachers from large sites and 1 teacher from small sites
|
To assess teaching practice around social and emotional learning in order to (1) understand for whom successful participation in the BSC is feasible, and (2) assess the feasibility of detecting meaningful changes in teaching practice |
|
Winter 2018 & Spring 2019
|
9 individuals from each of the 8 sites participating in the BSC
|
To assess organizational climate in order to (1) understand for whom successful participation in the BSC is feasible, and (2) assess the feasibility of detecting meaningful changes in organizational climate
|
|
Winter 2018 & Spring 2019
|
9 individuals from each of the 8 sites participating in the BSC
|
To understand the organizational climate of early childhood programs and how teachers and caregivers experience their work. |
|
Spring 2019
|
9 individuals from each of the 8 sites participating in the BSC
|
To assess engagement in the BSC, and spread of BSC activities within an organization |
|
Spring 2019 |
6 individuals from each of the 8 sites participating in the BSC |
To answer key implementation and outcomes research questions that cannot be answered through secondary measures review nor observation only |
aAfter obtaining OMB approval.
Although the paper and pencil versions of the pre-/post- surveys will be available upon request, the survey will also be available online. Web-based instruments used for survey data collection will be programmed to automatically skip questions not relevant to respondents, thereby reducing burden. The web-based application also allows respondents to complete the survey at a time convenient to them.
Further, utilizing a web-based discussion forum for core BSC teams reduces burden by allowing participants to engage in discussion, ask questions, and offer feedback at a time convenient to them. Additionally, using a web-based discussion forum allows participants to choose whether they want to engage in certain discussions.
With regard to collecting qualitative data through focus groups, audio recorders will be used with permission from participants to later confirm direct quotes or other details from the focus groups.
None of the study instruments will ask for information that can be reliably obtained from alternative data sources, including administrative data collection. Furthermore, the design of the study instruments ensures that the duplication of data collected through each instrument is minimized. Finally, the Feasibility Study will utilize data collected as part of the BSC for secondary analysis to minimize duplication.
Information being requested or required has been held to the minimum required for the intended use. Most of the organizations included in the study will be small organizations, including child care centers and Head Start/Early Head Start programs.
Burden will be minimized for respondents by convening focus groups in a central location, and, to the extent possible, combining proprietary measures into a comprehensive survey to minimize the number of surveys participants complete.
We do not expect there to be consequences of the proposed timing of our data collection. In order to understand change over time in individual beliefs and behaviors, as well as organizational change in culture around continuous improvement efforts and supports for children’s social and emotional learning, we will collect some information at both the beginning and end of the BSC process (e.g., surveys, classroom observations). Other sources of information for the feasibility study are one-time data collections at the conclusion of the BSC process (e.g., BSC focus group).
There are no special circumstances for the proposed data collection efforts.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on July 28, 2017, Volume 82, Number 144, page 35213, and provided a 60-day period for public comment. A copy of this notice is included as Attachment T. During the Notice and comment period, ACF received and responded to two (2) requests for information about the project. No comments were received.
The contractor consulted with experts to complement the knowledge and experience of the team (Table A.8). Consultants included program administrators, policy experts, and researchers. Collectively, these consultants have specialized knowledge in continuous quality improvement methods, implementation science, evaluation, social-emotional development, Head Start administration, organizational contexts, and coaching. We also engaged experts with specialized knowledge and skills in the areas of research design and data collection methods relevant to this work.
Table A.8 CCL Project expert consultants
Name |
Affiliation |
Lindsey Allard Agnamba |
School Readiness Consulting |
Paula Jorde Bloom |
McCormick Center for Early Childhood Leadership at National Louis University |
Andrea Urbano |
Univ. of Mass. Donahue Institute |
Yvette Rodriguez |
Vice President, Head Start & Children Services; Action for Boston Community Development |
Debi Mathias |
BUILD Initiative |
Megan McClelland |
Oregon State University |
Karen Bierman |
Penn State University |
Adam Winsler |
George Mason University |
Allison Metz |
NIRN |
Shira Mattera |
MDRC |
Kathy Bigelow |
University of Kansas |
Judith Carter |
University of Kansas |
Mary Louise Hemmeter |
Vanderbilt University |
Angel Fettig |
Univ. of Mass. Boston |
Shannon Monahan |
Mathematica Policy Research |
Noreen Yazejian |
FPG Child Development Institute |
Karen Taylor |
FPG Child Development Institute |
Donna Bryant |
FPG Child Development Institute |
Shannon Wanless |
University of Pittsburgh |
Mary Catherine Arbour |
Harvard Medical School |
Sabrina Selk |
National Institute for Children’s Health Quality |
No incentives will be distributed to respondents for participating in this study.
Information collected will be kept private to the extent permitted by law. The consent statements provided to all study participants include assurances that the research team will protect the privacy of respondents to the fullest extent possible under the law, that respondents’ participation is voluntary, and that they may withdraw their consent at any time without any negative consequences.
The consent statement for the primary measures, which are surveys, focus groups, and interviews, will be provided in the instruments’ introductory sections. (Consent for secondary measures is not included because secondary measures are considered part of the BSC implementation and will only be used by the Feasibility Study team as secondary data/document review.) This text will be presented on the first page of the web-based survey after the respondent accesses it and included in the email containing the link to the survey for the ECWES (as this survey must be completed through an external site operated by New Horizons, the owner and publisher of this proprietary measure, per New Horizons protocol). For interviews and focus groups, this information will be read to respondents prior to beginning the focus group and interview questions. The interviewer or facilitator will read a consent statement that includes assurances that the information shared will be kept private and reported in a manner that will not identify individual respondents. Consent will be provided verbally by the respondent after the interviewer or facilitator has read the consent statement. All materials to be used with respondents as part of this information collection, including consent statements and instruments, will be submitted to Child Trends Institutional Review Board for approval.
As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
In order to gauge for whom and under what circumstances successful participation in the BSC is possible, the pre-/post- survey involves some sensitive questions around topics of perceived problems and income. Through the consent process, respondents will be informed that all responses will be kept private and they can choose not to answer a question if they wish.
Table A.12 summarizes the estimated reporting burden and costs for each of the study instruments included in this information collection request. The estimates include time for respondents to review instructions, search data sources, complete and review their responses, and transmit or disclose information. This information collection request is for two years. Figures are estimated as follows:
BSC Selection Questionnaire. We expect up to 18 individuals to complete the BSC selection questionnaire. The questionnaire will be submitted only one time and is expected to take 1 hour to complete. Thus, the total annual burden for participants is 9 hours.
Pre-work Assignment: Team Building Activities. We expect up to six BSC team members from each of the eight sites (n=48) to participate in the team building activities. The team building activities will occur only one time and is expected to last 1 hour. Thus, the total annual burden for participants is 24 hours.
Pre-work Assignment: Data Collection Planning Worksheet. We expect that two BSC team members from each of the eight sites (n=16) will formulate the Data Collection Planning Worksheet. The Data Collection Planning Worksheet is expected to take two hours to complete. Thus, the total annual burden for participants is 16 hours.
Plan, Do, Study, Act Planning Form & Tracker. We expect that six BSC team members from each of the eight sites (n=48) will complete the Plan, Do, Study, Act Planning Form & Tracker once a week for 11 months (48 times). The worksheet takes an average of 15 minutes to complete. Thus, the total annual burden for participants is 288 hours.
Discussion Forum Postings. We expect that up to six BSC team members from each of the eight sites (n=48) will post on the discussion forum once a week for 11 months (48 times). Posting is expected to take an average of 15 minutes. Thus, the total annual burden for participants is 288 hours.
Learning Session Day 1 Evaluation. We expect that six BSC team members from each of the eight sites (n=48) will complete the evaluation for each Learning Session (4 times). The evaluation is expected to take an average of ten minutes to complete. Thus, the total annual burden for participants is 16 hours.
Learning Session Overall Evaluation. We expect that six BSC team members from each of the eight sites (n=48) will complete the evaluation for each Learning Session (4 times). The evaluation is expected to take an average of 15 minutes to complete. Thus, the total annual burden for participants is 24 hours.
Action Planning Forms. We expect that six BSC team members from each of the eight sites (n=48) will complete the form after each Learning Session (4 times). The form is expected to take an average of 15 minutes to complete. Thus, the total annual burden for participants is 24 hours.
Teaching Pyramid Observation Tool (TPOT)/Teaching Pyramid Infant-Toddler Observation Tool (TPITOS). We expect to observe two teachers per Core BSC Team (n=16), plus two additional teachers at large sites and one additional teacher at small sites (n=12) for a total of 28 individuals. Beyond the direct observation, teachers will participants in an interview lasting 20 minutes. The observations will be completed two times for each teacher. Thus, the total annual burden for participants is 9 hours.
Early Childhood Work Environment Survey (ECWES). We expect to survey up to six BSC team members for each of the eight sites (n=48), plus four additional at large sites and two additional at small sites (n=24) for a total of 72 individuals. The survey will be administered twice and will take an average of 15 minutes to complete. Thus, the total annual burden for participants is 18 hours.
Pre-/Post Survey. We expect to survey up to six BSC team members for each of the eight sites (n=48), plus four additional at large sites and two additional at small sites (n=24) for a total of 72 individuals. The survey will be administered twice and will take an average of 41 minutes to complete. Thus, the total annual burden for participants is 49 hours.
Self-Report of BSC Activities. We expect to survey up to six BSC team members for each of the eight sites (n=48), plus four additional at large sites and two additional at small sites (n=24) for a total of 72 individuals. The survey will be administered once and will take an average of ten minutes to complete. Thus, the total annual burden for participants is 6 hours.
Core BSC Team Focus Group Topic Guide. We expect that up to six BSC team members for each of the eight sites (n=48) will participate in the focus group. The focus group will last an average of 1.25 hours. Thus, the total annual burden for participants is 30 hours.
Table A.12. Total annual burden requested under this information collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
|
BSC Selection Questionnaire |
18 |
9 |
1 |
1 |
9 |
$20.22 |
$181.98 |
Pre-Work Assignment: Team Building Activities |
48 |
24 |
1 |
1 |
24 |
$20.22 |
$485.28 |
Pre-Work Assignment: Data Collection Planning Worksheet |
16 |
8 |
1 |
2 |
16 |
$20.22 |
$323.52 |
Plan, Do, Study, Act Planning Form & Tracker |
48 |
24 |
48 |
.25 |
288 |
$20.22 |
$5,823.36 |
Discussion Forum Postings |
48 |
24 |
48 |
.25 |
288 |
$20.22 |
$5,823.36 |
Learning Session Day 1 Evaluation |
48 |
24 |
4 |
.17 |
16.32 |
$20.22 |
$329.99 |
Learning Session Overall Evaluation |
48 |
24 |
4 |
.25 |
24 |
$20.22 |
$485.28 |
Action Planning Form |
48 |
24 |
4 |
.25 |
24 |
$20.22 |
$485.28 |
TPOT/TPITOS |
28 |
14 |
2 |
.33 |
9.24 |
$20.22 |
$186.83 |
Early Childhood Work Environment Survey (ECWES) |
72 |
36 |
2 |
.25 |
18 |
$20.22 |
$363.96 |
Pre-/Post- Online Survey |
72 |
36 |
2 |
.68 |
48.96 |
$20.22 |
|
Self-report of BSC activities |
72 |
36 |
1 |
.17 |
6.12 |
$20.22 |
$123.75 |
Core BSC Team Focus Group Topic Guide |
48 |
24 |
1 |
1.25 |
30 |
$20.22 |
$606.60 |
|
|
|
|
|
801.64 |
|
$16,209.16 |
To calculate the annualized cost to respondents for the hour burden, we assume that the typical respondent will be Head Start grantee staff (including Head Start collaboration managers/administrators) and child care service providers (both directors and teachers). Based on May 2016 data on our expected respondents from the Bureau of Labor Statistics, we use a mean hourly wage of $20.22. Data can be found at https://www.bls.gov/oes/. There will be no direct cost to the respondents other than their time to participate in the data collection activities.
Honoraria will be provided directly to individual participants as compensation for their time participating in the study. Honoraria are “payments given to professional individuals or institutions for services for which fees are not legally or traditionally required in order to secure their participation,” (Graham, 2006). Based on OMB guidance, honoraria is the term most appropriate for payments to schools, teachers, and administrators, and is usually paid after participation as a token of appreciation (Graham, 2006). Honoraria will be available for early care and education (ECE) personnel completing the surveys, observations, and focus groups. Each focus group member will receive a $25 honorarium. The lead teacher in each classroom observed will receive a $25 honorarium at each of the two observation periods (pre- and post). An honorarium of $25 will be provided to ECE personnel for their completing the ECWES and a $25 honorarium for their completion of the survey at each of the two survey collection periods (pre- and post). These honoraria will be provided in the form of a gift card. A single honorarium will not be provided to each program to disperse to survey participants because (1) programs will not be able to verify whether staff have completed the online survey as they will not have administrative access to the online survey data, and (2) even if they could confirm survey completion, there would be additional burden placed on program staff for monitoring which employees had completed the online surveys and dispersing the honoraria across participants.
Table A.13 Proposed respondent honoraria for completion of feasibility study data collection activities
Activity/instrument |
Length of activity (in minutes unless otherwise specified) |
Amount |
|
|
Feasibility study data collection |
|
|
|
|
TPOT/TPITOS |
2.5 hours (includes interview and classroom observation) |
$25 honorarium per classroom (given to the lead classroom teacher) |
|
|
Pre- and post- survey |
41 |
$25 honorarium per respondent |
|
|
Early Childhood Work Environment Survey (ECWES) |
15 |
$25 honorarium per respondent |
|
|
Core BSC team focus group |
75 |
$25 honorarium per respondent |
|
|
|
|
|
|
|
The total cost for the data collection activities under this current request will be $1,082,663. Annual costs to the federal government will be about $541,331.50. This includes personnel effort (590 hours for four Class I Senior staff persons, 150 hours for a Class II Associate staff, 105 hours for a Class III Intermediate staff, and 855 hours for four Class IV Junior staff persons) plus other direct costs and indirect costs, and a fee. The total direct costs are $879,407, the total indirect costs are $132,428, and our fee is $70,828. Cost calculations are based on budgeted Child Trends wages for each Class of project staff. These wage calculations were based on the planned data collection activities for the feasibility study, as detailed in the contract between Child Trends and OPRE for this project. To calculate the estimated cost numbers, the average staff wage for each Class was calculated and multiplied by the total number of hours allocated to that Class for the data collection activities to create a cost estimate for each Class. Finally, estimates for all four Classes were added together as the total estimated cost to the federal government.
This is a new data collection.
We have developed a coding rubric which will help organize the information as it relates to our research questions and their related hypotheses. Our rubric corresponds to the Hexagon Tool originally developed by the National Implementation Research Network (NIRN) to document feasibility (Blase, Kiser, & Van Dyke, 2013). The original Hexagon Tool is typically used as a self-assessment tool by program staff to determine “fit and feasibility” of an innovative or evidence-based practice for adoption in a particular site. Each staff member rates the program’s “fit and feasibility” to adopt the new practice, and then the staff meets as a group to discuss ratings and come to consensus about whether to move forward with the innovation or new practice. We have modified both the original NIRN rubric and the rating process for the purposes of this Feasibility Study.
For this Feasibility Study, we will use a combination of analytic strategies, including case description, theory testing, and examination of rival explanations; we will also use a combination of analytic techniques, including pattern matching and cross-case synthesis. The hypotheses associated with our 10 research questions lend themselves to theoretical testing; we will use pattern matching to detect whether we see evidence to support the hypotheses associated with each of our 10 research questions. However, we will also be open to alternative explanations for the patterns we find, and seek to provide a general description of how the BSC was implemented in this set of eight early care and education sites in Boston. Because we are interested in how the BSC functions to support quality improvement in social and emotional learning practices in both Head Start and child care settings, we will use the coding rubric to code evidence specific to each program participating in the BSC separately, and look for similar patterns across Head Start programs within our sample of four programs. Similarly, we will look for patterns across the four child care programs participating in the BSC. We will also be looking to document patterns that emerge across the eight programs participating in the BSC regardless of auspice, and also look for patterns across other characteristics of the participating programs using cross-site analysis and synthesis.
To make sure everyone is using the coding rubric and scoring system in the same way, all coders will code the same few data sources and meet to reconcile as a group. Then, between 10% and 20% of the remaining data sources will be double coded by two feasibility study team members to determine reliability of the use of the coding rubric. Only coders who can reach 85% reliability with other coders will be permitted to continue coding on their own. Once all evidence for each program involved in the BSC is coded, Feasibility Study staff will convene to evaluate the collective evidence that was brought to bear to reach consensus on whether there is sufficient evidence to characterize the BSC as “successful” in bringing about a change in individual practice and organizational culture regarding support for social and emotional learning. As noted earlier, we will examine evidence as it pertains to each program individually, and also look to compare patterns across programs for a cross-site comparison and synthesis.
Table A.6 contains the timeline for the data collection and reporting activities. Data collection is expected to occur between winter 2018 and summer 2019, after obtaining OMB approval. Child Trends will produce a final report.
Based on the analysis of the data from this feasibility study, we will make recommendations for whether to move forward with a larger scaling of the BSC model in child care and Head Start settings. We also will summarize “lessons learned” from conducting the feasibility study that will inform the design of an evaluation study of a larger-scale implementation of the BSC with an expanded set of programs, potentially in other geographic locations.
Table A.16. Schedule for the Culture of Continuous Learning Project Implementation & Feasibility Study
Activity |
Timinga |
BSC implementation & feasibility study data collection |
|
|
|
BSC selection questionnaire Pre-work assignment: Data collection planning form |
Winter 2018 Spring 2018 |
Pre-work assignment: Team building activities |
Spring 2018 |
Action planning form |
Summer 2018 – Spring 2019 |
Plan, Do, Study, Act form & tracker Discussion forum postings Learning Session day 1 evaluation Learning Session overall evaluation TPOT/TPITOS |
Summer 2018 – Spring 2019 Summer 2018 – Spring 2019 Summer 2018 – Spring 2019 Summer 2018 – Spring 2019 Summer 2018 & Spring 2019 |
Early Childhood Work Environment Survey (ECWES) |
Summer 2018 & Spring 2019 |
Pre- and post- survey |
Summer 2018 & Spring 2019 |
Self-report of BSC activities |
Summer 2019 |
Core BSC team focus group |
Summer 2019 |
Analysis |
|
Data processing and analysis for feasibility study report |
Summer 2019 |
Reporting |
|
Feasibility study report |
Winter 2020 |
Final report |
Spring 2020 |
aAfter obtaining OMB approval
All instruments will display the expiration date for OMB approval.
No exceptions are necessary for this information collection.
References
Blase, K., Kiser, L. & Van Dyke, M. (2013). The Hexagon Tool: Exploring Context. Chapel Hill, NC:
National Implementation Research Network, FPG Child Development Institute, University of North
Carolina at Chapel Hill.
Bureau of Labor Statistics (2016). Occupational Employment Statistics: Occupational Employment and Wages. Washington, DC: Bureau of Labor Statistics. Retrieved from https://www.bls.gov/oes/
Douglass, A. (2016). Innovations in early educator professional development: Collaborative
learning and improvement. Paper presented at the Annual Meeting of the American
Educational Research Association, Washington, D.C.
Graham, J.D. (2006). Questions and answers when designing surveys for information collections (OMB memo). Washington, DC: U.S. Retrieved from http://www.cio.noaa.gov/itmanagement/pdfs/OMBSurveyGuidance_0106.pdf
Iwanicki, E. F., & Schwab, R. L. (1981). A Cross Validation Study of the Maslach Burnout Inventory 1 2. Educational and psychological measurement, 41(4), 1167-1174.
Metz, A., Naoom, S.F., Halle, T.G., & Bartley, L. (2015). An integrated stage-based framework for implementation of early childhood programs and systems (OPRE Research Brief 2015-48). Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Retrieved from https://www.acf.hhs.gov/programs/opre/resource/an-integrated-stage-based-framework-for-implementation-of-early-childhood-programs-and-systems
Office of Information and Regulatory Affairs (2016). Questions and Answers When Designing Surveys for Information Collection. Washington, DC: Office of Management and Budget, Office of Information and Regulatory Affairs. Retrieved from https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |