Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase 1

Attachment D

Implementation Interview protocol

ECE-ICHQ Implementation
Interview Protocol

Fall 2015

Instructions to Interviewer:

This protocol should be tailored for specific centers/sites based on documents you receive, the site’s answers to SAQ questions, and the site’s answers to questions in this protocol. Before the interview, you will need to carefully review the completed SAQ and any available documents. If you are able to answer a question completely using materials you have already received from the site, or if a question does not apply to a particular site, do not ask the question.

Request that the center pull together the following documents (if available) ahead of time so you can have them to review during the visit. If possible, conduct the document review prior to the interview to tailor the questions. You may need to skip some questions during the interview if the respondent indicates that a document is available for review.

• Organizational chart or staffing plans

• Center and classroom schedules

• Staff handbook

• Standard operating procedures

• Performance appraisal forms

• Written guidance on regulations or standards staff must follow (for example, licensing regulations, program specific standards such as Head Start Program Performance Standards, or standards for a quality rating and improvement system)

• Tools used to monitor and document compliance with regulations or standards

• Reporting tools staff use to track or document compliance with regulations or standards that apply to the classroom

• Mission statement

• Strategic plan

• Quality improvement plan

• Training, professional development, coaching, or technical assistance plan(s)

• Staff input form/tool (used for staff to evaluate the center or its services)

• Parent input form/tool (used for parents to evaluate the center or its services)

The interview may occur in multiple sessions over the course of the site visit. The first time through, skip questions that will require the respondent to gather information or that you can obtain through a review of an existing document. Complete as much of the interview without interruption as possible. Conduct any additional document review and then complete the interview with the respondent.

Document the start and end times for each session that you conduct the interview on the “Implementation Interview Time Log” at the end of the protocol. Section questions may be asked across different interview occasions; note the sections from which you ask questions during each session.

Ask the cognitive interview questions (section VI) once a section is complete. Ask the final cognitive interview questions at the end of the interview.

Introductory script:

Thank you very much for agreeing to participate in this discussion. Your participation is very important to the Assessing the Implementation and Cost of High Quality Care and Education study. As you may already know, the Administration for Children and Families within the U.S. Department of Health and Human Services is sponsoring the study to measure the cost of operating quality early education programs and they have contracted with Mathematica Policy Research, an independent research organization, to design and conduct the study. I’m [NAME] and I work for Mathematica.

As part of the study, we are conducting this interview to learn more details about your organization and what your center does to support the care and development of young children. This interview collects information about staff selection and support, the use of tools to operate your program, training, coaching, and technical assistance activities, and other activities that can support quality improvement. We are focused on learning about the early care and education services for children from birth to age 5 offered by your organization at this address only. We use the term “center” throughout this interview to refer to this location.

Your participation in this interview will help us understand the activities and resources that can support quality early care and education. Your participation is voluntary and you may skip questions you do not want to answer. Information you provide will be treated in a private manner, and the study will not identify individuals or centers in any of its reports.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and it expires 03/31/2018. The interview will take approximately 2.5 hours in total, but we may complete it at different times over the course of our visit. Not every question will apply to your center and there are no right or wrong answers. Because this is an exploratory study, we also want to hear about your perceptions of the questions we are asking. We will pause for some reflective questions after each key topic.

Do you have any questions before we get started?

I would like to record our discussion so I can listen to it later when I write up my notes. No one outside of our research team will listen to the recording. If you want to say anything that you don’t want recorded, please let me know and I will be glad to pause the recorder. Do you have any objections to being part of this interview or to my recording our discussion?

I. Staff selection

We’d like to start by discussing the hiring process and staff turnover.

A. The hiring process

1. How does your center get the word out about open positions?

1. Would you say there is an informal way of searching for staff such as word of mouth through existing staff, or posting a sign? Please describe.

2. Is there a more formal way of searching for staff such as through print or online advertisements? Please describe.

1. If so, do you specify education requirements and other qualifications in job announcements and descriptions?

3. Do the methods vary based on the type of position, such as a director or classroom staff? Please describe.

2. Who is involved in outreach, advertising, or recruitment for open positions?

1. Does this vary depending on the type of position for which you are hiring? Please describe.

3. About how many applicants do you typically receive for the following positions:

• Center director or site administrator?

• Education or curriculum specialists?

• Staff who work directly with children in the classroom? [ask about lead teachers and assistants separately]

4. Does someone do an initial review and screening of resumes to identify applicants that meet minimum qualifications?

1. If so, who?

2. Does this vary depending on the type of position for which you are hiring? [ask about each type of position: director, education specialist, classroom staff]

5. Do you typically bring in multiple applicants to interview for the following positions:

• Center director or site administrator?

• Education or curriculum specialists?

• Staff who work directly with children in the classroom? [ask about lead teachers and assistants separately]

For each position with a “yes” then ask:

1. Who is involved in conducting interviews?

2. Who is involved in the assessment and discussion of candidates?

3. Who is involved in making the final hiring decision?

6. Who is ultimately accountable for selecting and hiring staff?

a) In what ways, if any, are they held accountable?

7. What happens during the interview process specifically for lead teachers? [Allow the respondent to describe the process first, follow-up as needed to address the specific questions that follow]

1. Do you discuss scenarios of classroom situations? If so, how?

2. Must candidates prepare or present a lesson or age-appropriate activity for discussion? For observation?

3. In what ways, if any, do you discuss what quality education and care means to a job candidate?

8. What, if any, other processes that we have not yet discussed do you have in place at this center to help you hire staff with the qualifications you are seeking?

9. In the past 12 months, have you hired staff to fill a position(s) associated with implementing a new initiative to help improve quality (for example, a coaching model, or adoption of a new curriculum or assessment tool)? [If no, skip to question 9. If yes, ask a and b below.]

1. Who determined the position requirements?

2. In what ways did the hiring process differ from that described above for staff who work directly with children ages 0-5?

10. How successful do you feel you are in hiring staff who have the qualifications and skills you are seeking? Why do you think you are successful or not?

11. What are the reasons you have hired or have current staff who do not fully meet the qualifications or skills desired for their position? [Use responses from question F10 on the SAQ to tailor this question]

1. If so, what, if any, plans do you have in place for staff who do not meet the requirements to help them to meet them?

12. What is the biggest challenge you face in filling vacant positions for staff who work directly with children ages 0-5?

B. Staff turnover

1. [Reference response to question F14 on the SAQ about how many individuals who work directly with children ages 0-5 have left the center by their own decision during the past 12 months to tailor this question.]

1. [If there has not been much turnover] What factors do you think have enabled/encouraged staff to stay?

2. [If there has been a lot of turnover] What are the main reasons staff have left? What factors do you think may have encouraged staff to leave?

2. [Reference response to question F15 on the SAQ about how many individuals who work directly with children ages 0-5 have been terminated during the past 12 months when asking this question.] What are the reasons for termination?

3. Are there particular positions or levels of staff that are more difficult to retain than others?

1. Are you able to retain people whom you think of as highly qualified?

2. Do you have difficulty retaining teaching staff once they have obtained a higher degree or credential?

4. When turnover occurs, how does it affect daily operations/work load?

1. How are these operational challenges addressed?

5. What steps or strategies have you taken in the past 12 months to retain current staff and prevent future turnover?

II. Selection and Use of Tools to support key functions

[Interviewers: much of the information about tools can be collected with the SAQ and through document review. This section of the protocol should be used to fill in any missing information and to ask follow up questions. Please review answers to the relevant questions in the SAQ and relevant documents to tailor this section.]

Now we’d like to discuss any resources or tools you may use to support certain activities in your center.

A. Center administration

Let’s start with resources or tools you may use to assist with administration of the center.

Information about policies and standard operating procedures

1. [Review question H1 on SAQ and ask questions to fill in missing information, as needed, about written information provided to staff about benefits or performance appraisal, such as a staff handbook. Request hard copies of any written information to fill in topics covered below.]

What topics are covered by written information for staff about things such as benefits or performance appraisal processes (for example, a staff handbook)?

Check all that apply:

• Expectations for staff (such as hours and conduct)

• Benefits for staff (such as health insurance and paid time off)

• Policies or procedures for staff development and performance appraisal

• Purpose or mission statement

• Other, specify

2. [If answer to question H1 on SAQ is “no”] In what ways, if any, is information about benefits or the performance appraisal process communicated to staff?

3. [Review question H2 on SAQ and ask questions to fill in missing information, as needed, about written information about policies and procedures, such as standard operating procedures. Request hard copies of any written information to fill in topics covered below.]

What topics are covered by written information about policies and procedures?

Check all that apply:

• Hours of operation

• Child to staff ratios and group size limits

• Health and safety procedures

• Other, specify

4. [If answer to question H2 on SAQ is “no”] In what ways, if any, are staff informed about policies and procedures they should or must follow?

Meeting requirements typically associated with licensing

5. In what ways, if any, are staff informed about the need to meet requirements about:

1. Child to staff ratios and group size limits

2. Health and safety procedures

3. Other requirements that may be part of licensing compliance in your state (if applicable)

6. What tools or processes, if any, do you use in this center to monitor and document staff compliance with these requirements?

1. Describe (or name) tool/process

2. How long has it been in place?

3. How is the information used for program administration?

4. How is the information used in communicating changes to staff, when necessary?

5. How often is the tool/process reviewed or updated?

7. Are there any other specific reporting tools or processes staff in this center who work directly with children ages 0-5 must use to track or document compliance with these requirements?

1. Name of tool

2. How long has tool been in place?

3. How often is the tool/process reviewed or updated?

Meeting other standards

8. How are staff in this center made aware of requirements or standards they must follow for: [include all that apply based on responses to questions D1, D2, and E2 on SAQ that indicate whether the center has Head Start or preK classrooms, participates in a QRIS, or is accredited]

• Head Start Program Performance Standards

• As a state preK-funded program

• For participation in [insert QRIS name]

• As an accredited center

1. Is there written guidance? [If yes, ask a and b. If no, determine how they are made aware, then go to question 9.]

2. How are requirements or standards distributed to staff?

3. How often are requirements or standards distributed to staff?

9. What tools or processes, if any, do you use in this center to monitor and document staff compliance with these standards? [Specify differences across sets of standards]

1. Describe (or name) tool/process

2. How long has it been in place?

3. How is the information used for program administration?

4. How often is the tool/process reviewed or updated?

10. Are there any other specific reporting tools or processes staff in this center who work directly with children ages 0-5 must use to track or document compliance with these standards? [Specify differences across sets of standards]

1. Name of tool

2. How long has tool been in place?

3. How often is the tool/process reviewed or updated?

11. Are there any other programs or funding sources not yet discussed for which the center must adhere to certain requirements or standards beyond licensing?

[If yes, then ask]

1. How long have these standards been in place in your center?

2. In what ways, if any, do the process or tools for monitoring compliance with these standards differ from those already discussed?

3. How is the information used for program administration?

Adaptability of policies and procedures

12. Please describe the process for determining whether changes in policies or procedures are needed to support new initiatives or quality improvement activities?

1. How is the need for change assessed?

2. Who is involved? Are staff who work directly with children ages 0-5 involved?

13. How flexible is the center in changing or developing policies and procedures to support new initiatives or quality improvement activities? Please provide an example or two.

14. What characteristics of your center facilitate the ability to change or adapt policies and procedures to changing needs?

15. What characteristics can impede the ability to change or adapt policies or procedures?

B. Center purpose and planning

Now let’s talk about processes and tools used for program planning.

Purpose or mission statement

1. [Review question H3 on SAQ and ask questions to fill in missing information, as needed, about a written purpose or mission statement. Request a hard copy of statement to fill in concepts covered below.]

What concepts are mentioned in the purpose or mission statement?

Check all that apply:

• Quality of care

• Child development

• School readiness

• Parent engagement

• Community engagement

• Meeting needs of families/community

• Other, specify

2. In what ways, if any, is a message about the purpose or the focus of the center discussed or communicated with staff who work directly with children? Please describe.

Program planning

3. What information or process helps you make planning decisions for this center—such as planning for growth either in the number of children served or staff needed, achieving goals, or planning improvements?

4. Would you say there is an informal process for program planning? If so, what does this process look like and who is involved? How often does it occur?

5. Do you make use of any written plans or tools to assist in program planning? If so, what are they? Does this center have a strategic plan, for example? [Request a hard copy of any planning documents or tools to fill in topics covered below.]

What topics are covered in the strategic plan or other planning tool?

Check all that apply:

• Enrollment targets

• Staff professional development goals

• Leadership activities

• Quality improvement plans or goals, specify

• Other, please specify

For each planning tool, ask:

1. Who was involved in developing the strategic plan or other planning tool?

2. When was the current version created or updated?

3. To which staff are the activities or goals of the strategic plan communicated?

Check all that apply:

• Administrators

• Teachers

• Assistant teachers

• Aides

• Specialists

1. How are the activities and goals of the strategic plan communicated to staff?

2. How often is the plan or other tool reviewed for progress?

3. How often is the plan or other tool updated?

4. Does this center have any kind of quality improvement plan? If yes, then ask:

1. What are the main areas or activities targeted for improvement? [Request a hard copy of any quality improvement plans to document areas or activities targeted for improvement but ask respondent about key areas.]

2. Was the plan developed as part of participation in a particular program or initiative (for example, as part of participation in QRIS)? If yes, specify the program or initiative.

3. Who was involved in developing the plan? Specifically ask: Were classroom staff involved and, if so, how?

4. How often is the plan reviewed and updated?

5. How is progress tracked and by whom?

6. In what ways, if any, is the quality improvement plan shared with staff?

C. Instructional planning and child assessment

Now I’d like to discuss what processes or tools help staff who work directly with children in the classroom decide what activities to do and how.

Instructional planning and child assessment for infants/toddlers.

Let’s start by focusing on what is used in rooms with infants and toddlers (less than 36 months). [If the center does not serve infants or toddlers then skip to question C7.]

1. What information or resources, if any, are provided to staff who work directly with infants and toddlers (less than 36 months) to help them decide what activities to do with the children? [If information or resources other than a specific curriculum are noted, then ask the questions a – c. Specific questions about curriculum will be asked below.]

1. How were these resources developed?

2. How long have they been in place?

3. Are they used within all rooms with infants and toddlers? By all staff who work directly with infants and toddlers?

[If not used consistently across rooms or teachers] In what proportion of the rooms with infants and toddlers is this resource used? What are the reasons for the variation in use?

2. I see that you noted [curriculum name from question G1 on SAQ] as something that is used to guide activities for infants and toddlers. [Ask questions a – f below for each curriculum mentioned either in response to general question (#1 above) or as SAQ follow-up.]

1. How was this curriculum selected (or developed internally) and by whom?

2. Who is trained on the curriculum? Select one and explain, if applicable

• All staff who work with infants and toddlers

• Just some staff, explain

3. How are staff trained on the curriculum and by whom?

4. Is training provided to keep people current on the curriculum?

[If yes] If there is ongoing training, how frequently does it occur?

5. [Refer to question G1c on SAQ, if answer is NO ask the following] In what proportion of the rooms with infants and toddlers is this curriculum used? What are the reasons for the variation in use?

3. What tools or processes, if any, do staff use to inform the care and instruction of infants and toddlers (children less than 36 months)? These can include anything used to conduct screening, monitoring, or assessment for a range of purposes. [Refer to question G2 in the SAQ. If processes or tools other than those reported on SAQ are mentioned, ask the same set of questions as those on the SAQ.]

4. I see that you noted [name] as something that is used to inform the care and instruction of infants and toddlers. [Ask questions a – g below for each process/tool mentioned either in response to general question (#3 above) or as SAQ follow-up.]

1. [If response to G2b is “other, specify” or is otherwise unclear, clarify or confirm the purpose] What is the purpose of the tool/process? Is the tool linked with the curriculum used?

2. How was this tool selected (or developed internally) and by whom?

3. Who is trained on the process or tool? Select one and explain, if applicable

• All staff who work with infants and toddlers

• Just some staff, explain

4. How are staff trained on the process or tool and by whom?

5. Is training provided to keep people current on the process or tool?

[If yes] If there is ongoing training, how frequently does it occur?

6. [Refer to question G2c on SAQ, if answer is NO ask the following] In what proportion of the rooms with infants and toddlers is this tool used? What are the reasons for the variation in use?

5. In what ways, if any, are staff who work with infants and toddlers supported in interpreting and using information they gather about children to guide what they do in the classroom or inform how they interact with children?

1. Do staff have access to written guidance or support materials?

2. Do staff work collaboratively with each other, or with someone else, to assess results and plan activities? [If yes, then ask questions below]

   1. Who works together?

   2. How frequently?

6. What other standardized tools, templates, or forms are used to help staff who work directly with infants and toddlers plan activities, or track and report on child progress? For example, are there written daily reports to parents, or other forms that provide information to parents about child activities or progress?

1. What is the name and purpose of the tool/template/form?

2. Who developed it?

3. Is it supported by an automated system?

4. [If applicable, based on answers to questions above about curriculum/assessment] Is it linked with the curriculum or child assessments in use?

5. For how long has it been used?

6. What is the frequency of its use?

7. Is it used within all rooms that serve infants and toddlers? If not, why not?

8. Is it used by all staff who work with infants and toddlers? If not, why not?

Instructional planning and child assessment for preschoolers

Now let’s talk about what is used in rooms with preschool children (ages 3-5). [If the center does not serve preschoolers then skip to question C13.]

7. What information or resources, if any, are provided to staff who work directly with preschool children (ages 3-5) to help them decide what activities to do with the children? [If information or resources other than a specific curriculum are noted, then ask the questions a – c. Specific questions about curriculum will be asked below]

1. How were these resources developed?

2. How long have they been in place?

3. Are they used within all rooms with preschoolers? By all staff who work directly with preschoolers?

[If not used consistently across rooms or teachers] In what proportion of the rooms with preschoolers is this resource used? What are the reasons for the variation in use?

8. I see that you noted [curriculum name from question G3 on SAQ] as something that is used to guide activities for preschoolers. [Ask questions a – f below for each curriculum mentioned either in response to general question (#7 above) or as SAQ follow-up.]

1. How was this curriculum selected (or developed internally) and by whom?

2. Who is trained on the curriculum? Select one and explain, if applicable

• All staff who work with preschoolers

• Just some staff, explain

3. How are staff trained on the curriculum and by whom?

4. Is training provided to keep people current on the curriculum?

[If yes] If there is ongoing training, how frequently does it occur?

5. [Refer to question G3c on SAQ, if answer is NO ask the following] In what proportion of the rooms with preschoolers is this curriculum used? What are the reasons for the variation in use?

9. What tools or processes, if any, do staff use to inform the care and instruction of preschoolers (children ages 3-5)? These can include anything used to conduct screening, monitoring, or assessment for a range of purposes. [Refer to question G4 in the SAQ. If processes or tools other than those reported on SAQ are mentioned, ask the same set of questions as those on the SAQ.]

10. I see that you noted [name] as something that is used to inform the care and instruction of preschoolers. [Ask questions a – g below for each process/tool mentioned either in response to general question (#3 above) or as SAQ follow-up.]

1. [If response to G4b is “other, specify” or is otherwise unclear, clarify or confirm the purpose] What is the purpose of the tool/process? Is the tool linked with the curriculum used?

2. How was this tool selected (or developed internally) and by whom?

3. Who is trained on the process or tool? Select one and explain, if applicable

• All staff who work with preschoolers

• Just some staff, explain

4. How are staff trained on the process or tool and by whom?

5. Is training provided to keep people current on the process or tool?

[If yes] If there is ongoing training, how frequently does it occur?

6. [Refer to question G4c on SAQ, if answer is NO ask the following] In what proportion of the rooms with preschoolers is this tool used? What are the reasons for the variation in use?

11. In what ways, if any, are staff who work with preschoolers supported in interpreting and using information they gather about children to guide what they do in the classroom or inform how they interact with children?

1. Do staff have access to written guidance or support materials?

2. Do staff work collaboratively with each other, or with someone else, to assess results and plan activities?

[If yes, then ask questions below]

1. Who works together?

2. How frequently?

12. What other standardized tools, templates, or forms are used to help staff who work directly with preschoolers plan activities, or track and report on child progress? For example, are there lesson plan templates or forms that provide information to parents about child activities or progress?

1. What is the name and purpose of the tool/template/form?

2. Who developed it?

3. Is it supported by an automated system?

4. [If applicable, based on answers to questions above about curriculum/assessment] Is it linked with the curriculum or child assessments?

5. For how long has it been used?

6. What is the frequency of its use?

7. Is it used within all rooms that serve preschoolers? If not, why not?

8. Is it used by all staff who work with preschoolers? If not, why not?

Development screening and assessment

13. [Review responses to G2 and G4 on SAQ to determine if development screening and assessments are conducted for children ages 0-5. Ask the following questions, as needed, for clarity.] What process, if any, do you have in place to identify children with development delays and connect them with the necessary services?

1. Do you conduct development screening in the center? If not, are children referred somewhere else for screening?

2. Do you conduct development assessments if a child screens positive for a possible delay, or do you make a referral for further assessment?

3. When a child is determined to have a development delay, what role, if any, does the center play in connecting or providing him or her with the necessary services?

Planning and coordination

14. How do staff who work directly with children ages 0-5 work together or with other staff (such as specialists or managers) to plan or coordinate instruction or other classroom activities focused on quality improvement?

1. On what topics do staff coordinate?

2. Who is involved in these interactions?

3. What is the format of this coordination?

1. Is there planned group meeting time for specific staff?

2. Does it occur on more of an ad-hoc basis?

3. Is there an automated system that supports planning and coordination?

4. With what frequency does this occur?

5. Does this type of coordination occur consistently across staff who work with children ages 0-5, or does it vary (for example, based on the age of children served or type of staff such as lead teachers or aides)? Please explain.

15. Do staff who work directly with children ages 0-5 get paid planning time, when teachers are not responsible for children?

1. Is this provided to all staff or only some staff? Please explain differences.

2. Approximately how much time is provided? Specify for different staff if it varies.

D. Child and family support

Next, let’s discuss tools used to provide children and their families with supports or services.

1. [Review responses to section I in the SAQ about supports or services available to children and tailor this question appropriately. Ask the questions below about each support or service reported in the SAQ.]

1. What is the process for connecting and providing children with this support or service?

1. Are teaching staff trained to identify, connect, or provide children with this service or are specialized staff involved in the process?

2. Are specific tools used to identify, connect, or provide children with this service (for example, electronic systems or hard copy forms)?

2. How is information about the support or service provided to children used to inform classroom practice or other activities at the center?

E. Information systems and use of technology

1. Are there electronic systems or technology that you use in this center to support program operations or the care of young children beyond those we have already discussed (such as for curriculum, assessment, or planning)?

III. Training

Now I’d like to switch gears and talk about any training that occurs.

Let’s start with training that may happen for newly hired staff.

A. Training for new hires

1. In what way, if any, does your center orient new staff to the center and to general procedures? [Refer to question K1 on the SAQ to tailor question and clarify or fill in missing information.]

1. Who conducts this orientation?

[If answer to question K1 on SAQ is no, then ask] How does the orientation process differ across new staff?

2. What process, if any, does your center use to orient new staff who will work directly with children ages 0-5 to the classroom practices and activities, and the supports available for children and their families?

1. Is there a standard process followed for all new classroom staff or does it differ by type of staff or age of children cared for? (for example, teachers, assistants, aides, or for staff in infant/toddler classrooms vs. classrooms for preschool age children)

1. Who is involved with orienting new teaching staff?

2. Do they shadow existing staff or observe the children and routines in their assigned classrooms before fully assuming their responsibilities?

3. Do new teaching staff hires have a specified probationary period? If so, what criteria must be met within the probationary period?

3. [Refer to question K2 on the SAQ to tailor question and clarify or fill in missing information.] For initiatives or practices for which new staff must be formally trained prior to or soon after the start of working with children at your center:

1. Why is this training required, and by whom? For example, is the training required by a specific program or funder with which the center is affiliated?

2. How is the training provider selected? Are there certain criteria that are used?

B. Ongoing training and professional development

Now let’s talk about training and professional development for current staff.

1. Do you have access to training and professional development opportunities for your staff, including both in-person and online opportunities?

1. Would you say you have a great deal of access, sufficient access, limited access, or no access?

2. [If limited or none, then ask] What restricts access to training and professional development?

[If no access, skip to next section]

2. How do you make decisions at your center about the training and professional development needs of staff who work directly with children ages 0-5? [Lead with this general question, then ask specific questions below as needed.]

1. Do you develop a training plan for each staff member or for categories of staff?

2. Who develops training plans or makes decisions about training?

3. What information is used to identify training needs and make decisions about training?

Check all that apply

• Information from the interview process

• Staff supervision and performance assessment

• Formal or informal classroom observations

• Regulatory or other requirements

• Other, specify

4. On what criteria are decisions about who will receive training and when based?

Check all that apply

• Time

• Resources

• Staff performance

• Other, specify

5. How often are plans or decisions about training reviewed and updated

3. Do you have some current priorities for training and professional development of staff who work directly with children ages 0-5 at your center?

1. What are the priorities?

2. Who set the priorities?

3. How were the priorities set?

4. Do these priorities differ across the staff? If so, in what ways? (for example, based on the age of the children they work with or the education level of staff)

4. In what ways, if any, do staff who work directly with children ages 0-5 have input into setting priorities for training and professional development?

1. Which staff contribute input?

2. How do staff contribute (for example, through surveys, focus groups, informal discussions)?

5. How do staff who work directly with children make a request to pursue training, professional development, or educational opportunities? (For example, is there a formal process or form or is it done more informally?

6. What information, if any, do you require from staff to track progress in or completion of a training or professional development activity?

7. When new training or development opportunities come to your attention what criteria, if any do you use to decide whether to pursue them and for which staff?

8. [Refer to question K3 in SAQ to tailor these questions and fill in missing information for each training held.] We have some follow-up questions about the training sessions for management, supervisory staff, or staff who work directly with children ages 0-5 that have been held over the past 12 months?

1. You noted that [enter answer to SAQ question K3f] provided the training. On what criteria was this individual or entity selected?

2. How have you used information from follow-up activities, tracking of indicators, or staff evaluations about the success of training?

1. Is information shared with coaches or supervisors to target competency development?

2. Is information used in selecting and hiring staff with the skills and competencies needed?

3. Is information used to adjust training priorities as a whole or plans for individual staff?

4. Does the information feed into the performance appraisal process for individual staff?

9. In the past 12 months, have staff who work directly with children ages 0-5 visited other classrooms to observe new practices or skills? [If no, go to next question. If yes, ask the questions below.]

1. For what purpose?

2. Which staff observed other classrooms?

3. Where did the observations occur?

Select one

• Within same center

• At a different center within same program

• At a different center, different program

• Other, specify

4. With what frequency did the observations occur?

5. What happened after the observation(s)?

6. How was classroom coverage arranged for staff to participate in observations?

7. Were staff paid for their time spent observing other classrooms?

C. Resources for and success of training

1. Is a system in place that allows staff who work directly with children ages 0-5 to be released from time in their classrooms to participate in training or professional development activities? If yes, then ask the following questions:

1. Does the center arrange classroom coverage so that care can be provided to children while staff participate in these activities?

   1. Who provides coverage (for example, other staff members, substitutes who are contracted for this purpose)?

2. What expenditures, if any, are made by the center to provide this coverage?

   1. Are funds budgeted specifically for this purpose?

2. How do you determine whether the training your staff receives meets your program goals and their professional development needs?

3. In the past 12 months, have you been able to address the training priorities and needs identified? [Tailor based on responses to section B above.]

1. If some were not met, why not? Select all that apply

• Were you unable to find services/providers that meet your needs?

• Did you not have the funds to cover the costs of the needed training?

• Did you not have time to release staff for a particular activity?

• Other, specify

2. If resources are limited, how do you prioritize which needs you address?

IV. Coaching and technical assistance

Now I’d like to ask some questions about coaching and technical assistance your center has received over the past 12 months.

1. [Refer to question M1 in SAQ to tailor these questions and fill in missing information for each coaching, mentoring, or consultation activity listed. If no coaching, mentoring, or consultation activities were listed, confirm that there were none.] We have some follow-up questions about the coaching, mentoring, or ongoing consultation with a specialist you or your staff members who work directly with children ages 0-5 participated in to improve skills or gain new skills in working with children or to make other quality improvements during the past 12 months?

1. What information was used to identify the need for this type of assistance or to direct the content of the assistance?

2. Is a specific plan in place for this coaching or assistance?

1. Does the plan specify responsibilities of all persons involved, the purpose of the assistance, and towards what goal?

2. Who is involved in developing the plan?

3. Is adherence to the plan regularly reviewed? If so, with what frequency?

3. You noted that [enter answer to SAQ question M7] provided the assistance. On what criteria was this individual or entity selected?

4. How have you used information from follow-up activities, tracking of indicators, or staff evaluations about the success of training?

1. Is information shared with coaches or supervisors to target competency development?

2. Is information used in selecting and hiring staff with the skills and competencies needed?

3. Is information used to adjust training priorities as a whole or plans for individual staff?

2. [Ask only of centers that reported hiring a coach/mentor/consultant.] In instances when a coach, mentor, or consultant was hired by the center, what criteria were considered in making the selection made and by whom?

3. Has the center requested specific coaching or technical assistance over the past 12 months and not received it? Please explain?

4. How do you keep track of the coaching and technical assistance your staff receives?

1. Do you have an electronic tracking system?

2. If so, who enters data into the coaching and technical assistance tracking system?

5. Do you have specific professional development goals for staff tied to the coaching and technical assistance they receive? If yes, please describe.

V. Quality assurance and quality improvement

Now we’d like to learn about the types of things you do to support staff and improve services.

A. Workforce development

We’ll start with questions about job descriptions, supervisory activities, and the performance appraisal process.

1. Do staff who work directly with children ages 0-5 have written job descriptions?

1. What expectations are detailed in job descriptions and for what skills or competencies?

1. For example, are expectations included about use of a curriculum, or conducting child assessment?

2. Do descriptions embed requirements from regulatory or other entities (for example, licensing, Head Start, state-funded Pre-K, corporate entity)?

2. [Refer to question N5c on SAQ to tailor question.] What are the reasons that supervisory activities with staff who work directly with children ages 0-5 do not occur with the frequency expected?

3. Does the regularity or frequency of supervision vary for different types of staff (lead teachers, assistants) or by the supervisor? Please explain.

4. [Refer to question N6 on SAQ to tailor question.] How is the performance of staff who work directly with children ages 0-5 evaluated at the center?

1. Do you use a standardized performance appraisal form for all staff? [Request a copy if yes]

2. [If no, then ask] What written information, if any, do you keep about performance appraisal?

5. In what ways, if any, are staff made aware of the process and procedures used for performance assessment?

[Refer to questions N6 and N7 on SAQ to inform questions 6-8 that follow.]

5. Who participates in the performance appraisal of staff who work directly with children ages 0-5?

6. What criteria are used to evaluate performance?

1. Are criteria tied to the responsibilities or expectations detailed in employee job descriptions?

2. [Refer to question A.1, if expectations are detailed in job descriptions, then ask] In what ways, if any, are staff held accountable for meeting the expectations included in their job description?

5. What sources of information are used to inform the performance appraisal of staff who work directly with children ages 0-5?

6. [If answer to N6d on SAQ is YES, then ask] What goals are typically outlined for staff who work directly with children ages 0-5?

7. How are staff recognized or rewarded for strong performance?

11. [If answer to question N7 in SAQ is YES, then ask question separately for each purpose specified] Who conducts the observation?

12. [If answer to question N7 in SAQ is YES, then ask question separately for each purpose specified] What happens after observation of staff who work directly with children ages 0-5 is conducted?

1. What discussion occurs or how is feedback provided to staff?

2. If a standardized tool is used, are staff provided with scores to identify areas of strength and areas needing attention?

13. Are there opportunities for staff who work directly with children ages 0-5 to receive recognition, incentives, or rewards outside of the standard performance appraisal process?

1. Under what circumstances does this occur?

2. How often does this occur?

3. How is it decided which staff receive the recognition or reward?

14. Has the center introduced a new initiative, practice, or procedural change within the past 12 months that required changes in the responsibilities for staff who work directly with children ages 0-5 or the requirements they must follow?

1. Briefly describe the type of change that occurred.

2. How were staff assessed on their ability to implement this change?

3. Were staff recognized or rewarded for their response to or implementation of this change in a distinct way?

B. Use of information to monitor performance

Now I have some questions about the use of information to monitor or evaluate the performance of the center.

1. How is high quality and effective service delivery defined within the center?

2. How do you determine if your center is meeting its goals? Do you assess center performance or progress toward organizational goals?

1. What metrics or outcomes do you track and why?

2. With what frequency is this information collected? With what frequency is it reviewed or analyzed, and by whom (which type of staff member)?

3. How does the center use this information to inform quality improvement or decision-making?

4. How is this information communicated to staff who work directly with children ages 0-5? In what format? How often?

3. [Refer to question II.A.9 and 11 whether center uses information about meeting standards in program administration. If not, then skip this question] How does your center use the information or data collected to track and monitor existing standards (such as those for Head Start, QRIS, state pre-k, or any other program) to assess center performance or progress toward organizational goals?

1. With what frequency is this information collected? With what frequency is it reviewed or analyzed?

2. How do you use this information to inform quality improvement or decision-making?

3. How is this information communicated to staff who work directly with children ages 0‑5? In what format? How often?

4. In what ways, if any, has the center used the information you collect to change or improve the quality of care provided by the center? [Provide up to 3 examples]

5. Has the center needed to collect and track information to monitor implementation of a new initiative or practice over the past 12 months? (for example, keeping fidelity or practice logs)

1. What is tracked and why?

2. With what frequency is this information collected? With what frequency is it reviewed or analyzed?

3. How do you use this information to inform quality improvement or decision-making?

4. How is this information communicated to staff who work directly with children ages 0‑5? In what format? How often?

C. Program evaluation

1. In what ways, if any, are staff involved in evaluating the center and the services it provides to children?

1. How often is staff input sought?

2. Do they provide input using a specific tool or form? [If yes, request a copy]

3. What other mechanisms are used to elicit feedback from the staff about the quality of the center’s services?

2. In what ways, if any, are parents involved in evaluating the center and the services it provides to children?

1. How often is parent input sought?

2. Do they provide input using a specific tool or form? [If yes, request a copy]

3. What other mechanisms are used to elicit feedback from parents about the quality of the center’s services?

3. What other methods not yet discussed are used to monitor or evaluate the quality of services provided in this center?

4. Are there ways you would like to monitor or evaluate the quality of services provided but are not able to?

1. What would you like to do or what information would you like to collect and use?

2. What is holding you back from doing this?

D. Facilitative administration, communication, and partnerships

Now let’s discuss issues surrounding communication and partnerships.

1. Is there a particular person within the center who is considered the “champion” for quality or who promotes a shared understanding of the center’s goals and expectations?

1. Was the individual formally appointed in the position or was it an informal role?

2. How do they help with the implementation of quality care? Please provide a few examples.

3. Can you describe staff perception of this champion/individual?

2. In what ways, if any, has the center developed a shared understanding (or buy-in) of the mission of the center or of quality education and care in general among staff who work directly with children ages 0-5?

1. What about among other staff in the center?

3. How do staff who work directly with children ages 0-5 and management communicate with each other to facilitate or support quality?

1. How does this communication occur? Select one

• There is a regular consultation or feedback process (specify frequency)

• It occurs on an ad hoc basis

2. What level of involvement, if any, do teaching staff have in identifying issues or challenges that affect quality?

3. How have teaching staff participated in developing strategies or modifying procedures or processes to address issues or challenges, if at all?

4. What methods, if any, are used at the center to communicate information such as new initiatives, accomplishments, issues, new staff, and staff departures to staff who work directly with children ages 0-5?

5. [If the respondent reported in question H4 in the SAQ that they DO NOT have staff meetings, then ask] Why don’t you hold staff meetings?

6. [If the respondent reported in question H4 in the SAQ that they have staff meetings, then ask] We have a couple questions about your staff meetings.

1. [If the respondent reported in question H4a in the SAQ that staff meetings are mandatory only for some staff, then ask] Which staff are not required to attend, and why?

2. [Examine responses to question H4a and H4b to ask appropriate questions about participation in staff meetings] Is it correct that staff who work directly with children ages 0-5 [are / are not] required to attend staff meetings?

3. [If responses to question H4b in SAQ are 1,2, or 3 indicating lower than 75% attend then ask] Why is the proportion of staff who work directly with children ages 0-5 that attend staff meetings less than 75%?

4. Who facilitates or runs the meetings?

5. What is the format of these meetings?

6. What is the typical agenda? What topics are most often addressed or discussed?

7. When do they occur? During paid planning time or evenings? Are staff paid for participating in staff meetings?

7. Does your center have an oversight board (either internal as part of a larger organization, or an external board of directors for example)?

1. What is the type and frequency of interaction with this board? Which staff are responsible for those interactions and for carrying out any preparation and follow-up activities?

2. What is the focus of the board’s oversight and interests in the center? What topics are typically reported to or discussed with this board?

3. What data and information are regularly shared with this board?

8. Does your center have a parent governing council or advisory group?

1. What is the type and frequency of interaction with this group? Which staff are responsible for those interactions and for carrying out any preparation and follow-up activities?

2. What is the focus of the group’s oversight and interests in the center? What topics are typically reported to or discussed with this group?

3. What data and information are regularly shared with this group?

9. How does your center connect with community partners to support the provision of quality care for children ages 0-5?

1. Do you have staff that serve on community service coordination committees/boards?

2. Do community partners serve roles on any committees required for your program (for example a health services committee)?

10. What gaps in service delivery or types of support do partners help fulfill? What capacities do partners bring or help build to support quality in your program?

11. How are problems that affect quality education and care for young children identified across service systems or partners in your community?

1. Through what means are potential solutions developed and problems addressed?

2. Who, or what entities are typically involved?

E. Other quality initiatives or improvements

1. What quality initiatives or improvement activities are going on in your center that we have not yet discussed?

1. What is the type of initiative or improvement?

2. Is it part of a community, state, or federal initiative?

3. How long has your center been involved with this initiative or activity?

4. How are teaching staff involved?

5. Who within the center manages the implementation and monitors progress of this initiative or activity?

6. Was there a clear plan for implementing this initiative/activity that included specific tasks and timelines to enhance accountability?

7. How were challenges to implementation addressed?

8. What additional resources has the center received to assist with this initiative/activity?

9. What additional costs has the center incurred to support this initiative/activity?

2. How sustainable are the key activities that you feel are important to the care and education services you provide?

1. Are there activities or initiatives that are expected or likely to end soon?

VI. COGNITIVE INTERVIEW questions

[Ask the set of questions below following each of the five main sections of the interview. Ask the last question at the end of the interview.]

1. Please tell me about your overall experience answering questions on the hiring process and staff turnover in the “Staff Selection” section.

1. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

2. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

3. In this section, I asked you about hiring staff for positions associated with implementing a new initiative to help improve quality. What comes to mind when you hear the phrase “initiative to help improve quality?”

2. Please tell me about your overall experience answering questions in the “Selection and Use of Tools to Support Key Functions” section.

1. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

2. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

3. Please tell me about your overall experience answering questions in the “Training” section.

1. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

2. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

4. Please tell me about your overall experience answering the questions in the “Coaching and Technical Assistance” section.

1. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

2. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

5. Please tell me about your overall experience completing the “Quality Assurance and Quality Improvement” section.

1. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

2. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

6. Please tell me about your overall experience participating in the interview.

1. Overall, were some sections easier or more difficult for you to answer?

   1. If so, which ones were easier or more difficult? Why?

Implementation interview time log

Instructions to Interviewer:

Document the start and end times and the sections covered for each time you conduct any part of the interview.

Start time	End Time	Sections covered

1. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0355 and it expires 03/31/2018.