Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase 1

Implementation interview protocol

Quality assurance & quality improvement MATHEMATICA POLICY RESEARCH

module 5: Quality assurance and quality improvement

Now we’d like to learn about the types of things you do to support staff and improve services provided by your center.

We’ll start with questions about job descriptions, supervisory activities, and the performance appraisal process.

A. Workforce development

1. Do classroom staff have written job descriptions?

□ Yes □ No Go to 2

1. What expectations are detailed in job descriptions and for what skills or competencies? (Probe on use of curriculum or conducting child assessment)

2. What requirements from regulatory or other entities are included in job descriptions? (Probe on licensing, Head Start, state-funded Pre-K, corporate entity)

3. In what ways, if any, are staff held accountable for meeting the expectations included in their job description?

2. Who supervises lead teachers?

2. Who supervises aides or assistants?

2. What is the typical number of staff that a supervisor is responsible for?

3. Does supervision occur through:

□ One-on-one meetings

□ Group meetings

□ Both

2. Are supervisory activities conducted on:

□ A regular basis,

□ An as needed basis, or

□ Other (specify)

2. What is the frequency of supervisory activities?

___ times per day

___ time per month

___ times per year

2. Does the regularity or frequency of supervision vary for different types of staff (lead teachers, assistants) or by the supervisor? Please explain.

2. Do supervisory activities occur with the frequency expected?

□ All of the time

□ Most of the time

□ About half the time

□ Less than half the time

□ Rarely

[If answer anything but 1st option, ask] Why don’t supervisory activities occur with the frequency expected? Please explain.

The next set of questions ask about how the performance of classroom staff is evaluated.

2. How is the performance of classroom staff evaluated?

□ Through a formal process on a regular schedule

□ Through informal feedback as needed

□ Combination of formal and informal

□ Other (specify)

2. What criteria are used to evaluate performance?

1. Are criteria tied to the responsibilities or expectations detailed in employee job descriptions?

□ Yes □ No

2. Are criteria tied to the quality of teaching? If so, how is the quality of teaching assessed?

3. Are criteria tied to how closely the curriculum is followed? If so, how is this assessed?

12. Does the performance appraisal process include annual goal setting?

□ Yes □ No Go to 13

a. What goals are typically outlined for classroom staff?

b. Are those goals linked to training or professional development?

□ Yes □ No

13. Is the frequency of performance appraisal the same for all classroom staff?

□ Yes □ No Go to 16

a. How often does performance appraisal occur?

____times per day

____time per month

____times per year

b. How often does performance appraisal occur for lead teachers?

____times per day

____time per month

____times per year

c. How often does performance appraisal occur for assistants or aides?

____times per day

____time per month

____times per year

14. Who participates in the performance appraisal of classroom staff?

15. In what ways, if any, are staff made aware of the process and procedures used for performance assessment?

16. Are there any other sources of information that are used to inform the performance appraisal of classroom staff?

15. How are staff recognized or rewarded for strong performance?

15. Are there opportunities for classroom staff to receive recognition, incentives, or rewards outside of the standard performance appraisal process?

□ Yes □ No Go to 19

a. Under what circumstances does this occur?

b. How often does this occur?

c. How is it decided which staff receive the recognition or reward?

15. Has the center introduced a new initiative, practice, or procedural change within the past 12 months that required changes in classroom staff’s responsibilities or the requirements they must follow?

□ Yes □ No Go to B1

a. Please describe the type of change that occurred.

b. How were staff assessed on their ability to implement this change?

c. Were staff recognized or rewarded for their response to or implementation of this change in a distinct way?

B. Center planning

Now let’s talk about processes and tools used for program planning.

[Refer to discussion during pre-visit call about center planning]

1. Who was involved in developing the strategic plan or other planning tool?

2. When was the current version created or updated?

3. To which staff are the activities or goals of the strategic plan communicated?

• Administrators

• Teachers

• Assistant teachers

• Aides

• Specialists

4. How are the activities and goals of the strategic plan communicated to staff?

5. How often is the plan or other tool reviewed for progress?

6. How often is the plan or other tool updated?

7. Does this center have any kind of quality improvement plan?

□ Yes □ No Go to next section

a. What are the main areas or activities targeted for quality improvement?

b. Was the quality improvement plan developed as part of participation in a particular program or initiative (for example, as part of participation in QRIS)? If yes, specify the program or initiative.

c. Who was involved in developing the plan? Specifically ask: Were classroom staff involved and, if so, how?

d. How often is the plan reviewed and updated?

e. How is progress tracked and by whom?

f. In what ways, if any, is the quality improvement plan shared with staff?

C. Use of information to monitor center performance and to adapt policies and procedures

Now I have some questions about the use of information to monitor or evaluate center performance.

1. Do you assess center performance or progress toward organizational goals?

□ Yes □ No Go to 2

a. How do you determine if your center is meeting its goals?

b. What metrics or outcomes do you track and why?

c. With what frequency is this information collected?

d. With what frequency is it reviewed or analyzed, and by whom (which type of staff member)?

e. How does the center use this information to inform quality improvement or decision-making? (Probe on feedback mechanisms for quality improvement)

f. How is this information communicated to classroom staff? In what format? How often?

2. What level of involvement, if any, do classroom staff have in identifying issues or challenges that affect quality?

2. How have classroom staff participated in changing procedures or processes to address issues or challenges, if at all?

3. [Ask as applicable] Does your center use the information or data collected to track and monitor existing standards (such as those for Head Start, QRIS, state pre-k, or any other program) to assess center performance or progress toward organizational goals?

□ Yes □ No Go to 5

a. With what frequency is this information collected? With what frequency is it reviewed or analyzed?

b. How do you use this information to inform quality improvement or decision-making? (Probe on data-driven decision making)

c. How is this information communicated to classroom staff? In what format? How often?

d. What are the benefits of collecting and tracking this information?

5. In what ways, if any, has the center used this information to change or improve the quality of care provided by the center? (Probe for concrete examples)

5. Has the center needed to collect and track information to monitor implementation of a new initiative or practice over the past 12 months? (for example, keeping fidelity or practice logs)

a. What is tracked and why?

b. With what frequency is this information collected?

e. How is this information communicated to classroom staff? In what format? How often?

7. Please describe the process for determining whether changes in policies or procedures are needed to support new initiatives?

a. Who is involved? Are staff who work directly with children ages 0-5 involved?

b. How is the need for change assessed?

7. How flexible is the center in changing or developing policies and procedures to support new initiatives? (probe for concrete examples).

a. What characteristics of your center facilitate the ability to change or adapt policies and procedures?

b. What characteristics make it difficult for the center to change or adapt policies or procedures?

C. Program evaluation

1. In what ways, if any, are staff involved in evaluating the center and the services it provides to children?

a. Do staff provide input using a specific tool or form?

□ Yes [If yes, request a copy] □ No

b. In what other ways is staff feedback sought about the quality of the center’s services?

2. In what ways, if any, are parents involved in evaluating the center and the services it provides to children?

a. Do they provide input using a specific tool or form?

□ Yes [If yes, request a copy] □ No

b. In what other ways is parent feedback sought about the quality of the center’s services?

3. Are there any other ways that the Center monitors or evaluates the quality of services provided in this center?

3. Are there ways you would like to monitor or evaluate the quality of services provided but are not able to?

□ Yes □ No Go to section D

a. What would you like to do or what information would you like to collect and use?

b. What is holding you back from doing this?

D. Facilitative administration, communication, and partnerships

1. Is there a particular person within the center who is considered the “champion” for quality or who promotes a shared understanding of the center’s goals and expectations?

□ Yes □ No Go to 2

a. Was the individual formally appointed in the position or was it an informal role?

b. How do they help with the implementation of quality care? (Probe for concrete examples).

c. Can you describe staff perception of this champion/individual?

2. Does this center or organization have a written purpose or mission statement?

□ Yes □ No GO TO 3

a. How is the statement shared with staff?

Mark all that apply

□ Posted somewhere visible

□ Written in staff or center handbook

□ Discussed during staff meetings

□ Other (specify)

b. Which staff in this center have received a copy of the mission statement?

MARK ALL THAT APPLY

□ Administrators

□ Lead Teachers

□ Assistant teachers

□ Aides

□ Specialists

2c. How often is the mission statement updated?

Mark only one

□ Annually

□ Every other year

□ As needed

□ Other (specify)

2. In what ways, if any, has the center developed buy-in to the mission of the center or a shared understanding of quality education and care among classroom staff? (Probe for concrete examples)

a. What about among other staff in the center?

2. What concepts are emphasized in written or verbal messages about the purpose of the center?

MARK ALL THAT APPLY

• Quality of care

• Child development

• School readiness

• Parent engagement

• Community engagement

• Meeting needs of families/community

• Other, specify

4. How do classroom staff and management communicate with each other to facilitate or support quality?

a. How does this communication occur?

• Formally: There is a regular consultation or feedback process (specify frequency)

• Informally: It occurs on an ad hoc basis

6. What methods, if any, are used within the center to communicate information such as new initiatives, accomplishments, issues, new staff, and staff departures to classroom?

6. Does your center have staff meetings that include classroom staff?

□ Yes □No GO TO 8

a. Are staff meetings mandatory?

MARK ONE ONLY

□ Mandatory for all staff

□ Mandatory for some staff (specify for which staff they are mandatory)

□ Not mandatory

[If not mandatory] Which staff are not required to attend, and why?

b. Are classroom staff required to attend staff meetings?

□ Yes □ No

c. What proportion of classroom staff typically attend?

MARK ONE ONLY

1 □ Less than 25%

2 □ Between 25 to 50%

3 □ Between 50 to 75%

4 □ More than 75% but not quite all

5 □ All staff

[If response is 1,2, or 3 indicating lower than 75% attend then ask] Why is the proportion of classroom staff that attend staff meetings less than 75%?

d. What is the frequency of these meetings?

MARK ONE ONLY

1 □ Weekly

2 □ Bi-weekly

3 □ Monthly

4 □ Quarterly

5 □ Other (specify)

e. Who facilitates or runs the meetings?

f. What is the format of these meetings?

g. What is the typical agenda? What topics are most often addressed or discussed?

h. When do they occur? During paid planning time or evenings? Are staff paid for participating in staff meetings?

6. [If they DO NOT have staff meetings, then ask] Why don’t you hold staff meetings?

6. Does your center have an oversight board (either internal as part of a larger organization, or an external board of directors for example)?

□ Yes □ No GO TO 10

a. What is the type and frequency of interaction with this board?

b. Which staff are responsible for those interactions and for carrying out any preparation and follow-up activities?

c. What is the focus of the board’s oversight and interests in the center?

d. What topics are typically reported to or discussed with this board?

e. What data and information are regularly shared with this board?

10. Does your center have a parent governing council or advisory group?

□ Yes □ No GO TO 11

a. What is the type and frequency of interaction with this group?

b. Which staff are responsible for those interactions and for carrying out any preparation and follow-up activities?

c. What is the focus of the group’s oversight and interests in the center?

d. What topics are typically reported to or discussed with this group?

e. What data and information are regularly shared with this group?

11. How does your center connect with community partners to support the provision of quality care (for children ages 0-5)?

a. Do staff serve on community service coordination committees/boards?

□ Yes □ No

b. Do community partners serve roles on any committees required for your program (for example a health services committee)?

□ Yes □ No

12. What gaps in service delivery or types of support do partners help fulfill?

a. How do partners help support quality in your program?

12. How are problems that affect quality education and care for children identified across service systems or partners in your community?

a. Through what means are potential solutions developed and problems addressed?

b. Who, or what entities are typically involved?

E. Wrap-up

1. How sustainable are the key activities that you feel are important to the care and education services you provide?

a. Are there activities or initiatives that are expected or likely to end soon? Why?

2. What do you view as the key aspects of your center that contribute to your support of quality services?

3. What do you view as the key challenges or areas in need of improvement in supporting the quality of services your center provides?

Cognitive Interviewing Questions for Module

1. Please tell me about your overall experience answering questions in the “Quality Assurance and Quality Improvement” section.

2. [Ask only if perceived difficulty in responding]: I noticed that you paused when responding to one question in particular. The question reads as follows [repeat question]. Was this question difficult to understand? If so, why?

3. Were any other questions in this section not clear to you or difficult to respond to for some other reason?

1. If so, can you think of other ways that the question might be worded so that it is easier to understand or easier to respond to?

2. What additional information would you need to answer the question?

THANK YOU FOR YOUR PARTICIPATION!

15