Early Childhood Training and Technical Assistance (T/TA) Cross-System Evaluation Project

Early Childhood T/TA Cross-System Evaluation Project

Appendix B2: Semi-structured Interview Protocol for Regional-Level T/TA Providers¹

INTRODUCTION: Thank you for taking time to meet with us. As we explained in our email, we contacted you on behalf of the Administration for Children and Families (ACF), which in partnership with us at NORC is designing and implementing a utilization-focused evaluation for the specific purpose of documenting, assessing, and enhancing ACF’s Early Childhood Training and Technical Assistance (EC T/TA) System. The goal of this interview is for us to learn more from you about your T/TA goals, stakeholders, services, desired outcomes, dissemination practices, evaluation data quality and utility, T/TA effectiveness, data systems, data systems users, evaluation needs, and general impressions. We are still in a preliminary stage of gathering information to inform the design of the national cross-system evaluation that can also support your own continuous quality improvement efforts. We started by reviewing existing documentation and other available materials relevant to your T/TA activities. In the course of this interview, we may ask you to confirm or elaborate on our findings from our review of existing documents describing your T/TA activities. We expect this conversation to take no more than 90 minutes.

Please know that your participation is voluntary and choosing not to participate will have no bearing on your participation and access to the research resources developed as a result of these discussions. No public reports will result from the information we collect. It will be used for internal purposes to inform the evaluation design and technical support for stakeholders’ respective evaluation activities, and therefore, we will share what we learn with ACF and other stakeholders. Your name and other identifiable information will never be included and we will summarize responses during these internal discussions. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.

Thank you again in advance for sharing your knowledge and insights with us.

1. Goals

We want to begin the discussion today by hearing about your short- and long-term T/TA goals.

1. What are your intended short-term T/TA goals?

   1. Do your short-term goals differ by T/TA recipient group (e.g., Regional T/TA providers, CCDF Administrators, HS grantees, etc.)?

2. What are your long-term T/TA goals?

   1. Do your long-term goals differ by T/TA recipient group (e.g., Regional T/TA providers, CCDF Administrators, HS grantees, etc.)?

3. The goals of the broader EC T/TA System are to support early care and education programs and early educators in delivering quality services to children and their families, through high quality professional development, practical resources and approaches that build early childhood program capacity, support adoption and implementation of early childhood program best practices, and promote consistent practices across communities, states, tribes, and territories. With these goals in mind:

   1. How do your goals align with those of the broader EC T/TA System?

   2. How do your goals differ from those of the broader EC T/TA System?

4. How do other stakeholders within the EC T/TA System (e.g., National Centers, OCC State Capacity Building Center, etc.) support your goals? PROBE: Are there other things you’d like to see the EC T/TA System do to support your goals?

2. Stakeholders

Next we want to learn about your T/TA stakeholders; specifically, your partners in providing T/TA and the recipients of your T/TA services, and how they might inform or shape your services.

1. Who are your primary T/TA partners (e.g., National Centers, OCC State Capacity Building Center, State/local ECE entities, etc.)?

   1. How do you obtain information about your partners’ views and needs?

   2. How do your partners’ views and needs affect your T/TA goals?

   3. How do your partners’ views and needs affect the services you provide?

   4. How are your T/TA resources allocated to meet identified partner needs?

2. Who are the primary recipients of your T/TA services (include anyone you consider a direct consumer/user or secondary beneficiary, including but not limited to CCDF Administrators, HS grantees, Regional-Level T/TA providers, State/local ECE entities, etc.)?

   1. Do you differentiate between recipients of direct T/TA services and consumers/users of your products? IF YES, how? [Alternatively, IF YES, can you say more?]

   2. How do you assess your T/TA recipients’ needs?

   3. How do you assess your T/TA recipients’ readiness for change?

   4. How do your service recipients’ needs affect your T/TA goals?

   5. How do your service recipients’ views and needs affect the services you provide?

   6. Can you share with us the protocols that you use to assess their needs (e.g., forms, descriptions, instruments, illustrative examples, results summaries)?

   7. What other information, if any, do you collect from T/TA recipients?

      1. Does the information you collect vary across recipients?

      2. If so, how?

3. T/TA Services

Now we would like to learn more about the specific T/TA services you provide.

1. What T/TA services, products or tools does you currently provide?

2. How do you determine what T/TA services to provide?

3. In your view, what are the biggest obstacles or barriers to your development of T/TA services? PROBE: Any other barriers?

4. In your view, what are the biggest obstacles or barriers to your provision of T/TA services? PROBE: Any other barriers?

5. What records do you keep on services provided? PROBE: What format do you use to record these services?

6. Do you keep a record of service requests? PROBE: What format do you use to record these requests?

7. Do you coordinate T/TA requests, service planning and delivery, and data or record keeping with other T/TA providers in the EC T/TA System (e.g., National Centers, OCC State Capacity Building Center, OHS Regional T/TA providers, fellow OHS/OCC Regional-Level T/TA providers)?

   1. If so, how?

   2. How are records kept in such instances?

8. Do you collect any direct feedback from recipients about your T/TA services? IF YES,

   1. What kind of feedback (i.e., specific topics or issues)?

      1. How do you do this?

   2. Do you record this feedback? PROBE: How?

   3. How do you use this information?

   4. Are these records available? IF MENTIONED: Are these summaries, analyses, etc. available?

4. T/TA Effectiveness

Next we would like to learn about how you evaluate the effectiveness of your T/TA.

1. How do you assess the effectiveness of your T/TA?

POTENTIAL PROBES:

1. What information do you use to evaluate your effectiveness?

2. Do you collect information regularly? How frequently? From whom?

   1. Could you share any of that information with us?

3. Do you consult data collected by anyone else (e.g., T/TA recipients, other T/TA providers, National Centers)?

   1. If so, do you use it to assess the extent to which your T/TA is having the intended effects? How?

1. How effectively are you meeting your own T/TA goals?

   1. On what evidence do you base your opinion?

2. How effectively are you meeting EC T/TA System goals?

   1. On what evidence do you base your opinion?

5. Outcomes

Next, we want to learn about the specific outcomes you are interested in or the information you use to measure progress toward your goals.

1. What information do you collect to measure progress towards your T/TA goals? PROBE, IF NECESSARY: Do you measure any outcomes or outputs?

PROBES:

1. How is information collected?

2. How frequently is it collected?

3. How is it recorded?

4. How do you use this information?

5. Do you analyze the data you collect? IF YES:

   1. What kinds of analyses do you complete?

6. Do you report on these analyses? If so, how? With whom?

1. Do you have different outcomes or information for specific types of T/TA you provide (e.g., webinars, workshops, train-the-trainer, institute, structured peer learning, tailored approaches, etc.)?

   1. How do you use different metrics for specific types of T/TA you provide? PROBE, IF NECESSARY:

      1. Do you analyze these?

      2. Can you give me an example of a recent analysis you completed?

6. Dissemination

Now we want to explore how you disseminate information.

1. What are your dissemination goals?

2. What specific dissemination methods do you use (e.g., ECLKC, CCTAN, other website(s), Webinars, conference calls, in-person briefings/seminars, publications/newsletters)? PROBES:

   1. Do methods differ by dissemination goal/outcome desired? IF YES: How do they differ?

   2. Do methods differ by audience? PROBES:

      1. How do you disseminate information to T/TA partners (e.g., National Centers, OCC State Capacity Building Center, State/local ECE entities, etc.)?

      2. How do you disseminate information to T/TA recipients (e.g., CCDF Administrators, HS grantees, Regional-Level T/TA providers, State/local ECE entities, etc.)?

   3. What methods do you use most? Why?

   4. What methods do you use least? Why?

3. What information do you use to guide your dissemination planning or quality improvement efforts?

4. Do you have summary information on the numbers and types of products/tools distributed? Or on who accesses them? IF YES:

   1. What kinds of information?

   2. How do you use the information?

5. Do you collect any direct feedback from recipients about your dissemination methods?

   1. If yes, how do you do this? PROBES:

      1. How often?

      2. From whom, exactly?

   2. What do you do with that information? PROBE:

      1. Do you have procedures for recording this feedback?

      2. Do you analyze the feedback in any way?

      3. Do you summarize or report the feedback in any way?

      4. Could you share any of that feedback with us?

      5. How do you use this information?

6. How do you assess the effectiveness of your dissemination methods? PROBES:

   1. What information do you collect?

   2. Do you collect information regularly? How frequently? From whom?

      1. Could you share any of that information with us?

   3. Do you consult data collected by T/TA recipients? IF YES:

      1. How do you use it?

      2. Do you use it to assess the effects of your T/TA?

7. Overall, how effective are your dissemination methods? PROBE: What are your most successful? What are your least successful?

   1. Are some approaches more successful with certain target groups?

   2. Do you have any specific metrics relating to your dissemination methods?

7. Data Quality & Utility

Now we would like to hear your thoughts about the quality and utility of the evaluation data you collect and use.

1. Of all the evaluative information you collect or consult, what do you find most informative? How/why? PROBES, AS NECESSARY:

   1. What do you measure best?

      1. What do you find most informative? Why?

      2. What do you think is most reliable? Why?

   2. What do you measure least well?

      1. What do you find least informative? Why?

      2. What do you find least reliable? Why?

   3. Is there any additional information/data that could enhance your T/TA products and services? [ALTERNATIVELY: What are you not assessing/measuring that you would like to?] PROBES:

      1. What information do you need to enhance your T/TA provision?

      2. What additional information would you want for management purposes?

      3. What additional information would you want for evaluation purposes?

      4. Does your current data meet all of these needs?

8. Data Systems & Information Technology

This next set of questions asks you to describe the data systems and information technology you use to make available and document the use of your T/TA services, tools and resources.

1. (Questions for an OHS Regional T/TA Provider): We understand that you are required to use the Head Start Enterprise System to track T/TA provision to HS grantees. In this next set of questions, we would like to learn more about how you use the Enterprise System.

   1. How regularly do you complete the T/TA provider form and submit it on the Enterprise System? PROBE FOR SPECIFIC FREQUENCY: Weekly? After each T/TA session?

      1. Are you aware of any differences in T/TA provider form completion among Specialists (i.e., Early Childhood, Health, Grantee, System)?

   2. Do you record any information in addition to the general T/TA provider form? Please describe.

      1. Are you aware of any differences as a function of type of Specialist (i.e., Early Childhood, Health, Grantee, System)?

   3. What procedures are in place to check or monitor completeness or check validity of data entered into the Enterprise System?

   4. Do you use the information recorded on the Enterprise System for tracking or other purposes? IF YES:

      1. Please describe how you use this information.

   5. What are the most meaningful or informative data points in the Enterprise System?

      1. How/why are these informative to you?

   6. What are the least meaningful or informative data points?

      1. How/why are these not informative?

   7. What are the most frequently reviewed/analyzed data in the Enterprise System? Why?

   8. What are the least reviewed/used? Why?

   9. What would you want to have more information about (i.e., that you’re not currently collecting or that you might want to collect differently)?

   10. How could the Enterprise System better support your work? [PROBE: How would you like to be able to use it?]

2. (Questions for the OCC State Capacity Building Center State System Specialists and Director): We understand that you are required to use the TA Tracker to track the T/TA you provide to States. In this next set of questions, we would like to learn more about how you use the TA Tracker.

   1. How regularly do you record specific information in TA Tracker? PROBE AS NEEDED FOR SPECIFIC FREQUENCY: After each T/TA session? Weekly? Please describe.

      1. Are you aware of any differences in the information recorded depending on the type of Specialist (i.e., State System Specialists, Infant/Toddler)?

   2. What procedures are in place to check or monitor completeness or check validity of data entered into the TA Tracker?

   3. Do you use the information recorded in the TA Tracker for tracking or other purposes? IF YES:

      1. Please describe how you use this information.

   4. What are the most meaningful or informative data points in the TA Tracker?

      1. How/why are these informative to you?

   5. What are the least meaningful or informative data points?

      1. How/why are these not informative?

   6. What are the most frequently reviewed/analyzed data in the TA Tracker? Why?

   7. What are the least reviewed/used? Why?

   8. What would you want to have more information about (i.e., that you’re not currently collecting or that you might want to collect differently)?

(Below are data systems questions for both OHS and OCC Regional-level T/TA Providers)

3. Do you use any other data systems to document your T/TA services, tools, and resources? IF YES:

   1. Please describe the system and what other purpose/s, if any, the system serves?

   2. What do you find useful about the system?

   3. How could the system better support your work? [PROBE: How would you like to be able to use it?]

4. Do you use any other data systems to make available T/TA services, tools, and resources? IF YES:

   1. Please describe the system and what other purpose/s, if any, the system serves?

   2. What do you find useful about the system?

   3. How could the system better support your work? [PROBE: How would you like to be able to use it?]

5. Do you use any web analytics or information from websites that record information related to your T/TA services?

   1. From what websites do you obtain web analytics?

   2. Do you use ECLKC or CCTAN web analytics?

   3. How do you use the information?

6. Do you have one integrated data system or separate systems for different purposes [PROBE: documentation of needs, characteristics of T/TA recipients, services requested/provided, tools and resources accessed by T/TA recipients, data on recipient outcomes, etc.]? [ALTERNATIVELY: Are you able to link or look across the different types of information/data you use?]

7. Are your files computerized, and if so, how? (e.g., Excel spreadsheets, case records for each recipient, database of services provided, etc.)

8. Do you have a separate computer system for accounting (e.g., income and expenses)? IF YES:

1. Is it integrated with T/TA data (e.g., by linking costs to specific activities)?

9. Data System Users

Now we would like to learn more about the people who use these systems.

1. Who accesses your data system(s)?

2. Can the National Centers access your data system(s)?

   1. If so, which system(s) do they use and for what purpose?

   2. If not, is this something you would consider doing in the future?

3. Do you collect feedback about the data system(s) from data system users?

   1. If so, how is this feedback used?

10. Your Evaluation Needs

An important goal of the national cross-system evaluation is to support stakeholders’ continuous quality improvement efforts. We want to know what research, data resources or tools could support your efforts.

1. What research, data resources or tools could support your evaluation activities?

2. What research, data resources or tools could support your efforts to understand the reach of your products?

3. What research, data resources or tools could support your efforts to understand the effects of your services?

11. General Impressions

Our last few questions are about your general impressions of the T/TA system as well as your ideas for designing a maximally useful national evaluation.

1. What is important to retain or build upon amidst the transformation of the T/TA System?

2. What do you most want to know about the T/TA system and its functioning?

3. What research question would you prioritize for this evaluation?

4. What haven’t we asked about that surprises you?

5. What else should we ask or know about before trying to design an evaluation?

CLOSING: Thank you for participating in this discussion today. We appreciate your willingness generously share with us your expertise and time. If you have any additional information, comment or questions, please do not hesitate to email or call us at [INSERT INTERVIEWER INFO HERE].

1 Questions are set up to be tailored to either an OHS regional T/TA provider (i.e., Coordinator) or an OCC State Capacity Building Center T/TA provider (i.e., State Systems Specialist or Center Director).