Interview Guide for Grantee Evaluators

Attachment 7B

Grantee Evaluator Interview guide

Form Approved

OMB No.: 0920-XXXX

Expiration Date: XX/XX/XXXX

Interview Guide

Program Evaluator for Grantee Program

Public reporting burden of this collection of information is estimated to average one hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0920-XXXX)

Hello. My name is [full name], and I’m with ICF Macro. We are working with the Centers for Disease Control and Prevention, or CDC, to better understand how grantees funded through the Colorectal Cancer Control Program, or CRCCP, implement their programs and how implementation changes over time. As a person who is involved with the evaluation of this program at the grantee level, we would like to discuss with you evaluation efforts that have occurred since the program’s inception. Additionally, because of your close proximity to the program, we feel you can offer valuable insight into how [name of State]’s CRCCP has been implemented. This interview should take no more than 1 hour of your time. Let me share a copy of our Informed Consent Form, and you can follow along as I read it aloud.

[Provide the participant with a copy of the Informed Consent Form. Read the statement aloud.]

We want to make sure you know y that our answers to any questions that I ask you during this interview will be treated in a secure manner. In evaluation reports, manuscripts, and other materials developed to dissemination information about the evaluation, descriptive data for each program will be presented so that various program models can be shared. Specific quotes will not be attributed to any particular individual.

I want to ask for your permission to audiotape our interview. This will allow me to capture all the details of our discussion. The tape will be kept secure and used only by the evaluation team to supplement the notes we take. Your name will be removed from the notes and transcript.

Do you have any questions?

Let’s begin.

Background

1. Can you tell me about your roles and responsibilities with CRCCP?

2. When and how did you become involved with CRCCP?

3. Do you work full- or part-time with CRCCP?

4. What are your specific duties as related to the evaluation of this program?

5. How has your involvement in the program changed over time?

6. If the evaluator is employed by another agency: Please briefly describe your agency and how long it has been involved with [name of State]’s health department, as well as with work related to colorectal cancer control and prevention.

Implementation

Now we want to ask you a few questions about your involvement in the implementation of [name of State]’s CRCCP. We are particularly interested in understanding what resources, including funding, staffing, partnerships, evaluation, and technical assistance, are needed to evaluate the implementation of [name of State]’s CRCCP.

Conducting Evaluation

1. Did you participate in the development or identification of any tools, such as a logic model, an action plan, or an implementation plan, which were used to guide the implementation of CRCCP activities? If so, please describe your participation and the tools.

2. Does [name of State]’s CRCCP have an evaluation plan that specifically addresses implementation? If so, how was it developed?

3. Please briefly describe the evaluation plan.

• What role did you play in its development?

• To what extent does the evaluation plan include process measures? To what extent does it include outcome measures?

• Does the evaluation plan include an explicit evaluation of the effectiveness of the program? If so, what indicators and data sources have been identified?

4. How is the success of [name of State]’s CRCCP screening promotion efforts measured? What outcomes do you measure?

• Do you have baseline data? If so, please describe these data.

• How do you intend to collect follow-up data? At what time (Examples: pre/post or posttest only design)?

• Do you collect data at specified points over time (Example: time series)? What length of time?

• Do you follow a sample of participants over time (Example: cohort study)? What length of time?

• Do you have a comparison group?

• What does the evaluation activity tell you about the success of the promotion efforts?

5. Please describe the process that [name of State]’s CRCCP uses to make evaluation-related decisions during the implementation of this program.

• Who is involved? What are their roles?

• Are there regularly scheduled evaluation meetings? If so, who participates?

• What are the strengths of this process?

• What are the weaknesses?

• Has this process changed over time? If so, please describe the changes.

6. Has [name of State]’s CRCCP developed any measures of effectiveness to assist in evaluating its efforts? Measures of effectiveness are objective or quantitative measures developed by [name of State]’s CRCCP to demonstrate the accomplishment of objectives as identified in its program plan.

• If so, what process was used to develop these measures?

• Who was involved?

• How did [name of State]’s CRCCP decide which measures to choose?

• Have you encountered any difficulties in using these measures? If so, please describe the difficulties.

• What are the benefits, if any, of using these measures?

7. What additional data, if any, does [name of State]’s CRCCP use to aid in the evaluation of its program?

• Do you have data sharing agreements with any partner organizations? If so, please describe these agreements.

8. To date, what evaluation activities have been conducted?

• What activities have been most useful? Least useful?

• Which activities have been the most difficult to conduct?

• In what ways have you used the data or findings?

• Have the evaluation findings been used to inform decisions about program planning or implementation? Please provide examples.

9. Have you conducted any evaluation activities related to building partnerships (e.g., assessing partner satisfaction or impact on partner organizations)? If yes, please briefly describe these activities.

10. How were these evaluation findings used?

• Did you encounter challenges in evaluating these activities? If so, how did you overcome those challenges?

• Can you give examples of how the evaluation findings have been used? If so, please describe the examples.

11. Have you conducted any evaluation activities related to building State-level legislative or leadership support for colorectal cancer screening? If yes, please briefly describe these activities.

• What are the available data sources?

• What data collection tools or instruments are you using?¹

• How were these evaluation findings used?

• Did you encounter challenges in evaluating these activities? If so, how did you overcome those challenges?

• Can you give examples of how the evaluation findings have been used? If so, please describe the examples.

12. Have you conducted any evaluation activities related to screening promotion activities conducted by either your CRCCP or your partner organizations? If yes, please briefly describe these activities.

• What are the available data sources?

• What data collection tools or instruments are you using?²

• How were these evaluation findings used?

• Did you encounter challenges in evaluating these activities? If so, how did you overcome those challenges?

• Can you give examples of how the evaluation findings have been used? If so, please describe the examples.

13. Are you aware of any competing issues or priorities that have affected any of your evaluation efforts? If so, please describe them.

14. What strategies have been used to minimize the impact of those competing issues?

15. Were there groups, individuals, or other factors that facilitated or hindered evaluation activities? If so, please explain the factors and how they facilitated or hindered the activities.

Evaluation Infrastructure

1. Besides you, are there other evaluators currently involved in the evaluation?

• If so, what organizations are they affiliated with?

• What are their responsibilities?

• Were they hired specifically to evaluate [name of State]’s CRCCP? Were there any difficulties in hiring them?

• Do you know how their positions are funded? If so, please describe the funding.

• How do all of the evaluation activities fit together?

• How do you envision the other evaluators’ responsibilities changing over time?

2. What roles do [name of State]’s CRCCP staff members play in evaluating program activities?

3. Are they involved to the extent they should be? What role, if any, do partners play in evaluation efforts?

4. Have recruitment and retention of evaluation staff been a problem? If so, please describe the problems experienced.

5. What strategies, if any, are in place to minimize turnover or its impact on the program?

6. Are evaluation activities adequately staffed and funded?

7. Do you anticipate needing additional evaluation staff in the future? If so, please describe what role they would play.

8. Other than CDC funding, are you aware of any additional resources used to support the evaluation of [name of State]’s CRCCP activities? If so, please describe these resources and the activities they support.

Technical Assistance and Training

1. What technical assistance or training, if any, have you received to assist with your efforts for this program?

• What were the topics of the technical assistance or training?

• Who provided it? Who else participated in the technical assistance or training?

• Was it useful?

• What additional technical assistance or training, if any, would be helpful to you in carrying out your responsibilities related to evaluating [name of State]’s CRCCP?

2. Have you or any other evaluators provided any technical assistance or training to partners or others affiliated with [name of State]’s CRCCP? Please describe what was provided.

• What was the topic?

• Who was involved in the development and delivery of the technical assistance or training?

• To whom was it provided?

• Were there any barriers or facilitators to providing the technical assistance or training?

• How was it funded?

• Was feedback obtained from participants? If so, what were the results?

Conclusion

Now we would like to hear your thoughts on how well you think [name of State]’s CRCCP is working.

1. In your opinion, what are the key strengths of [name of State]’s CRCCP?

2. In your opinion, what are the key limitations of [name of State]’s CRCCP?

3. What has been accomplished by [name of State]’s CRCCP that otherwise would not have been?

4. Overall, do you think [name of State]’s CRCCP utilizes an effective approach to improve colorectal cancer screening? If so, why? If not, why? Is there another approach you would recommend using? Why?

5. What concerns, if any, do you have about future efforts to evaluate [name of State]’s CRCCP?

6. Is there anything else you would like to say about [name of State]’s CRCCP that we have not discussed?

Those are all the questions I have for you at this time. Before we wrap up, do you have any questions for me about anything we’ve talked about?

Thank-you again for taking the time to speak with me. We sincerely appreciate and value your input!

1 Request to see or have copies made of data, data collection tools, or evaluation reports.

2 Request to see or have copies made of data, data collection tools, or evaluation reports.