Att 2B_Response to Public Comments

November 19, 2012

Cancer Council and Science, Epidemiology, and Evaluation Committee

National Association of Chronic Disease Directors

2872 Woodcock Boulevard, Suite 220

Atlanta, GA 30341

Dear Members of the Cancer Council and Science, Epidemiology, and Evaluation Committee:

Thank you for taking time to review and comment on the plan to evaluate grantee attainment of selected activities of National Comprehensive Cancer Control Program (NCCCP) Priorities. I appreciate your commitment to improving the quality and usefulness of NCCCP evaluation efforts. All of your comments were carefully considered. Included with this letter you will find responses to the suggestions and remarks you submitted on October 9, 2012.

As the NCCCP evolves, it is important to monitor achievements, identify opportunities for improvement, and assess and address technical assistance needs. The planned evaluation will help us complete these key tasks.

Again, thank you for your ongoing support to the National Comprehensive Cancer Control Program.

Sincerely,

Angela Moore, MPH

Program Evaluation and Partnership Team Lead

Division of Cancer Prevention and Control

Comprehensive Cancer Control Branch

Centers for Disease Control and Prevention

NATIONAL COMPREHENSIVE CANCER CONTROL PROGRAM EVALUATION

Responses to Comments and Suggestions

NOTE: (C) indicates a comment; (R) indicates a response from CDC. Responses are in blue.

Web-based Survey:

• (C) In order to maximize the web-based survey response rate, CDC should

• make the survey mandatory.

(R) Participation in the survey is voluntary, and grantees will not be penalized should they choose not to participate. However, CCCB staff will encourage participation by announcing the survey on grantee calls and websites, and by providing an overview of the survey on a grantee call. The importance of grantee feedback will be emphasized on the grantee call, and is also noted in the invitation to participate in the survey and survey introduction and consent sections.

• offer the survey when grantees are on a call together; offer the survey when grantees are at a meeting together.

(R) The survey is an online instrument. It was designed to be completed independently by each grantee. Therefore, the survey will not be offered in person; nor will grantees have the option of completing it together. However, as noted in the response above an overview of the survey will be provided on a grantee call.

• ensure confidentiality of responses.

(R) Grantees will be providing information specific to their programs, so we cannot guarantee confidentiality. However, we will take steps to limit the possibility that data will be identifiable. For example, survey data will be stored on the contractor’s secure server, and a limited number of project team members will have access to the data.

• follow up more than once with non-responses.

(R) Multiple follow-up contacts with non-responders are planned, including reminder e-mails and one-on-one follow-up calls by Program Consultants.

• reduce the content and time needed.

(R)Skip patterns are embedded in the survey to help minimize the amount of time required to complete it. Additionally, word limits have been placed on open-ended response fields to encourage brief answers and minimize burden.

• (C) With regard to reducing the burden of the web-based survey,

• shorter is usually better.

(R) The evaluation team has worked to keep the instrument as short as possible while still gathering information pertinent to the evaluation.

• don’t force answers to proceed; make sure the questions are understood (test the question); make sure if there is a forced choice that the choices are inclusive of the possible answers.

(R) Responses are only required for survey items that determine skip patterns. For these questions, “Don’t know” and “Choose not to answer” are included as response options.

• can any of the information be collected through MIS? Or is any of the information available through MIS?

(R) The survey was designed to address gaps in information collected through MIS. Some MIS data (e.g., partnership composition) will be abstracted and analyzed as part of the NCCCP evaluation.

• make all the questions easy to answer by having check boxes.

(R) Most of the survey items include a menu of response options. Some open-ended items are necessary to collect details about grantees’ efforts to implement the NCCCP Priorities.

• serving on a coalition does not always translate into coordination. Attending meetings reduces time available to accomplish objectives.

(R)CDC is particularly interested in learning about CCC staff participation on other state partnerships. Survey items related to implementation of the NCCCP Priorities will help to capture information about coordination of chronic disease prevention and control efforts.

• the way these questions are structured invites duplication of effort, which is not the probable intent. For example, tobacco-free living, nutrition and physical activity, and patient-centered medical home initiatives are areas where the CCC program may not be implementing an intervention, but partnering with other entities.

(R)The following text has been added to the survey: “Note: As illustrated in the example responses for questions about your program’s efforts to implement the priorities, it is understood that interventions are often implemented in collaboration with other chronic disease programs, state agencies, and coalition partners. Please report on CCC-led and collaborative interventions.” For further clarification, example responses have been added for all of the open-ended survey items related to implementation of NCCCP Priorities.

• #27 Patient Navigation is targeted to women with abnormal screening results––not increasing use of cancer screening. Evidence-based interventions may not be limited to patient navigation or community health worker intervention titles.

(R) The patient navigation and community health worker item has been revised as follows:

Question: Is your program implementing a patient navigation or community health worker initiative?

Response Options:

• Yes

• No

Question: What are the aims of the patient navigation or community health worker initiative?

Response Options (select all that apply):

• Increase the use of cancer screening

• Support appropriate treatment

• Address financial barriers to screening and/or care

• Provide emotional support to cancer patients and/or caregivers

• Provide other supports (e.g., transportation, child care, elder care, language translation) to cancer patients

• Improve access to cancer clinical trials

• Increase participation in cancer clinical trials

• Other (please describe)

• the information will be more complete if it asks what methods are being used (multiple choice), who is involved, key partners, expected outcomes.

(R)The survey has been revised to include example responses for items related to implementation of the NCCCP Priorities; we expect that these model responses will encourage grantees to provide complete, useful descriptions of their interventions.

Focus Groups:

• (C) The best way for CDC to organize the focus groups, to ensure honesty and openness among participants is

• groups based on experience with the CCC program (71.4% of respondents),

• groups based on similar project homes (50% of respondents),

• groups based on similar job titles (35.7% of respondents),

• groups based on geography (28.6% of respondents),

• groups from mixed project homes (7.1% of respondents),

• groups of mixed job titles (7.1% of respondents), or

• allow group members to self-select (7.1% of respondents).

(R) We will use these recommendations to inform the composition of focus groups.

• (C) CDC can ensure diverse representation of CCC grantees on the focus groups by

• planning the focus group for time when grantees are meeting together (92.9% of respondents),

• making participation in a focus group mandatory (50% of respondents), and

• offering virtual focus group opportunities (35.7% of respondents).

(R)We agree that the best option would be to hold focus groups in person during a time when grantees are attending a prescheduled meeting. However, due to the decreasing number of opportunities for in-person grantee meetings it is possible that this opportunity will not be available. If we are not able to hold focus groups during a prescheduled grantee meeting, we will conduct phone or web-based focus groups. Participation in focus groups is voluntary, and grantees will not be penalized should they choose not to participate.

• (C) To reduce the burden of focus groups, CDC could

• share the focus group topics with participants ahead of time (76.9% of respondents agreed).

(R) We plan to share focus group topics with participants ahead of time.

• conduct them when participants are already gathered for a different meeting so no or little travel is required.

(R)Participants will not be required to travel for these focus groups. We will either hold them at a prescheduled meeting or conduct phone or web-based focus groups.

• hire a contractor to conduct the focus groups.

(R)Focus groups will be conducted by a contractor without CDC staff present.

• if possible, collect information through MIS.

(R) Evaluation planning included thoughtful development and thorough review of data collection instruments to ensure that they complement, rather than duplicate, data collected through MIS.

• target questions that are focused on areas people are likely to understand and be confident to answer without any CDC staff listening.

(R) We plan to share focus group topics with participants ahead of time, and focus groups will be conducted by a contractor without CDC staff present.

• have several different types of focus groups and let the participants decide what kind of focus group they would like to be in, depending on how much time CDC has for results. There could be virtual, regional, or tribal, state and territorial focus groups. If the participant has participated in a survey monkey it would eliminate them from a focus group.

(R) We will consider experience with the CCC program, project home, job title, and geography when organizing focus groups. If we are not able to hold focus groups during a prescheduled grantee meeting, we will conduct phone or web-based focus groups. Participation in focus groups is voluntary, and grantees will not be penalized should they choose not to participate. Survey participants are eligible to participate in the focus groups, which are intended to collect rich data on evaluation technical assistance needs and tools.

Quality, Utility, and Clarity of Information Collected:

• (C) I like key informant interviews––setting up times to call participants and giving them a predicted length of time for the interview would be helpful.

(R) Data for this evaluation will be gathered through multiple methods, including surveys, focus groups, and, as time and funding resources allow, a limited number key informant interviews. Participants will be informed of the estimated time required to complete each data collection activity.

• (C) Decide the purpose of the information and what actions are to be taken before creating the surveys and interview guide.

(R) The evaluation focus areas are (1) existing evaluation capacity building tools and (2) grantees’ implementation of the new NCCCP Priorities. Participants will be informed of the purpose of each data collection effort, as well as plans to use the evaluation data for NCCCP program improvement.

• (C) Asking participants for input on how it is presented once preliminary information is compiled.

(R) Results of the evaluation will be disseminated through reports and manuscripts. We can gather feedback from grantees on additional dissemination strategies.

• (C) Consider packaging so that programs can use it to share with their coalitions to show national program trends that align with CDC priorities. This could be used as programs revise and update their cancer plans.

(R) Results of the evaluation will be disseminated through reports and manuscripts. The evaluation team will consider how to best package evaluation results to facilitate utilization of findings.

• (C) Reduce the number of questions asked and simplify to get quality information.

(R) The evaluation team has worked to keep the instrument as short as possible while still gathering information pertinent to the evaluation. Skip patterns are embedded in the survey to help minimize the amount of time required to complete it. Additionally, word limits have been placed on open-ended response field to encourage brief answers and minimize burden.

• (C) Be sure to communicate that grantees will also benefit from the findings.

(R) Survey and focus group introductory materials note that these data collection efforts are intended to inform improvements to the National Comprehensive Cancer Control Program. Data collected may help inform grantees’ efforts to implement the NCCCP Priorities. These benefits will be highlighted during a grantee call.

• (C) If the participant could save their responses and come back to them later that would give them a chance to look up information and give the most honest answer possible.

(R) Survey participants will have the option to save their responses and return to the survey at a later time to complete it. This will allow participants to consult with program staff, coalition members, and review program documents as needed to complete the survey.

CDC’s Plans for Feedback from Grantees of the CCC Program:

• (C) At a grantee meeting, it might be helpful to meet with CCC officials in small group discussions about what works and what doesn't.

(R) Planned data collection activities are limited by the level of resources available for the NCCCP evaluation. However, there are often informal opportunities for grantees to communicate with CDC staff about program achievements and opportunities for improvement during grantee meetings. This information will also likely be captured in responses to open-ended survey questions.

• (C) It would be very interesting to find out how cancer programs see themselves in the future, and in the light of healthcare reform, chronic disease, and public health. It seems that time and circumstance are moving forward, but the cancer programs are holding their ground in the past.

(R) Though this is an interesting topic, it is outside of the scope of the NCCCP evaluation. However, responses to open-ended survey items may provide some insights on this issue.

• (C) Allow for work to be done on the survey, saved, and returned to later. There is so much to complete it might facilitate more thoughtful and complete answers.

(R) Survey participants will have the option to save their responses and return to the survey at a later time to complete it. This will allow participants to consult with program staff, coalition members, and review program documents as needed to complete the survey.

• (C) Utilize the very cumbersome reports first, and then ask for very specific feedback that doesn't encourage duplication of effort that burns out the public and staff.

(R) Evaluation planning included thoughtful development and thorough review of data collection instruments to ensure that they complement, rather than duplicate, data collected through MIS. The NCCCP evaluation includes a review of any evaluation reports submitted by grantees.

• (C) Qualitative is better than quantitative for honest feedback––it gives the respondent freedom to express themselves.

(R) Qualitative and quantitative data will be collected from grantees.

• (C) Don’t limit feedback to states; ensure feedback from territories and jurisdictions is included.

(R) Respondents will include all CCC programs–– states, tribes, territories, and jurisdictions.

• (C) Be sure to keep the time commitment minimal.

(R) The evaluation team has worked to keep the instrument as short as possible while still gathering information pertinent to the evaluation. Skip patterns are embedded in the survey to help minimize the amount of time required to complete it. Additionally, word limits have been placed on open-ended response fields to encourage brief answers and minimize burden. Participants will be informed of the estimated time required to complete each data collection activity.