Part B_17AW_rev 4 06 17

Assessment of Targeted Training and Technical Assistance (TTA) Efforts on the Implementation of Comprehensive Cancer Control

Supporting Statement: Part B

April 6, 2017

Program Official/Project Officer:

Angela Moore, MPH

Public Health Advisor

Division of Cancer Prevention and Control

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Hwy NE, MS-F75

Atlanta, GA 30341-3717

OFFICE: 770-488-3094

FAX: 770-488-4335

Table of Contents

Part B: Collection of Information Employing Statistical Methods

B1. Respondent Universe and Sampling Methods

B2. Procedures for Collection of Information

B3. Methods to Maximize Response Rates and Deal with Nonresponse

B4. Tests of Procedures or Methods to be Undertaken

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Attachments

1 Authorizing Legislation: Public Health Service Act, Research and Investigation

2 Federal Register Notice

3a List of Awardees for DP13-1314 and DP13-1315

3b DP13-1314 and DP13-1315 Conceptual Framework

4 Evaluation Matrix

5a Case Study Interview Introductory Letter

5b Worksheet for Identifying Case Study Interviewees

5c Case Study Interview Introductory Email

5d Case Study Interview Matrix

6a Case Study Interview Guide for DP13-1314 Program Directors or Managers

6b Case Study Interview Guide for DP13-1315 Program Directors or Managers

7a Case Study Follow-Up Interview Guide for DP13-1314 Program Directors or Managers

7b Case Study Follow-Up Interview Guide for DP13-1315 Program Directors or Managers

8a Case Study Interview Guide for DP13-1314 Evaluators

8b Case Study Interview Guide for DP13-1315 Evaluators

9a Case Study Interview Guide for DP13-1314 Partners

9b Case Study Interview Guide for DP13-1315 Partners

10a Web-based survey Pre-notification Email

10b Web-based survey Invitation Email

10c Web-based survey First Reminder Email

10d Web-based survey Second Reminder Email

10e Web-based survey Thank You Email

10f Screen shots of Web-based Survey Multiple Options

10g Screen shots of Web-based Survey Single Option

11a In-depth Interview Recruitment Email

11b In-depth Interview Guide

12 Data Collector Non-Disclosure Agreement

13a Case Study Interview Informed Consent

13b In-depth Interview Informed Consent

14 IRB Approval Letter

B1. Respondent Universe and Sampling Methods

The goal of this information collection is to document the implementation of training and technical assistance (TTA) administered to 65 National Comprehensive Cancer Control Program (NCCCP) grantees and their partners, and 55 National State-Based Tobacco Control (NSBT) grantees and their partners. The TTA is administered by 10 organizations: 8 organizations that provide TTA received awards under funding opportunity announcement DP13-1314 and 2 organizations that provide TTA received awards under funding opportunity announcement DP13-1315.

The assessment of the DP13-1314 and DP13-1315 cooperative agreements will consist of three data collection activities: 1) case studies of DP13-1314 and DP13-1315 awardees; 2) a web- survey with NCCCP and NSBT grantees, partners and coalition members; and 3) in-depth interviews with NCCCP and NSBT grantees who report receiving the most TTA from DP13-1314 and DP13-1315 awardees. Exhibit 1 displays the expected number of respondents for each data collection activity by clearance year.

Exhibit 1. Number of Respondents by Data Collection Activity

Type of Respondents	Form Name	# Respondents Clearance Year 1	# Respondents Clearance Year 2	Total # Respondents
DP13-1314 and DP13-1315 Awardee Organizations	Worksheet for Identifying Case Study Interviewees	10	0	10
DP13-1314 Program Directors or Managers	Case Study Interview Guide for DP13-1314 Program Directors or Managers	81	0	8
	Case Study Follow-Up Interview Guide for DP13-1314 Program Directors or Managers	0	81	8
DP13-1315 Program Directors or Managers	Case Study Interview Guide for DP13-1315 Program Directors/ Managers	21	0	2
	Case Study Follow-Up Interview Guide for DP13-1315 Program Directors/ Managers	0	21	2
DP13-1314 Evaluators	Case Study Interview Guide for DP13-1314 Evaluators	81	0	8
DP13-1315 Evaluators	Case Study Interview Guide for DP13-1315 Evaluators	21	0	2
DP13-1314 Partners	Case Study Interview Guide for DP13-1314 Partners	162	0	16
DP13-1315 Partners	Case Study Interview Guide for DP13-1315 Partners	42	0	4
NCCCP and NSBT Program Directors, Staff, Coalition Members, and Partners	Web-based survey	1,5603	0	1,560
	In-Depth Interview Guide	10	0	10

¹ one per awardee

² two per awardee

³ Wave 1 = 65 NCCCP PDs + 55 NSBT PDs, N=120. Wave 2 = 120 X 3 = 360. Wave 3 = 360 X 3 = 1,080. Total N = 120 + 360 + 1,080 = 1,560

Case Studies

The respondent universe for case studies is comprised of the 10 organizations (Attachment 3a) that have been funded under DP13-1314 and DP-13-1315 cooperative agreements to provide support via training and technical assistance to National Comprehensive Cancer Control Programs (NCCCP) and National State-Based Tobacco Control (NSBT) Programs. As a part of the case studies, ICF (contractor) will conduct case study interviews with key staff and partners from DP13-1314 and DP13-1315 awardee organizations who are involved with the cooperative agreement and in the planning, implementation, and evaluation of training and technical assistance (TTA). Using purposive non-probability sampling, which is a sampling approach that facilitates the selection of individuals based on specific characteristics, 4 individuals from each of the 10 awardee organizations will be invited to participate in case study interviews, for a total of 40 respondents. The contractor will work with awardees to identify and recruit participants in accordance within specific target populations (e.g., involved in the planning, implementation, and evaluation of the cooperative agreement and TTA) so that their responses will provide meaningful contributions to our evaluation questions (Attachments 5a – 5c).

Web-based survey

The respondent universe for the web-based survey is comprised of individuals affiliated with either the NCCCP or NSBT programs that could have received TTA from DP13-1314 and DP13-1315 awardees and can include program directors, managers and staff, partners, and/or coalition members. The team will use a snowball sampling approach to administer the survey. The initial sample will include all 65 NCCCP grantee program directors and 55 NSBT grantee program directors, for a total of approximately 120 respondents. At the end of the survey, participants will be asked to provide contact information for up to three additional individuals from their staff, coalition, or partners that may have received TTA from one of the DP13-1314 and DP13-1315 awardees. Using this information, two additional waves of survey administration will occur, and referred individuals who participate in subsequent waves will also be asked to refer up to three additional people to whom we can send the survey. In total, we will collect information from up to 1560 individuals (Initial sample = up to 120 potential participants; Second wave: up to 360 potential participants; Third wave: up to 1080 potential participants).

In-Depth Interviews

The respondent universe for the in-depth interviews will include affiliates of the NCCCP and NSTB programs that participated in the web-based survey. Based on responses to the web-based survey, the contractor will identify 10 respondents that report receiving the most TTA from one or more of the DP13-1314 and/or DP13-1315 awardee organizations. These individuals will be recruited to participate in in-depth interviews to explore, in-depth, the type, quality, and perceived effectiveness of TTA received from DP13-1314 and DP13-1315 awardees.

B2. Procedures for Collection of Information

Case Studies

Interview guides tailored for each specific audience will be used to collect information from participants. All participants (program directors or managers, program evaluators, and partners) will be interviewed at time point one to occur immediately upon receiving OMB approval (beginning 2017) and program directors or managers will also be interviewed at time point two to occur before the before the end of the cooperative agreement, approximately 12-15 months after OMB approval. The first wave of interviews with program directors or managers, program evaluators, and partners will explore implementation of the cooperative agreement and TTA (fidelity, dose, intensity, duration); barriers, facilitators, and contextual factors that affect implementation of cooperative agreement and TTA; and perceptions regarding the quality and effectiveness of specific TTA efforts (Attachments 6a and 6b; 8a-9b). The second wave of interviews will be conducted with program directors or managers only and will follow up on many of the same topic areas, with an emphasis on any changes that occurred throughout the length of the cooperative agreement (Attachments 7a and 7b). All interviews will be conducted virtually via telephone.

Upon OMB approval, the contractor will send an Introductory Letter and Introductory Email to each DP13-1314 and DP13-1315 awardee program director or manager to provide them with information about the case studies and confirm their interest and willingness to participate (Attachment 5a and 5c). Within one month, the contractor will schedule a conference call with each awardee via email and work with a program director or manager from each awardee program to complete the Worksheet for Identifying Case Study Interviewees to identify appropriate respondents for each interview (Attachment 5b).

Two project team members will lead each interview, one as the lead interviewer and the other as the primary note-taker. Each interviewer will be trained in the full project protocol, including each of the tailored interview guides. At the start of each interview, the interviewer will read aloud the informed consent and ask the respondent to give verbal consent to participate and for the interview team to audio record the interview for analysis purposes (Attachment 13a). The interviewer will use the appropriate tailored interview guide to generate questions and probes for gathering information throughout the interview (see Attachments 6a-9b). Each interview will be audio recorded to serve as a back up to interview notes.

Web-based survey

The web-based survey will be administered one time, in 3 waves, using a snowball sampling approach. The contractor will manage the information collection process on CDC’s behalf. To generate the initial sample for the web-based survey, the contractor will work with CDC to gather contact information for all 65 NCCCP grantee program directors and all 55 NSTB grantee program directors, for a total of approximately 120 program directors. The contractor will compile this information into a master file for purposes of administration of the web-based survey.

For the web-based survey, all potential respondents will receive the following rounds of communication: a pre-notification email informing them of the web-based survey (Attachment 10a); an invitation email with the survey link (Attachment 10b); a first reminder email sent 1 week after the invitation email (if they have not yet responded to the survey) (Attachment 10c); a second reminder email sent 1 week after the first reminder (if they have not yet responded to the survey) (Attachment 10d); and a thank you email sent within 1 week of completion of the survey (Attachment 10e). Using a snowball sampling method, the survey will be administered to NCCCP and NBST grantee program directors, staff, partners, and coalition members (Attachment 10f and 10g).

Information will be collected and stored and maintained by the contractor and protected under data privacy policies. Both quantitative and qualitative analyses will be performed. Quantitative analyses, using SPSS, will involve using descriptive statistics to determine frequency distributions and corresponding variances for responses to each web-based survey question. Qualitative thematic analyses will be conducted on open-ended questions. Analysis will focus on describing: 1) the reach of DP13-1314 and DP13-1315 awardee TTA efforts; (2) the TTA received among respondents, including type, dosage, frequency and format; and (3) individuals’ perceptions of the effectiveness of the TTA received.

In-Depth Interviews

The contractor will follow similar procedures for conducting the in-depth interviews as those used for the case study interviews. To recruit in-depth interview participants, CDC will send a Recruitment Email to each selected NCCCP or NSBT program director to provide them with information about the interviews and confirm their interest and willingness to participate (Attachment 11a). At the start of each interview, the interviewer will read aloud the informed consent and ask the respondent to give verbal consent to participate and for the interview team to audio record the interview for analysis purposes (Attachment 13b). The interviewer will use a semi-structured interview guide to generate questions and probes for gathering information throughout the interview (Attachment 11b). Each interview will be audio recorded to serve as a back up to interview notes.

Collectively, the information collection will be used to answer the following key questions:

1. How are DP13-1314 and DP13-1315 awardees implementing the components of their respective cooperative agreement to build capacity among their target audience(s) (e.g., NCCCP and NSBT programs)?

1. To what extent are DP13-1314 and DP13-1315 awardees implementing TTA?

2. To what extent did DP13-1314 and DP13-1315 awardees achieve planned short-term outcomes for their respective cooperative agreements?

3. To what extent did DP13-1314 and DP13-1315 awardees’ TTA activities contribute to enhnaced program implementation and achievement in NCCCP and NSBT program priorities and goals?

4. What are the essential elements of a TTA model that build the capacity of NCCCP and NSBT grantees?

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Case studies will include the 10 organizations currently participating in the DP13-1314 and DP13-1315 cooperative agreements. To maximize response, the contractor will send multiple communications to each DP13-1314 and DP13-1315 awardee program director or manager to provide them with information about the case studies and confirm their interest and willingness to participate (Attachment 5a - 5c). The contractor will also provide awardees with support for identifying and scheduling interviews with appropriate respondents. In the event that one or more is unable or unwilling to participate, that organization will not be substituted with an alternate organization and the total number of case studies will be decreased. Participation is voluntary across all organizations.

For the web-based survey, to ensure that items and responses are understandable by respondents, six representatives from state health departments were asked to pilot the web-based survey. Answers were electronically submitted to the contractor who recorded pilot completion times. The pilot participants submitted, via a section in the electronic instrument or via email, their feedback on the clarity and efficiency of the web-based survey. Edits to the web-based survey were made based on this feedback. The average time to complete the data collection tool including time for reviewing instructions, gathering needed information and completing the data collection tool, was approximately 15 minutes. In addition, the contractor will use several rounds of communication reminders to help ensure a higher response rate.

For in-depth interviews, only affiliates who report receiving a high level of TTA support from one or more Dp13-1314 or DP13-15 awardee organizations will be included in data collection. This will maximize their ability to respond to questions and probes pertaining to their TTA experiences.

B4. Tests of Procedures or Methods to be Undertaken

CDC staff and contractors, who comprise the study team, were involved in the development, review, and approval of data collection instruments and other supporting documents. The survey was pilot tested with a small sample of NCCCP program directors to confirm the amount of time needed to complete the survey.

B5. Individuals Consulted on Statistical Aspects and Individual Collecting and/or Analyzing Data

CDC provides overall direction for all data collection planning and implementation activities, including overseeing the data collection protocol and data reporting.

The contractor, ICF International, will recruit and collect all data for all data collection activities described. ICF will also analyze and report assessment results.

The principal contacts for each organization are below.

Staff Name	Contact Information
CDC
Angela Moore, MPH Public Health Advisor	Phone: 770-488-3094 Email: cyq6@cdc.gov
Trina Pyron, MA Public Health Advisor	Phone: 770-488-0971 Email:dfo4@cdc.gov
ICF International
Sarah O’Dell, MPH Project Manager	Phone: 404-321-3211 Email: Sarah.Odell@icfi.com

Angela Moore, MPH

Public Health Advisor

Comprehensive Cancer Control Branch

Division of Cancer Prevention and Control

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Hwy NE, MS-F75

Atlanta, GA 30341-3717

OFFICE: 770-488-3094

FAX: 770-488-4335

Information will be collected and analyzed by CDC’s contractor, ICF International.