GenIC Request

Attachment 3

Request for Approval Under Generic Clearance for CDC Fellowship Programs Assessments (OMB Control Number: 0920-1163)

TITLE OF INFORMATION COLLECTION: CDC Evaluation Fellowship Program Assessment

Instruction: This form should be completed by the primary project representative at the CIO sponsoring the genIC, after consultation with the Center, Institute, or Office (CIO) PRA contact. An FTE is required to serve as the primary investigator for all information collection requests. The completed form should be routed from the PRA contact to DSEPD Information Collection Request Liaison Fátima Coronado, fcoronado@cdc.gov.

Instruction: Please provide no more than two sentences for each item in this box.

Goal of the study: The goal of this data collection is to inform the CDC Evaluation Fellowship Program service improvement, ongoing program management activities, and assessment of Fellowship outcomes.

Intended use of resulting data: CDC’s Program Performance and Evaluation Office (PPEO) will use the findings from the web-based surveys and phone interviews to identify the strengths of the Fellowship Program, best practices, and areas for improvement. Staff will use information to revise Fellowship components, identify areas for additional training for Fellows, and provide additional guidance to Host Programs and Mentors.

Methods to be used to collect data: Information will be collected through web-based surveys and individual phone interviews.

Subpopulation to be studied: Participants will include up to 150 Alumni who completed the PPEO Evaluation Fellowship Program within the last 10 years and up to 147 Mentors who have supervised fellows over the past 10 years. Participants may work within CDC, other governmental agencies, nongovernmental organizations, or other settings.

How data will be analyzed: For the surveys, descriptive analysis will be used for quantitative items (i.e., frequencies and crosstabs) and content analyses for qualitative items. Content analyses also will be conducted for interview data. Results will be reported in aggregate with no personal identifying information. Survey data will be brought together with interview data in a mixed methods approach.

CIO or Division PRA Contact
Name: Dan Kidder, PhD.

Email: dtk8@cdc.gov

Phone: 404.639.6270

Project Representative

Instruction: Complete the fields below with information about the project lead.

Name: Kimberley Freire

Title: Evaluation Lead

Affiliation (CIO/Division): OD/OADPS/PPEO

Email: kfreire@cdc.gov

Phone: 770-488-4994

Abbreviated Supporting Statement A

DETERMINE IF YOUR INVESTIGATION IS APPROPRIATE FOR THIS GENERIC CLEARANCE MECHANISM

Instruction: Before completing and submitting this form, first determine if the proposed investigation is appropriate for the Data Collection for CDC Fellowship Programs Generic ICR mechanism. Complete the checklist below. If you select “yes” to all criteria in Column A, the Data Collection for CDC Fellowship Programs Generic IR mechanism can be used. If you select “yes” to any criterion in Column B, the Data Collection for CDC Fellowship Programs Generic ICR mechanism cannot be used.

Column A	Column B
Information gathered is intended for CDC fellowship service improvement and program management purposes. [X ] Yes [ ] No	The investigation is conducted to contribute to generalizable knowledge. [ ] Yes [X ] No
Data collection will be completed in 90 days or less. [ X ] Yes [ ] No	Data collection is expected to require greater than 90 days. [ ] Yes [ X ] No
No incentive (e.g., money, reimbursement of expenses, token of appreciation) will be provided to participants. [ X ] Yes [ ] No	An incentive (e.g., money, reimbursement of expenses, token of appreciation) will be provided to participants. [ ] Yes [X ] No

Did you select “yes” to all criteria in Column A?

If so, the Data Collection for CDC Fellowship Programs Generic ICR might be appropriate for your investigation. You may proceed with this form.

Did you select “yes” to any criterion in Column B?

If so, the Data Collection for CDC Fellowship Programs Generic ICR is not appropriate for your investigation. Stop completing this form now and consult your PRA contact about alternatives.

PURPOSE

Instruction: Provide a brief description of the collection purpose and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

The CDC Evaluation Fellowship Program, started in 2011, represents a major commitment by CDC to program evaluation and program improvement. It signifies CDC’s investment in making program evaluation a standard part of practice and to developing a cadre of professionals with the skills to make that happen. The Fellowship is under the leadership of CDC’s Program Performance & Evaluation Office (PPEO), with Fellows placed in Host Programs where their day-to-day program evaluation activities occur.

Each Fellow serves for about two years, based at a Host Program across the agency with which they are matched. Fellows receive ongoing professional development, networking, and social/cohort-building opportunities. The Fellowship has a dual purpose of strengthening the capacity of CDC’s programs to conduct more and better monitoring and evaluation (M&E), as well as providing a strong professional development opportunity for new public health evaluators.

The tenth cohort of the Fellowship started in August 2020, and as we begin the second decade of the Fellowship, this data collection provides an opportunity for Fellowship Alumni and Mentors to provide important information that can be used to improve the program and assess its outcomes.

The purpose of this data collection is to assess the value of the Fellowship and determine whether, and to what extent, the Fellowship is reaching some of its key outcomes, particularly increasing evaluation capacity within CDC and among public health professionals who complete the Fellowship. All data will be analyzed with an eye towards identifying any recommendations for CDC, particularly improvements PPEO and the M&E Unit that manages the Fellowship can make in the next year and long term.

The objectives of this data collection are to asses: 1) the evaluation capacity of former Evaluation Fellows and changes since starting the Fellowship; 2) the evaluation capacity of Host Programs, specifically the extent to which participation in the Fellowship has helped create more institutional support and capacity for quality evaluation, and 3) the career trajectory of Evaluation Fellowship Alumni, specifically the extent to which participation in the Fellowship has benefited them professionally, after graduation/completion.

This quick, low-burden assessment is instrumental in helping PPEO learn about stakeholder perspectives, and it will yield immediate results that can be quickly used by program staff to make improvements to the current fellowship supports and future development. This information is not available from any other source.

DESCRIPTION OF RESPONDENTS

Instruction: Provide a brief description of the group(s) targeted for this information collection. These groups must have experience with the program.

Check all that apply.

[ ] Potential applicants or applicants

[ ] Current fellows (nonfederal employees)

[X] Alumni

[X] Mentors or supervisors

[ ] Employers of alumni

[ ] Other (describe): ____________________

TYPE OF COLLECTION

Instruction: Check all that apply.

[ ] Focus group

[ ] Face-to-face interview

[X] Telephone interview

[ ] Self-administered hard copy questionnaire

[X] Self-administered Internet questionnaire

[ ] Self-administered electronic questionnaire (e.g., fillable form)

[ ] Other (describe): ____________________

CERTIFICATION

Instruction: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is noncontroversial and does not raise issues of concern to other Federal agencies.

4. Information gathered will be used primarily to inform programs of efficiency and effectiveness of fellowship programs and will not be used for the purpose of substantially informing influential policy decisions.

5. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

6. With the exception of information needed to contact participants, personally identifiable information (PII) is collected only to the extent necessary and is not retained.

7. If this genIC requires collections of race and ethnicity data, the questions are consistent with HHS policy and standard OMB classifications.

8. A copy of the IRB approval or exemption determination with description of participation consent and secure collection, storage, and management of participant data and information is attached.

9. A currently valid OMB control number and expiration date is displayed in the upper-right corner at the beginning of the data collection instrument.

10. The following statement is displayed at the bottom of the first page of the data collection instrument or will be read to the participant prior to data collection: “Public reporting burden of this collection of information is estimated to average [number of] minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74 Atlanta, Georgia 30333; ATTN: PRA (0920-1163).”

    1. If the Privacy Act applies, the following statement is also included: “The Privacy Act applies to this information collection. The requested information is used toward assessment and continuous quality improvement of CDC fellowship activities and services. CDC will treat data/information in a secure manner and will not disclose, unless otherwise compelled by law.”

11. A Part II Worksheet is included in this submission.

Certified by CDC Sponsoring Program Division or CIO PRA Oversight Official:

Name:Dan Kidder, PhD.

Date of Certification (MM/DD/YYYY): 10/22/2020

Email: dtk8@cdc.gov

Phone: 404.639.6270

To assist review, please provide answers to the following questions:

Personally Identifiable Information

1. Is personally identifiable information (PII) collected? [X] Yes [ ] No

2. If Yes:

   1. Is the information that will be collected included in records that are subject to the Privacy Act of 1974?
      [X] Yes [ ] No

2. Please provide justification for collecting PII: Alumni and Mentor respondents will be asked if they are interested in participating in a follow-up interview. Respondents who select “yes” will click on a link to a separate form where they will complete their contact information. This information will not be linked to their survey responses and will be stored in a separate file.

3. Please describe efforts to use existing PII to avoid duplication (e.g., information from the Fellowship Management System [OMB No. 0920-0765], FedScope): Not applicable

4. In advance of any data collection, the following statement will be provided directly to the participant (e.g., in a written statement on a survey tool prior to beginning a questionnaire, read to participant prior to interview): “The Privacy Act applies to this information collection. The requested information is used toward assessment and continuous quality improvement of CDC fellowship activities and services. CDC will treat data/information in a secure manner and will not disclose, unless otherwise compelled by law.” Not applicable.

Sensitive Questions

Instruction: If sensitive questions will be asked, provide justification and specific use.

No sensitive questions will be asked.

BURDEN HOURS

Instruction: Complete Table 1 using the following column headings to calculate the burden hours for respondents.

• Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Potential applicants/applicants, (2) Current fellows (nonfederal employees), (3) Alumni, (4) Mentors or supervisors, (5) Employers of alumni, (6) Other (please describe).

• Form Name: Include the type of data collection (e.g., “Electronic survey of fellowship applicants,” “Telephone interview of recent graduates”).

• No. of Respondents: Provide an estimate of the number of respondents.

• No. of Responses per Respondent: Provide the number of times the same respondent will be contacted for data/information collection.

• Average Burden per Respondent (in hours): Provide an estimate of the amount of time required for a respondent to participate (e.g., time required to fill out a survey or participate in a focus group).

• Total Burden Hours: Provide the total burden hours by multiplying as follows:
  ([No. of Respondents] x [No. of Responses per Respondent] x [Average Burden per Respondent]) in each row. Then total the rows.

Table 1. Estimated Burden

Category of Respondent	Form Name	No. of Respondents	No. of Responses per Respondent	Average Burden per Respondent (in hours)	Total Burden Hours
Alumni (Completed the PPEO Evaluation Fellowship in previous 10 years) and current 2nd year fellows	Web-based survey	150	1	12/60	30
Current and Former Mentors	Web-based survey	147	1	10/60	25
A subset of Alumni (Completed the PPEO Evaluation Fellowship in previous 10 years) and current 2nd year fellows who completed the web-based survey.	Phone interviews	20	1	30/60	10
A subset of Current and Former Mentors who completed the online survey	Phone interviews	20	1	30/60	10
Totals		340			75

FEDERAL COST

Table 2. Estimated Cost to the Government

Staff or Contractor	Average Hours	Average Hourly Rate	Total Cost
FTE: Project Oversight, Instrument Development, Implementation, Analysis, and Reporting (GS-14 Step 7)	100	$65.31	$6,531.00
ORISE Fellow: Project Coordination, Instrument Development, Data Collection, Data Analysis, and Reporting	120	$26.71	$3,205.2020
Contractor: Instrument Development, Data Collection, Data Analysis, and Reporting	240	$125.00	$30,000.00
Total			$39,736.20

Link to U.S. Office of Personnel Management Pay Tables: ttps://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2020/general-schedule/

PROJECT SCHEDULE

Instruction: Provide an estimated schedule indicating start dates, allowing sufficient time for delays and unforeseen circumstances. Sample activities and time schedules are provided; please modify as needed.

Project Time Schedule
Activity	Time Schedule
Design methods and data collection instruments	By August 2020
IRB determination	By September 2020
Pilot test instruments with CDC employees who Alumni or Mentors	By mid-September 2020
Develop genIC request	By end of October
Submit genIC to ICRO (then ICRO into ROCIS)	By end of November, 2020
Receive OMB approval for genIC	By end of January 2021
Implement data recruitment and collection	As soon as genIC is approved or as indicated by the genIC data collection plan
Analyze data as planned	Approximately within 3 months of close of data collection
Produce technical report and lay audience fact sheets	Approximately within 6 months of close of data collection: communicate to leadership, program, or stakeholders about results and recommendations for improvement or actions
Submit findings for scientific publications, manuscript, presentation, and content on the CDC website	6 months or more from close of data collection.

Abbreviated Supporting Statement B

Selection of targeted respondents

Instruction: Please provide a description of how you plan to identify your potential group of respondents and how you will select them.

Respondents will consist of Alumni who completed the PPEO Evaluation Fellowship program (Fellowship) in the nine years prior to the survey (i.e., graduated 2012-2020), and Host Program Mentors who worked with at least one Fellow during this same time period. Seven potential respondents are in both the Alumni and Mentor categories and will be asked to complete both surveys. Because the survey is confidential, we will not combine the two surveys for the seven individuals in both categories so that their responses cannot be linked to specific individuals.

Most respondents are employees at CDC, other government agencies, or nongovernmental organizations (including academia and healthcare). Surveys will be sent to 86 Alumni who work at CDC and 64 Alumni who are external to CDC. Surveys also will be sent to the 147 current and former Mentors. Most Mentors (n=127) are current CDC employees, and 20 former Mentors are external to CDC. No sampling will be employed.

Administration of the instrument

Instruction: Identify how the information will be collected.

1. How will you collect the information? (Check all that apply)

[X] Electronic (surveys)

[X] Telephone (interviews)

[ ] In-person

[ ] Hard copy

[ ] Other, explain: ____________________

2. Will trained interviewers or facilitators be used? [ X ] Yes [ ] No [ ] N/A

Methods to maximize response

Instruction: Provide a brief description of the procedures planned to maximize response rates.

An introduction email (Attachment A) will first be sent from PPEO staff to encourage participation. Next, a recruitment email (Attachment B) will be sent by the Contractor. The Contractor will send follow-up emails (Attachment C) to non-respondents after one and two weeks. The survey will be open for three weeks and will close one week after the second follow-up email.

Analysis plan

Instruction: Provide a brief description of the analysis plan, including quality control procedures, and estimation procedures

For the surveys, descriptive analysis will be used for quantitative items (i.e., frequencies and crosstabs) and content analyses for qualitative items. Content analyses also will be conducted for interview data. Results will be reported in aggregate with no personal identifying information. Survey data will be brought together with interview data in a mixed methods approach.

Pilot testing

Instruction: Provide a brief description of pilot-test efforts.

Survey instruments were reviewed by10 current CDC employees who were either Evaluation Fellowship Alumni, Mentors, or both.

• Alumni: average 10 minutes (range 8-12 minutes)

• Mentor: average 7 minutes (range 5–10 minutes)

The estimate for burden hours is based on average times from the pilot test, using the higher estimate for the Alumni only (12 minutes) and Mentor only (10 minutes) categories. The seven potential respondents who are in both the Alumni and Mentor categories will be asked to complete both surveys, and the average burden hours for this group is 22 minutes per respondent total.

Instruction: Describe efforts to improve or refine the instruments based on the pilot-test findings and feedback.

[ ] No changes necessary, based on pilot-test findings and feedback.

[X] Changes (please describe): ___________________

Changes were made to the wording of some questions on both the Alumni and Mentor surveys to clarify the intention of the questions and/or revise language to align with terminology commonly used by Alumni and Mentors. In addition, some items from the Mentor survey were removed or integrated with another question to streamline and eliminate redundancies.

Consultation on statistical aspects

Were outside agencies, partners, or organizations consulted on statistical aspects of the design?

[ ] Yes

[X] No

Analyses will be descriptive (i.e., frequencies, crosstabs) and will not include advanced statistical analyses or probability sampling.

If yes, list the following information of all persons consulted.

Name: __________________

Agency/organization (e.g., companies, state or local governments): __________________

Title: __________________

Telephone number: __________________

Email address: __________________

Please ensure that all instruments, instructions, and scripts are submitted with this request.

List of Attachments

Attachment A – Intro_Email_GenIC_PPEO

Attachment B – Recruitment _Email_GenIC_PPEO

Attachment C – Follow-up_Email_ GenIC_PPEO

Attachment D – Mentor_Survey_ GenIC_PPEO

Attachment E – Mentor_ Interview_Recruitment_Form GenIC_PPEO

Attachment F – AlumniSurvey_GenIC_PPEO

Attachment G – Alumni_Interview_Recruitment_Form GenIC_PPEO

Attachment H – Alumni_Interview_ GenIC_PPEO

Attachment I – Mentor_Interview_ GenIC_PPEO

Attachment J – Combined_Interview _ GenIC_PPEO

PPEO Evaluation Fellowship Project Determination Description

PPEO Project Determination Approval

DATE SUBMITTED TO DSEPD INFORMATION COLLECTION REQUEST LIAISON (ICRL)

Instruction: Please indicate the date (MM/DD/YYYY) the request is submitted to the ICRL.

10/29/2020

Email the completed form to the DSEPD Information Collection Request Liaison, Fátima Coronado, at fcoronado@cdc.gov.

13