CPHP_Eval_Supporting_Statement_CDCFinal

OMB Information Collection Request Supporting Statement

Project Title:

Centers for Public Health Preparedness Program Evaluation

Program Official/Project Officer:

Alyson Richmond, Public Health Analyst/CPHP Project Officer

Coordinating Office for Terrorism Preparedness and Emergency Response

arichmond@cdc.gov

Phone: 404-639-7497

Fax: 404-639-7424

Date:

June 19, 2009

Table of Contents

A. Justification

1. Circumstances Making the Collection of Information Necessary………………...1

2. Purpose and Use of the Information Collection…………………………………..2

3. Use of Improved Information Technology and Burden Reduction……………….3

4. Efforts to Identify Duplication and Use of Similar Information………………….3

5. Impact on small Businesses or Other Small Entities……………………………...3

6. Consequences of Collecting the Information Less Frequently……………………3

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5……………..3

8. Comments in Response to the Federal Register Notice and Efforts…………...….3

to Consult Outside the Agency

9. Explanation of Any Payment or Gift to Respondents……….................................4

10. Assurance of Confidentiality Provided to Respondents……………………....…..4

11. Justification for Sensitive Questions…………………...........................................5

12. Estimates of Annualized Burden Hours and Costs………………………………..5

13. Estimates of Other Total Annual Cost Burden to Respondents and Record……...7

Keepers

14. Annualized Cost to the Federal Government……………………………………...7

15. Explanation for Program Changes or Adjustments……...……………………......8

16. Plans for Tabulation and Publication and Project Time Schedule………………..8

17. Reason(s) Display of OMB Expiration Date is Inappropriate…………………...10

18. Exceptions to Certification for Paperwork Reduction Act Submissions………...10

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods…...................................................11

2. Procedures for the Collection of Information………………………………...….13

3. Methods to Maximize Response Rates and Deal with Nonresponse….................14

4. Test of Procedures or Methods to be Undertaken…………………………….….15

5. Individuals Consulted on Statistical Aspects and Individuals Collecting.............15

and/or Analyzing Data

Exhibits

Exhibit 1: CPHP Evaluation Work Group………………………………………………...4

Exhibit 2: Estimated Annualized Burden Hours…………………………………………..6

Exhibit 3: Estimated Annualized Burden Costs…………………………………………...7

Exhibit 4: Annualized Cost to the Federal Government………………………………......8

Exhibit 5: Project Time Schedule………………………………………………………....8

List of Attachments

Applicable Law or Regulation………………………………………………Attachment A

60 Day Federal Register Notice……………………………………………..Attachment B

CPHP Interview Instrument ….……………………………………………..Attachment C

CPHP Customer/Partner Survey Instrument……………...…..……………..Attachment D

Customer/Partner Follow-Up Interview Instrument ………....……………..Attachment E

Telephone Call Establishing Initial Contact Information........... ....................Attachment F

for Customer/Partner Survey

Email Follow-Up to Customer Partner Survey Contact..................................Attachment G

Telephone Follow-Up to Customer Partner Survey…………………...…….Attachment H

A. Justification

1. Circumstances Making the Collection of Information Necessary

The classification of this Information Collection (IC) is new. Under the Authority of Sections 301(a) and 317(k) (2) of Public Health Service Act, the Centers for Disease Control and Prevention is responsible for administering and monitoring the Centers for Public Health Preparedness (CPHP) Program. The purpose of the CPHP Program is to strengthen terrorism and emergency preparedness by linking academic expertise to state and local health agency needs. The program brings together twenty seven accredited Schools of Public Health, funded through a 5-year Cooperative Agreement for years 2004-2009, with a common focus on public health preparedness to establish a national network of education and training resources. This goal of this CPHP Program Evaluation is to determine the value-added of the CPHP Cooperative Agreement program, identify outcomes of the collective work generated by the CPHPs, and inform the development of a new public health preparedness education and training Cooperative Agreement program. Section 238(j), Evaluation of Programs, of the Public Health Service Act (42 U.S.C. 241) is the authorizing law for this data collection (Attachment A).

Privacy Impact Assessment

Overview of the Data Collection System

The data collection system consists of 3 data collection instruments used to gather information describing CPHP Program implementation, outcomes and impact. These 3 data collection instruments are: 1) CPHP Interview Instrument (Attachment C), 2) CPHP Customer/Partner Survey Instrument (Attachment D), and 3) CPHP Customer/Partner Follow-Up Interview Instrument (Attachment E). The first and third instruments will be administered by telephone interview. The second instrument will be administered using a web-based survey or paper-based survey depending on respondent preference.

Data collection will be conducted by John Snow, Inc. (JSI), a public health consulting firm contracted to undertake the CPHP Program Evaluation project. Data collection will occur once per respondent for each instrument and will be utilized for a final evaluation report. It is anticipated that data collection will begin within two weeks of Office of Management and Budget (OMB) approval. The collected information will be maintained until completion of the evaluation or approximately 2 years following initial data collection.

Items of Information to be Collected

Data to be collected include multiple items regarding respondents’ perceptions about the effectiveness of the CPHP Program. Information will also be collected identifying programmatic outputs and outcomes.

Identifying information from CPHP Program staff and partners, such as name, organization, and contact information, will be known only to JSI, the data collection contractor. This information is only necessary for the purpose of scheduling data collection activities. Data collection records will not be maintained with individually identifying information and procedures will be followed to limit the linkage of this information to response data as described in Part A.10. (page 4). No identifiable information describing individual respondents will be included in the analyzed data and aggregate reports provided to CDC.

Identification of Website(s) and Website Content Directed at Children Under 13 Years of Age

The pool of survey respondents will be referred to a web-based tool specifically designed for web-based survey data collection (Survey Monkey). Only those respondents receiving the web link to the survey will have access to the web-based tool. For informational purposes, the survey instrument also includes references to specific web pages within the CDC internet site describing the CPHP Program. CDC’s website has a privacy policy and indicates the use of cookies. CDC’s website privacy policy can be found via the following link: http://www.cdc.gov/doc.do?id=0900f3ec80093c90. The web-based tool will be constructed within Survey Monkey. The following is a link to Survey Monkey’s privacy policy: http://www.surveymonkey.com/Monkey_Privacy.aspx. It is noted within this policy that the site does not function without the use of cookies. None of the content on these websites is directed at children less than 13 years of age.

2. Purpose and Use of the Information Collection

Data collected will be used by the CPHP Program staff at CDC for a one-time evaluation report for partners and other stakeholders, including Congress. The report will detail the value-added of the 5-year CPHP Cooperative Agreement program and identify outcomes of the collective work generated by the CPHPs. The report will also include findings and recommendations that will inform the development of a new public health preparedness education and training Cooperative Agreement program, slated to begin in the spring of 2010. This new Cooperative Agreement is mandated by the Pandemic and All-Hazards Preparedness Act of 2006. Funding is expected to be provided for both the current and future training and education Cooperative Agreements discussed in this Information Collection Request. This is a program evaluation and should be noted that though this is an essential project and has great utility as discussed above, there are limitations to the generalizability of the results to a universal training and education sample.

Privacy Impact Assessment Information

For the purpose of scheduling interviews with CPHP Program staff and partners familiar with the CPHP Program, it will be necessary to obtain identifying information such as name, organization, and contact information prior to the interview. As a result, interview respondents will be pre-identified as those who could provide useful data upon their consent. Contact information for the CPHP Program staff is currently known to CDC and will be provided to the data collection contractor for the purpose of scheduling an interview appointment with consenting respondents. Similarly, the process of building the sample for the CPHP Customer/Partner Survey and Interview data collections will require development of a dataset including names, organization and contact information of potential respondents. However, in all instances this information will be maintained in a separate record file for the purpose of managing the interview and survey processes only and will not at any time be recorded on the data collection instruments. The data collection instruments do not require information from respondents in an identifiable form.

Identifying information will be used only to schedule the interview process and to highlight accomplishments of individual CPHPs. No identifiable information describing individual respondents will be included in the analyzed data and aggregate reports provided to CDC. However, names of organizations may be used. The proposed data collection will have little or no effect on the respondent’s privacy. Highly sensitive information is not necessary for the purposes of this program evaluation and will not be collected.

3. Use of Improved Information Technology and Burden Reduction

The CPHP Customer/Partner Survey will be a web-based survey and disseminated via e-mail. All responses will be collected within the web-based tool, reducing burden on the respondent. 100% of responses for the CPHP Customer/Partner Survey will be electronic submissions, unless requested in paper form by a respondent.

4. Efforts to Identify Duplication and Use of Similar Information

No similar data are available.

5. Impact on small Businesses or Other Small Entities

No small businesses will be involved in this data collection.

6. Consequences of Collecting the Information Less Frequently

This request is a one-time data collection. Without collecting the data, no evaluation of the CPHP Program would occur, outcomes of the program would not be identified, and recommendations to inform the development of a new emergency preparedness and response training and education program would not be generated. There are no legal obstacles to reduce the burden on the respondents.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This request fully complies with the regulation 5 CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A 60-day Federal Register Notice (FRN) was published in the Federal Register on Friday January 16, 2009, vol. 74, No. 11, pp. 3047-3048 (Attachment B). There were no public comments.

Six representatives with evaluation experience were selected from the staff at existing CPHPs to form the CPHP Evaluation Work Group. This group serves as expert consultants to provide feedback on the evaluation plan, data collection instruments, and analysis throughout the entire CPHP Program evaluation project timeline (September 2008 – September 2010). Names, titles, telephone numbers, e-mail addresses, and organization represented for each work group member is listed below:

Exhibit 1: CPHP Evaluation Work Group

	Title	Phone	E-mail	Organization
Laura Biesediecki	Senior Program Manager	(202) 296-1099 ext. 152	lbiesiadecki@asph.org	Association of Schools of Public Health 1101 15th St, NW, Ste 910 Washington DC 20005
Kay Carpender	CPHP Project Coordinator	(979) 458-4313	skcarpender@srph.tamhsc.edu	Texas A & M University CPHP 1266 TAMU College Station, TX 77843-1266
Linda Degutis	CPHP Principal Investigator	(203) 785-3917	linda.degutis@yale.edu	Yale University CPHP 60 College Street, P.O. Box 208034 New Haven, CT 06520-8034
Lisle Hites	CPHP Evaluator	(520) 626-4439	lhites@email.arizona.edu	University of Arizona CPHP 1295 N. Martin Avenue Tucson, AZ 85724
Jon Links	CPHP Principal Investigator	(410) 955-9622	jlinks@jhsph.edu	Johns Hopkins University CPHP 615 N. Wolfe Street, Room E7531 Baltimore, MD 21205

9. Explanation of Any Payment or Gift to Respondents

No payment or gift will be offered to respondents.

10. Assurance of Confidentiality Provided to Respondents

Although the data collection contractor, JSI, will use identifiable information, such as names, telephone numbers or email addresses, to facilitate administration of the data collection process; this information will be maintained in a separate record file and will not be linked to response data. The data collection instruments ask respondents about their organizational or program experience and do not include items requesting information in an identifiable form. Furthermore, no identifiable information about respondents will be included in the data provided to CDC. Interview responses will be recorded on audiotapes, the corresponding transcripts, and in written notes taken at the time of the interview. The tapes will be transcribed, verified, and then destroyed at the end of the project. All transcripts and notes will be stripped of any individual names or other identifying information, although organizational names will remain as part of the data maintained by the data collection contractor. Any data provided to and maintained by CDC associated with this program evaluation will not be retrievable by name or other identifying characteristics of the individual respondents.

Privacy Impact Assessment Information

A. This submission has been reviewed by ICRO, who determined that the Privacy Act does not apply.

B. All IIF will remain with the contractor, JSI, and will not be shared with CDC. JSI datasets are maintained in secure password protected files stored on their computers. Case records will be identified by a random identifier. Any personally identifying information, necessary only for purposes for managing interview and survey administration, will be maintained as separate documentation. All electronic files maintained by JSI are protected by constantly updated firewall technology and active monitoring and management of network/port security. As described in Section 10 above (pages 4-5), all responses will be de-identified from the respondent prior to sharing the analyzed data with CDC.

C. Opportunities for voluntary consent will be provided to every respondent within each instrument (refer to Attachment C: CPHP Interview and Attachment D: CPHP Customer/Partner Survey). As part of the introduction and consent process at the time of the data collection request, each respondent will be provided with information describing the purpose of collecting the information and how the data will be used.

D. Respondents will be informed of the voluntary nature of their responses. There will be no effect on the respondent should they not respond to the data collection request. Data will be used solely for the purposes of this program evaluation and will not be shared outside JSI and CDC other than in final report form. The legal authority for this data collection is Section 238(j), Evaluation of Programs, of the Public Health Service Act (42 U.S.C. 241) is the authorizing law for this data collection (Attachment A).

11. Justification for Sensitive Questions

No questions of a sensitive nature will be asked of respondents.

12. Estimates of Annualized Burden Hours and Costs

A. The total number of respondents has decreased since the 60-Day FRN was published. The Pre-CPHP Interview Document Collection Protocol involves CPHP staff gathering and organizing existing CPHP data and is not subject to OMB approval. In addition, the number of respondents to the CPHP National Partner Interview Instrument will be 9 or less and so it is not subject to OMB approval. The number of respondents to the CPHP Customer/Partner Follow-Up Interview has also decreased since the 60-Day FRN was published. The numbers of respondents to the CPHP Interview and the CPHP Customer/Partner Survey have increased slightly due to a change in the unit of analysis (individual level versus organizational level) but have not caused an increase in the original projected burden hours. Therefore, the total number of respondents has decreased as well as the total burden hours (from 277 hours to 193.5 hours).

The CPHP Interview will have up to 81 total respondents (3 respondents per each of the 27 CPHPs). The 3 respondents at each CPHP will be interviewed one time collectively, and each interview is expected to last no longer than 90 minutes. The CPHP Customer/Partner Survey Instrument will have a total of 171 respondents (60 State Government, Local Government, Territorial Government respondents, 81 CPHP customer/partner respondents, and approximately 30 additional customer/partner respondents as identified by CDC CPHP staff ) and should take each respondent 20 minutes to complete. It is expected that no more than 20 CPHP Customer/Partner Follow-Up Interviews will be completed out of the 171 CPHP Customer/Partner Survey respondents, and the interview will last no more than 45 minutes. These projected response times are less than were listed in the 60-Day FRN due to survey instruments not being finalized at that time. Since then, they have been made more concise and will pose less of a time burden on the respondents. The following table presents a summary of the estimated total burden requested for this clearance:

Exhibit 2: Estimated Annualized Burden Hours

Type of Respondent	Form Name	No. of Respondents	No. of Responses per Respondent	Average Burden per Response (in hours)	Total Burden (in hours)
CPHP PIs, PCs, and Evaluators	CPHP Interview Instrument	81	1	1.5	121.5
CPHP Customers and Partners	CPHP Customer/ Partner Survey Instrument	171	1	20/60	57
CPHP Customers and Partners	CPHP Customer/ Partner Follow-Up Interview Instrument	20	1	45/60	15
Total 193.5 hours

B. The mean hourly wage is based on the United States national average for 2007 taken from the Bureau of Labor Statistics website.

Exhibit 3: Estimated Annualized Burden Costs

Form Name	Total Burden (in hours)	Hourly Wage Rate	Total Respondent Costs
CPHP Interview Instrument	121.5	$42.84*	$5,205.06
CPHP Customer/Partner Survey Instrument	57	$26.77**	$1,525.89
CPHP Customer/Partner Follow-Up Interview Instrument	15	$26.77**	$401.55
Total $7,132.50

*Wage estimate is for Education Administrators in Colleges, Universities, and Professional Schools

**Wage estimate is calculated as the average of hourly wages for Education, Training workers in state government and Epidemiologists in local government

Source: U.S. Department of Labor, Bureau of Labor Statistics, May 2007 National Occupational Employment and Wage Estimates. http://www.bls.gov/oes/current/oes_nat.htm

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no capital or maintenance costs associated with this IC.

14. Annualized Cost to the Federal Government

The external (contractor) costs to the federal government for conducting this program evaluation requiring OMB clearance is $160,000.00 for the entire length of the project. Costs for contract labor hours include evaluation planning and design, development of instruments, recruitment, data collection, preparation, analysis, report writing, and dissemination of findings. The government costs include personnel costs for federal staff involved in project oversight and development of this Information Collection Request; these efforts involve approximately 10% of a GS-13 public health analyst, 10% of a GS-13 scientist, and 5% of a GS-15 scientist. The total cost to the Federal government is $183,668.00.

Exhibit 4: Annualized Cost to the Federal Government

Labor:
CDC personnel for project oversight (10% GS-13, 10% GS-13, 5% GS-15)	$23,668.00
Contract labor for planning and design, data collection, data analysis, and dissemination of findings	$160,000.00
Total estimated government costs	$183,668.00

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Exhibit 5: Project Time Schedule

A.16 – 1 Project Time Schedule
Activity	Time Schedule
Pre-notification letter sent to potential CPHP interviewees and CPHP Customer/Partner respondents	Within 2 weeks after OMB approval
CPHP Interview data collection initiated	1 month after OMB approval
CPHP Customer/Partner Survey data collection initiated	1 month after OMB approval
CPHP Customer/Partner Follow-Up Interview data collection initiated	2 – 3 months after OMB approval
All data collection complete	4 - 12 months after OMB approval
Data cleaning, analysis, completion of evaluation report, publication and dissemination of report	12-16 months after OMB approval

Analysis Plan

Under the guidance and direction of CDC, JSI will conduct quantitative and qualitative analyses of the various data collected. The analysis and interpretation of the data collected will determine the value-added of the CPHP 5-year Cooperative Agreement program, identify outcomes of the collective work generated by the CPHPs, and inform the development of a new public health preparedness education and training Cooperative Agreement program.

JSI and CDC will determine the final program evaluation findings through the following steps:

Part 1: Content Analysis of Qualitative Data (CPHP Semi-Structured Interview, CPHP Customer/Partner Follow-Up Semi-Structured Interview)

1. Pre-Analysis: Debriefings with the interviewer and note-takers will be conducted immediately after each interview session. The debriefing will include the interviewer reviewing major themes for each topic. Each semi-structured interview will be transcribed, verified with the written notes and cleaned to eliminate any information in an identifiable form.

2. Content Analysis: The transcription data will be imported into a software application that aids the process of sorting, arranging and classifying qualitative information (e.g., NVivo, Atlas). The data will be systematically reviewed to identify coding categories. Coding categories will be operationally defined to allow precise categorization to reflect the purpose of the research; coding categories may also be removed or revised as the analysis continues. Coding categories will be compared and contrasted to determine if patterns exist within the data. Pattern coding will be then used to group the coding categories into a smaller number of sets, themes, or constructs. Summary sentences will be developed to describe all of the themes and statements within a particular pattern or code; a frequency count will be included. In addition, coding categories will be mapped across all instruments included in the evaluation.

Part 2: Analysis of Quantitative Data (CPHP Customer/Partner Survey):

The purpose of the CPHP Customer/Partner Survey is to gather information from representatives of organizations who have participated in CPHP-sponsored training or technical assistance activities describing their level of satisfaction with and perception of outcomes and impacts from the CPHP Program. The analysis of survey findings will include the following steps:

1. Pre-Analysis: As described, the survey will be administered electronically with an option for paper copy administration at the respondent’s preference. All written surveys received will be reviewed for ambiguous or non-responsive marks requiring resolution. Written survey responses will be double-key entered and reviewed to assure data entry quality. It is anticipated that most surveys will be completed online and a high number of written surveys is not anticipated. Online responses will be downloaded in a data file and merged with the written survey responses for final import into a statistical application.

2. Analysis: Data collected in response to the quantitative (close-ended) questions in the surveys will be analyzed using SPSS. The majority of quantitative items on the survey instrument are 5-point Likert scales. Simple descriptive statistics will be computed for the data collected from each question on the survey (e.g., frequency of each response and mean, median, and mode as appropriate). The responses will be tallied and the proportion of each type of response will be reported. In addition, the responses may be dichotomized into “strongly agree” plus “agree” responses and those with neutral or negative responses. The number and proportion of responses in these categories will also be reported. Tables and figures may be used to provide a visual representation of the responses. Where possible, differences in responses by categories of respondents such as organizational type, geographical region or degree of familiarity with the CPHP Program will be examined using advanced statistical tests such as the t-test, Analysis of Variance (ANOVA), and/or chi-square. However, as described in Part B of the Supporting Statement, it is recognized that with the exception of state health department respondents, results from this level of analysis may not be generalizable to the universe of potential customers or partners of the CPHP Program.

While the majority of data collected via survey will be quantitative, some open-ended questions are included on the instrument as well. Analysis of this qualitative data from the survey will be analyzed and reported in a manner similar to that already described in Part 1 of the Analysis Plan (page 9).

17. Reason(s) Display of OMB Expiration Date is Inappropriate

Display of the OMB Expiration Date is appropriate for this information collection.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification.

B. Collections of Information Employing Statistical Methods

The program evaluation of the CPHP Program is summative and retrospective in design. Further, the evaluation will focus on the aggregate outcomes of the CPHP Program and not on individual Centers. The overall sample of respondents includes key staff from all 27 CPHPs, representatives of organizations that may have utilized emergency preparedness training or technical assistance resources developed, delivered, or sponsored by the CPHP Program (i.e. customers), and a selection of key partners.

As described in Part A of the Supporting Statement, CDC will employ 3 related data collection instruments to gather information to complete the evaluation of the CPHP Program. These are: (1) CPHP Interview Instrument (Attachment C), (2) CPHP Customer/Partner Survey Instrument (Attachment D), and (3) CPHP Customer/Partner Follow-Up Interview Instrument (Attachment E). The respondent universe and sample for each instrument is described next.

1. Respondent Universe and Sampling Methods

The potential respondent universe for the CPHP Interview Instrument includes key staff from each of the 27 CPHPs across the country. It is our intent to sample all 27 CPHPs by interviewing up to 3 key staff together, such as the Principal Investigator, Center Director/Coordinator and/or the Center Evaluator of each Center (n = 81). The plan to survey the universe of CPHPs is justified by the small number of Centers, the low time burden of response, and the importance of complete CPHP representation because of the varied characteristics of the Centers (e.g. years of operation, areas of programmatic emphasis, and geographical location). It is anticipated that a participation rate of above 80% of CPHPs will be achieved.

The potential respondent universe for the second and third data collection instruments, the CPHP Customer/Partner Survey Instrument and Follow-Up Interview respectively, is a selection of 171 key partners and customers. Customers and partners may be representatives of organizations that have utilized emergency preparedness training or technical assistance resources developed, delivered, or sponsored by the CPHP Program, such as state and local health departments and a variety of other governmental and non-governmental organizations involved in public health emergency planning and response. Identical versions of the survey will be distributed to two groups from the following potential sample frame:

1. Customers and partners identified by the CPHPs. Each CPHP will identify up to 3 non state and major territory customers and/or partners they have worked with during the Cooperative Agreement (1 respondent from each CPHP-identified organization for a total of 81 respondents). All those identified will be contacted as respondents. Because the representatives of these organizations will be expected to have some familiarity with the CPHP Program, an 80% or higher survey response rate is anticipated. With the exception of state health department respondents, results from this level of analysis may not be generalizable to the universe of potential customers or partners of the CPHP Program. However, this information will still be very useful for informing the development of the new Cooperative Agreement and will also add insights on the accomplishments of the CPHP Program to date.

2. Representatives of organizations who may have utilized emergency preparedness training or technical assistance resources developed, delivered, or sponsored by the CPHP Program This sample will include Preparedness Coordinators within state and major territorial health departments, as identified by the CDC Public Health Emergency Preparedness (PHEP) Cooperative Agreement program (n = 60). The plan to survey state and major territorial health departments is justified by the small number of states and major territories, the low time burden of response, and the importance of complete representation of this group. Additionally, the PHEP Cooperative Agreement between CDC and states directs the states to work in conjunction with the CPHP Program to build workforce preparedness and response capabilities. The inclusion of each state and major territory in this evaluation seems appropriate and it is reasonable to anticipate a response rate around 90% in this group.

Thirty respondents in this group will include those from partner organizations such as the Association of State and Territorial Health Officials or the National Association of City and County Health Officials, American Indian and Alaska Native Tribal entities, and other non-profit groups that may have had contact with the CPHP Program. In order to gather accurate evaluation data, respondents that are, at a minimum, familiar with the program should be part of this sample frame. The universe of organizations in this group is very large and unknown, and within this universe, the data collection effort intends to focus on organizations with some knowledge of and experience working with the CPHP program. However, a complete set of these organizations is also unknown. The 27 CPHPs have worked with or provided training to thousands of such organizations over the period of the Cooperative Agreement. Therefore, it is not possible to develop a complete sample frame from which a random sample of 30 potential respondents that have used the services of the CPHP program can be drawn. It is important, nevertheless, for this class of organizations to be represented in the evaluation. Because the representatives of these organizations will be expected to have some familiarity with the CPHP program, we anticipate a reasonably high survey response rate of around 70%. We recognize that, with the exception of state health department Preparedness Coordinator respondents, results from this level of analysis may not be generalizable to the universe of potential customers or partners of the CPHP program. However, this information will still be very useful for informing the development of the new Cooperative Agreement and will also add insights on the accomplishments of the CPHP program to date.

The sample frame for the CPHP Customer/Partner Follow-Up Interview is a volunteer sample from the pool of respondents to the CPHP Customer/Partner Survey Instrument. Survey respondents will be provided the opportunity to reply under separate cover (so that responses are not linked to identifying information) with an indication of their willingness to participate in a follow-up semi-structured interview. The purpose of this data collection is to gather follow-up, additional qualitative data that explains the quantitative findings from the survey. Considering this intent, a representative sample of 20 respondents will be generated from the pool of volunteer respondents including those whose responses were both positive and negative regarding the CPHP Program. Of those who volunteer for a follow-up interview, it is anticipated that 100% will participate.

2. Procedures for the Collection of Information

As previously described, the procedures for information collection include semi-structured telephone interviews and web-based or paper surveys. All data collection will be completed by employed staff of the contractor, JSI.

Potential respondents for the CPHP Interview Instrument include the universe of CPHPs, thus sampling or estimation procedures are not required. The method of information collection is by semi-structured telephone interview. As described in the next section, respondents will have advance notice and will have scheduled appointments for the information collection. Semi-structured interviews will be conducted by JSI staff with training and experience in interview techniques and qualitative evaluation. Note takers will similarly have appropriate background and training. Transcription will be completed by a JSI staff member certified in transcribing.

Potential respondents for CPHP Customer/Partner Survey Instrument include the universe of Preparedness Coordinators for all state and major territorial health departments, as well as a large and diverse set of other governmental and non-governmental organizations involved in public health emergency preparedness planning and response. While sampling is not necessary for the first group (state, local, and territorial health departments), the challenges of developing a sample frame for the second group and the process for developing the sample are described in the preceding section. The survey will be primarily administered on-line, but a paper copy will also be available to potential respondents as many smaller, rural local health departments may prefer not to use web-based surveys; respondents will be offered the method most convenient for them. This flexibility should increase response rates and minimize perceived burden. The web administration option also should contribute to maximizing the timeliness, efficiency, and response rate of data collection.

Potential respondents for the CPHP Customer/Partner Follow-Up Interview Instrument will include those who volunteer for a follow-up interview in the CPHP Customer/Partner Survey. This is intended to add descriptive qualitative information to the quantitative survey findings. The method of information collection is by semi-structured telephone interview. Respondents will have advance notice and scheduled appointments for the information collection. Semi-structured interviews will be conducted by JSI staff with training and experience in interview techniques and qualitative evaluation. Note takers will similarly have appropriate background and training and transcription will be completed by a JSI staff member certified in transcribing.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Following OMB approval, interview and survey materials displaying the OMB approval number and expiration date will be sent electronically to respondents. With respect to the CPHP Interview Instrument, key CPHP staff to be interviewed will be contacted in advance to establish a time for an interview. They will also receive an advance copy of the interview instrument in order to allow thoughtful responses and efficient use of the interview time. Lead CPHP staff have been informed of the program evaluation effort in national CPHP Program meetings and conference calls. Thus, CPHP staff are anticipating the information collection process and, given that we are collecting information from active grantees, it is reasonable to expect that 100% of the CPHPs will participate.

With respect to the CPHP Customer/Partner Survey Instrument, the introduction to the survey will contain the purpose of the information collection and directions for accessing the web-based survey. This introduction will emphasize the importance of input from CPHP organizational partners and customers. Each correspondence with the respondents will be personalized to improve response and ensure that the correct respondent is the recipient of the survey. If a respondent chooses the option to receive a paper survey, a packet will be mailed to the respondent with information identical to that received by respondents opting for the web-based survey as well as instructions for returning the completed survey to JSI.

As previously described, the target respondents for the survey instrument will be selected from a sample of organizational representatives thought to have a high likelihood of knowing of or having had contact with the CPHP Program. Approximately 1 week prior to disseminating the survey each sample respondent will be contacted by email or telephone to confirm his or her contact information and to provide advance notice of the survey. The name, job title, and contact information of the respondent will be recorded in a sample tracking database (maintained separately from survey responses) which will be used for all subsequent follow-up correspondence.

A 3-step process will be used to maximize the response rate to the survey:

1. Telephone call establishing initial contact information (Attachment F)

2. Email follow-up (Attachment G)

3. Telephone follow-up (Attachment H)

The availability of a web-based format is also expected to increase the response rate because it will ease administration of the survey.

With respect to the CPHP Customer/Partner Follow-Up Interview Instrument, the potential respondents are a subset of the survey sample. As part of the survey administration process, potential respondents may volunteer to be contacted for a brief follow-up interview to collect additional qualitative information on their experiences, perceptions and opinions.

4. Test of Procedures or Methods to be Undertaken

In order to obtain an estimate of the burden associated with the information collection instruments, an Evaluation Work Group was convened with representation from four CPHPs and the Association of Schools of Public Health. This work group assessed the efficacy of elements selected for inclusion in the instruments and provided feedback on the length, content and structure of the instruments, as well as the overall study design.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The plans for statistical analyses for this study were developed under a contract with JSI. The JSI team also participated with CDC staff in the design of the data collection and will conduct the data collection and analysis. Contact information for the JSI team is provided below:

Lea Ayers LaFave, PhD, RN

Senior Project Director

Community Health Institute/JSI

501 South Street

Bow, New Hampshire, 03304

603.573.3335

Email: lea_ayers-lafave@jsi.com

Jonathan Stewart, MA, MHA

Director

Community Health Institute/JSI

501 South Street, Bow, NH 03304

tel: 603.573.3300 fax: 3301

Email: jstewart@jsi.com

Rachel Kohn, MSW, MPH

Senior Consultant

Community Health Institute/JSI

501 South Street

Bow, NH 03304

Phone: 603-573-3321

Fax: 603-573-3301

Email: Rkohn@jsi.com

4