Supporting Statement 16600057_Part B_071414

June 24, 2014

Supporting Statement for

Paperwork Reduction Act Submissions

OMB Control Number: 1660-0057

Title: Chemical Stockpile Emergency Preparedness Program (CSEPP) Evaluation and Customer Satisfaction Survey

Form Number(s): Blue Grass EPZ Residential Survey/ FEMA Form 008-0-7, Pueblo EPZ Residential Survey/ FEMA Form 008-0-9, Pueblo EPZ Residential Survey/ FEMA Form 008-0-3INT, Pueblo City Residential Survey/ FEMA Form 008-0-4, and Pueblo EPZ Business Survey/ FEMA Form 008-0-5

B. Collections of Information Employing Statistical Methods.



When Item 17 on the Form OMB 83-I is checked “Yes”, the following documentation should be included in the Supporting Statement to the extent it applies to the methods proposed:

1. Describe (including numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

Site Surveys: (Residents)

There are approximately 244,586 persons 18 years old or older residing inside an Emergency Planning Zone (EPZ) at the Blue Grass Army Depot, and the Pueblo Chemical Depot, a CSEPP-designated geographic region located in the immediate or surrounding areas of these sites. EPZs are further subdivided into an Immediate Response Zone (IRZ) and a Protective Action Zone (PAZ), each carrying specific levels of protective actions in the event of a disaster.

Table 1 shows the collection’s universe, sample size, margin of error, and sampling techniques for each site.

Table 1: Universe Populations, Sample Size, Margin of Error, and Sampling Technique (Site Surveys)

CSEPP Site Name	Total Sample or Name of Strata	Geographic Region	Universe: Population over 18 who are inside the geographic region	Margin of Error	Sample Size	Sampling Technique
Blue Grass Army Depot	Total Sample	Depot, IRZ, PAZ	150,765	3.4%	826	Stratify by County
Pueblo EPZ Residential	Total Sample	Depot, IRZ, PAZ	6,128	3.4%	754	Stratify by IRZ/PAZ
Pueblo City	Total Sample	Pueblo City	87,468	5.0%	382	Simple Random Sampling
Pueblo EPZ Business	Total Sample	Depot, IRZ, PAZ	225	8.3%	86	Simple Random Sampling
TOTAL			244,586		2,048

Response Rates:

Site Surveys: Response rates are calculated using final disposition codes and response rate formulas published by the American Association for Public Opinion Research (AAPOR). ¹ The response rates on the residential surveys for the interested sites on their last survey have been listed below in the table.

Response Rates across CSEPP sites

	Pueblo EPZ Resident	Pueblo City	Pueblo EPZ Business2	Blue Grass EPZ
Response rate (RR1)	10.6%	13.0%	12.0%	17.7%
Refusal rate (REF3)	35.9%	19.7%	11.5%	13.4%
Non-Contact (e.g., answering machine, call back, and language barrier)	25.9%	42.1%	34.2%	16.6%
Unknown Eligibility (e.g., no answer, always busy)	26.8%	22.7%	40.4%	50.4%
Partially completed questionnaire	0.8%	2.5%	1.9%	1.9%

The survey campaign has been conducted for several years. Because of the repetitive nature of this survey research, there are some communities where nearly every resident has participated in the survey at one time or another. This can have an adverse effect on participation in subsequent surveys. So aside from considerable rates of non-contact and unknown eligibility common in Random Digit Dialing (RDD), the survey effort is also experiencing increasing refusal rates at some sites.

Due to the availability of new communication technologies, such as caller ID, answering machines, and mobile phones, households are often hard to contact by wire line telephone. Preliminary results, from the National Health Interview Survey (NHIS), which were conducted during January-June 2008, indicate that more than 35.8% of American homes had only wireless telephones. Moreover, portability of telephone numbers, which allows residents to keep their phone number as they relocate to another provider, wire line or wireless, in the same general region, makes it challenging to reach residents within the EPZ boundary. In an effort to reach such segments of the population, the Pueblo site is also considering the use of web-based surveys along with telephone surveys. To avoid the possibility of under representing the subgroup of the population with only cell phones (e.g. 18 – 29 age groups), Pueblo would like to conduct a pilot test using web-based surveys in order to study the effect of these subgroups on the response rate. The pilot survey will be conducted using the same questionnaire. There are several limitations associated with the web-based surveys (e.g. limited internet usage in low-income families and among senior citizens) when it is implemented for surveying the general population. Hence the site would not completely switch to a new mode and treat these web-based surveys as complementary to the traditional telephone survey. The site would also like to investigate the effect of underrepresented data on the accuracy (or bias, if any) of the survey results.

When conducting the CSEPP telephone surveys, we make the assumption that non-response is independent of answers to questions on the questionnaires. Essentially we assume non-response is missing at random. We have checked this assumption by comparing the demographic percentages in the survey against U.S. Census data and past survey results. However, this decline in response rate requires attention and needs to be addressed appropriately. The following techniques will be adopted in conducting future surveys to increase the response rate by a certain amount:

1. Send a Pre-Notification Letter –This method has previously been adopted by some sites, such as Umatilla, and a difference in the response pattern is observed for Umatilla compared to the other sites. All the sites are recommended to follow this as a best practice.

2. Number of Call Attempts – CR Dynamics’ Computer Assisted Telephone Interviewing (CATI) system allows for the scheduling of callbacks either automatically or for a time determined by the interviewer’s contacts with a survey respondent. Telephone numbers are generally attempted up to three times. This number can be increased as the refusal rates have increased.

1. Describe the procedures for the collection of information including:

   1. Statistical methodology for stratification and sample selection,

There are now two remaining CSEPP sites in the United States. These two sites (CO and KY) have conducted a residential survey within the past 3 years. At this time, Blue Grass, KY, and Pueblo, CO are interested in conducting future surveys.

Survey Respondents:

• Residents (EPZ Surveys): Total sample consists of 2,048 residents meeting the following selective criteria:

• Live inside the Emergency Planning Zone (EPZ) “footprint” of a participating CSEPP site. Emergency Planning Zones are further subdivided into an Immediate Response Zone (IPZ) and a Protective Action Zone (PAZ).

• 18 years of age or older

• Residents (Pueblo City Survey): The Pueblo CSEPP site has shown interest in conducting a survey in Pueblo City, which is right outside of the emergency planning zones (EPZ) of the site. Local emergency managers are concerned with the possibility that Pueblo City residents either work inside the EPZ or may enter the EPZ to assist family or friends during a chemical emergency and may be exposed to lethal chemical agents. Previous to the first submission of the OMB paperwork a pilot test in Pueblo City had been performed. Pueblo City residents will be sampled with a margin of error of ^±5 percentage points, the FEMA standard. Pueblo has proposed a slightly larger margin of error than that which is used in the EPZ residential surveys (^±3.4 percentage points) because they have determined that more precision in the estimated survey proportions is not needed for this survey. With a ^±5 percentage point margin of error the sample size would be 382 as shown in Table 7.

• Businesses (Pueblo EPZ Survey): Generally in the business public surveys there have been the following restrictions:

• Each survey respondent must be employed by the said business.

• The business must be located inside the EPZ footprint of the CSEPP site.

Sampling Design and Methodology

Blue Grass and Pueblo site have elected to use Simple Random Sampling (SRS). Statisticians at IEM help CSEPP sites choose a sampling design that will most effectively represent their population and accomplish the specific goals of their survey.

For zones, States, and/or counties with relatively small populations within the geographical region defined to be sampled, CSEPP sites have elected to use the technique of over-sampling to collect a large enough sub-sample to make valid statistical comparisons with other sub-groups.³ When over-sampling is employed, the results of the over-sampled regions are weighted back to match their respective population proportion.

Sample sizes for each site have been calculated with standard statistical formulas.

Sampling Frame: The sample frame includes only those residents located inside the EPZ footprint for each CSEPP site. RDD is used to include both listed and unlisted numbers reducing the bias of traditional telephone directory sampling. To ensure that residents outside of the EPZ are not included in the sampling frame, Genesys purges the listed phone numbers geographically. This means that for the portion of the RDD sample representing listed households or businesses, addresses will be matched against site geographic boundaries to delete sample households located outside the EPZ. If a sub-region of the sample area has a very high incidence rate of residents living outside of the EPZ footprint, listed households will be sampled from that sub-region to ensure that residents will not be called outside the EPZ. For the Pueblo business surveys, the lists of telephone numbers for the businesses are provided by the site Public Information Officer (PIO).

The businesses are identified by the Pueblo County GIS system. They match the company names with their Computer Aided Dispatch (CAD) system which has the list of telephone numbers provided by the telephone companies. The two sources are used to create the list.

Length of Field Period

The length of time a CSEPP telephone survey remains active in the field is dependent on a number of factors including the sample design, sample size, and preference of the CSEPP site. Some CSEPP sites prefer the field period to be as short as possible in order to collect information directly after an outreach campaign. While some surveys have been fielded in as little as four days and others as long as eleven days, seven days is the average number of days a CSEPP telephone survey remains in the field.

2. Estimation procedure

For a general SRS, the sample mean or proportion is used to estimate the population mean or proportion. If it becomes apparent that a certain sub-region or sub-regions of the SRS are either over-represented or underrepresented in the survey results, survey weights may be applied so that the sample population proportions are consistent with the true population proportions. In these cases, a weighted sample mean or proportion is used to estimate the population mean or proportion.

For stratified samples, a sampling weight as defined in current statistical literature is applied to the sample.⁴ The weighted sample mean or proportion is used to estimate the population mean or proportion.

If a sampling design other than an SRS or stratified sample is deemed to be appropriate for a certain CSEPP site, similar valid statistical procedures will be incorporated to estimate the population means and proportions.

3. Degree of accuracy needed for the purpose described in the justification

CSEPP site-specific residential surveys employ random sample selection based on a 95 percent confidence level and a margin of error of ^±3.4 percentage points which exceeds the FEMA standard of ^± 5%.

The National Public Affairs Integrated Process Team (IPT) has been agreeable to CSEPP sites electing to use sample designs such as a SRS, stratified sample, cluster sampling, or other more complex designs. However, participating sites have elected to use either an SRS or stratified sample.

The region surrounding each CSEPP site is divided into emergency planning zones (EPZs). Emergency planning zones are divided into the Immediate Response Zone (IRZ), which is the area closest to the Army installation, and the Protective Action Zone (PAZ), which is the area surrounding the IRZ. Often CSEPP sites are interested in summarizing and making comparisons of the survey results by IRZ/PAZ, State, and/or county. Some of these regions of interest have very small populations, resulting in few if any completed surveys from that region when employing a SRS. In order to make comparisons among regions or subgroups of the survey results, the Public Affairs IPT has required a minimum of 30 survey respondents per region/subgroup.

Quality Assurance Plan:

To ensure that the highest quality of work is performed, a quality assurance plan is implemented in every survey process. The plan defines the relationships between IEM and the subcontractors and details the quality assurance activities used throughout the CSEPP survey effort.

Over the course of the survey work, CSEPP sites have chosen to modify and/or remove some of the core questions from their site questionnaire in order to produce a more customized survey instrument.

Prior independent research work on CSEPP sites conducted by the University of Arizona⁵ and by IEM has been reviewed in order to replicate successes and avoid shortcomings. Before each survey is conducted, IEM and the site’s Public Affairs Officer (PAO) carefully examine the questionnaire, review previous survey results, review the sites’ outreach campaigns and objectives, and make modifications where necessary. Questions found to yield inaccurate and/or unreliable results are eliminated and/or modified.

Core survey questions are reviewed for accuracy and reliability within each site and across all participating sites and labeled as either “Optional” or “Essential”. CSEPP sites are also able to add site-specific questions to the questionnaire. These questions must go through a review process before they are incorporated into the survey. Site-specific questions are provided to IEM project personnel who review them for validity, reliability, clarity of content, and question bias. Questions are modified as necessary, and final versions of the site-specific questions are approved by the appropriate site and then incorporated into the survey.

4. Unusual problems requiring specialized sampling procedures, and

For surveys with particularly low response rates and a substantial suspicion of non-response bias, it may be necessary to collect an additional sub-sample of completed surveys from non-respondents in order to confirm if non-response bias is present in the sample and make adjustments if appropriate.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Program and survey objectives require annual collections.

2. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Site Surveys: Several measures, listed below, have been taken in order to maximize response rates for the telephone surveys:

• At the beginning of the telephone call, respondents are told about the survey’s purpose, estimated response time, and who is sponsoring and conducting the survey.

• The length of the survey phone interviews are kept to 15 minutes or less.

• The survey interviewers are thoroughly trained.

• The data collection company, CR Dynamics, is able to partially complete phone interviews and then call-back and finish the interview at a different time.

• The telephone database provider company, Genesys, will use the most comprehensive screening tool available to increase the productivity of the sample numbers.

Control for Non-Response Bias: To avoid the possibility of under-representing a certain subgroup of the population, the CSEPP sites have chosen to use a stratified sample and/or over-sample a certain subgroup of individuals. If there is substantial suspicion of a non-response bias in the survey results, a sub-sample of the non-respondents will be collected and analyzed. If a nonresponse bias is found in the analysis, we would use weights to adjust the data for nonresponse. From the analysis of the respondents and non-repondents, we would determine the probability of responding to the survey for each person, which we will call Ai for person i. We would then calculate the probability that person i is measured in the survey, P (unit i selected in the sample and responds) = AiBi, where Bi is the probability that person i will be selected in the sample. The final weight for each person i will be:
.⁶

3. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Pilot Test

At the beginning of each site survey collection period, a pilot test was conducted to discover any potential problems with the survey instrument or process. To ensure that the call takers have received adequate training on the survey script, a minimum of one complete call per call-taker is conducted on the first day. For quality assurance purposes, CR Dynamics provides IEM the ability to monitor the live calls. Data from the first night is reviewed by IEM and improvements are made to the survey process as deemed necessary.

4. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Innovative Emergency Management, Inc.

Jack Long, Project Manager
2014 Tollgate Road, Suite 206
Bel Air, MD 21015
410-569-8191
410-569-9553 (fax)
jack.long@iem.com

CR Dynamics & Associates, Inc.

Patty Ramos, VP of Operations
7 East Redwood Street, 6th Floor
Baltimore, MD 21202
410-347-5600, ext 203
410-347-5603
patty@crdynamics.com

James Harris, MIS Director
7 East Redwood Street, 6th Floor
Baltimore, MD 21202
410.347.5600, ext 204
410.347.5603 (fax)
james@crdynamics.com

Genesys Sampling

Ashley Hyon
Marketing Systems Group
565 Virginia Drive
Fort Washington, PA 19034
215-653-7100
(215) 653-7114 (fax)
ahyon@m-s-g.com

1 http://www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/ standard_definitions

2 The 2013 Pueblo EPZ Business survey contacted businesses from a sampling frame which only included contact at four businesses within the EPZ boundaries. The response rates reflect the response rates for all businesses contacted regardless of their location.

3 In order to make valid comparisons between sub-groups, the PA IPT has required that each sub-group have a minimum of 30 survey respondents.

4 Lohr, Sharon L. (1999) Sampling Design and Analysis. Brooks/Cole Publishing Company, p. 103.

5 Williams, B., et al. Characteristics of Perceived Emergency Preparedness among Residents Living near the U. S. Army’s Chemical Weapons Stockpile Sites: A hierarchical linear model, 2000.

6 Lohr, Sharon L. (1999) Sampling Design and Analysis. Brooks/Cole Publishing Company, p. 265-266.

1