Download:
pdf |
pdfEvaluating Effectiveness of
the CSEPP Survey Tool
October 1, 2007
Prepared For
Mr. Steven Horwitz
DHS/FEMA Headquarters
500 C Street, SW
Washington, DC 20472
Prepared Under
Task Order Number: HSFEEM-07-J-0002
This document is for reference only. Readers should not construe this document
as representing official FEMA policy or regulations.
IEM/TEC07-059
This page intentionally left blank.
Evaluating Effectiveness of the CSEPP Survey Tool
Executive Summary
The public outreach survey tool is currently being used to evaluate the public
awareness of protective actions at participating Chemical Stockpile Emergency
Preparedness Program (CSEPP) sites. The Federal Emergency Management
Agency (FEMA) funded the development of the survey methodology and
questionnaire in a collaborative project with the CSEPP Public Affairs Integrated
Process Team (PA IPT) and IEM in 2000. IEM has provided technical and
administrative support to the PA IPT. Working with the PA IPT, IEM developed a
public awareness survey methodology, assisted in development of the survey
questions, and provided data analysis and report writing. In January 2007, IEM
was tasked to perform an assessment of the survey initiative in order to assess the
continuing need for the ongoing measurement of the public outreach efforts.
The success of the survey initiative is measured by how well it helps the CSEPP
PA IPT to identify and implement efforts designed to fulfill their vision of “a
public that can and will act appropriately upon notification of an emergency at a
chemical installation.” The goal of the overall survey concept was to design and
implement a public survey strategy to support the development of public outreach
and education efforts that will improve the emergency preparedness of citizens
living in the Immediate Response Zones (IRZ) and Protective Action Zones
(PAZ) surrounding participating CSEPP sites.
Aim of the Assessment
As part of its continuing efforts to better serve the CSEPP sites, IEM has
performed a detailed investigation of previous survey efforts, at the site level, in
order to assess the continuing need for ongoing measurement, analysis, and
implementation of best practices observed from previous public survey efforts.
The purpose of this evaluation is to examine the effectiveness of the survey tool
and to make recommendations to the program for enhancements and
modifications to the survey tool to support the changing landscape of CSEPP at
the various sites.
Outcome
The changes in public awareness in participating CSEPP communities over the
course of the survey work were analyzed to determine areas of notable
improvement and areas that still require additional efforts. Great improvements
have been made in residents’ knowledge of shelter-in-place as well as residents’
overall confidence in their ability to protect themselves in the unlikely event of a
chemical emergency. Because of the persistent efforts and targeted outreach, the
levels of trust and control of the residents in CSEPP communities have increased
over the past few years.
However, many CSEPP communities are still struggling with low levels of
awareness when it comes to family and school emergency plans. A large
i
Evaluating Effectiveness of the CSEPP Survey Tool
proportion of parents indicate they are confident their child is safe at school, yet
they are unaware of their child’s school emergency plan. Emergency plans are an
important area for future outreach. Based on the survey data, it is suggested that
Public Information Officers (PIOs) continue to focus on school preparedness and
get parents involved. They should also encourage all residents in the community
to develop an emergency plan. In the future, CSEPP sites should continue to
strive to reach as many residents as possible with valuable emergency
preparedness information. The IEM Trust and Control model can be used
effectively for public outreach decision-making by comparing it with the other
demographic variables. It is important to continue aggressive outreach efforts to
ensure that the levels of trust and control of residents do not decrease and
continue to improve where they can.
It was with the help of the survey tool that the improvements could be tracked
over time. Results from the survey data, along with IEM’s Trust and Control
model, assisted in identifying the focus areas for future outreach activities.
Suggestions for Improvement
The survey tool in general has been identified as the only means of assessing the
effectiveness of the public outreach campaigns apart from qualitative feedback
from activist activities and media articles. The CSEPP communities use the
survey results in their public outreach decisions. However, based on past
experiences, there are some areas that have been identified by the sites and IEM
that could be improved. One key issue is to ensure a short but effective survey. To
shorten the length of future surveys, only survey questions and response options
that work toward current program and site objectives should be included in the
questionnaire. Secondly, to continue to address the current needs of participating
sites, recommendations for future outreach can strive to take a different approach,
which will provide a comprehensive, big-picture evaluation of the public outreach
program for each site. Specific suggestions are described in the Future Survey
Efforts section of this report.
ii
Evaluating Effectiveness of the CSEPP Survey Tool
Table of Contents
Introduction ..............................................................................................1
Background ..................................................................................................... 1
Study Aim and Objectives.............................................................................. 1
Methodology and Approach........................................................................... 2
Trends in Survey Data .............................................................................3
Awareness ........................................................................................................ 4
Protective Actions ........................................................................................... 4
Family Emergency Plans................................................................................ 6
School Preparedness ....................................................................................... 7
CSEPP Information ........................................................................................ 9
Trust and Control Levels ............................................................................. 11
Level of Trust.......................................................................................................................... 11
Level of Control...................................................................................................................... 12
IEM’s Trust and Control Model .............................................................12
Application of the Trust and Control Model.............................................. 13
Focus Issues for Sites ...........................................................................16
Future Survey Efforts.............................................................................19
Data Collection .............................................................................................. 19
Response Rates ....................................................................................................................... 19
Questionnaire Development......................................................................... 20
Updates to Questionnaire to Address Appropriate Site Issues ................................................ 20
Short and Effective Questionnaire .......................................................................................... 20
Report Writing and Recommendations ...................................................... 22
Deliverables ............................................................................................................................ 22
Recommendations................................................................................................................... 22
Conclusion..............................................................................................23
Points of Contact ...................................................................................24
iii
Evaluating Effectiveness of the CSEPP Survey Tool
This page intentionally left blank.
iv
Evaluating Effectiveness of the Survey Tool
Introduction
Background
In 2000, IEM began assisting CSEPP in surveying the public to evaluate their
awareness of chemical emergency notification methods, sources of information
during an emergency if one were to take place, and willingness and ability to
follow recommended protective actions at participating CSEPP sites in the United
States. FEMA funded the development of the survey methodology and
questionnaire in a collaborative project with the CSEPP Public Affairs Integrated
Process Team (PA IPT) and IEM. IEM provides technical and administrative
support to the PA IPT.
This public awareness survey effort is an outgrowth of the CSEPP PA IPT’s
mission to identify and implement efforts designed to fulfill their vision of “a
public that can and will act appropriately upon notification of an emergency at a
chemical installation.” The information gathered in the surveys is used to identify
the presence of knowledge gaps so that those responsible for public outreach can
identify successful and unsuccessful outreach methods, as well as to evaluate
awareness trends over time. In response to these needs, IEM developed the Trust
and Control Model, which was designed to identify and track these changing
knowledge gaps. The Trust and Control Model used in the survey analyses has
proven to be a helpful and adaptable tool in identifying and shaping behavioral
components in the CSEPP communities.
In January 2007, IEM was asked to assess the survey tool. The purpose of this
assessment is to examine the survey’s effectiveness and make recommendations
for enhancements and modifications that reflect the changing landscape of CSEPP
at the various sites.
Study Aim and Objectives
The objectives of the assessment are the following:
Review the importance of the survey tool to evaluate awareness and
preparedness trends at the participating CSEPP sites.
Assess the effectiveness and efficiency of the survey tool in providing the
clients with the information needed to help in their public outreach decisions.
Review the usage of survey data in strengthening targeted public outreach
planning by the site officials.
Provide recommendations to CSEPP for future survey techniques and their
implementation strategy.
Goal: Identify factors to suggest improvement in the efficiency and effectiveness
of the survey tool used by the participating CSEPP sites to evaluate the public
awareness of protective action in the unlikely event of a chemical emergency.
1
Evaluating Effectiveness of the Survey Tool
Methodology and Approach
Survey work has been carried out at the following CSEPP sites: Anniston,
Deseret, Newport, Pine Bluff, Pueblo, and Umatilla. Outreach efforts have led to
significant improvements in the level of awareness since the first survey.
Respondents’ knowledge of how to protect themselves in the event of a chemical
emergency has increased over the years. In this analysis, IEM highlights these
improvements based on past survey data and perform a detailed site level
assessment that delineates changes in the survey data over the years.
Our approach focuses on providing decision-makers with information that details
the role of public awareness surveys in CSEPP communities. Analyses were
performed to estimate the impacts of the various outreach strategies that have
been implemented at the sites. In this approach, quantitative and qualitative
analyses were carried out for (a) determining specific trends in the past survey
responses at the participating sites; (b) analyzing the recommendations made for
future public outreach efforts; and (c) establishing the relationship between the
previous outreach strategies and the resulting areas of improvement.
In addition to IEM’s survey data analyses, the sites were also given an
opportunity to suggest recommendations for ongoing/previous survey techniques
based on their prior experiences. To this end, an online tool was designed for the
participating CSEPP sites seeking their input on the effectiveness of the current
survey tool.
In the subsequent sections of this assessment report, the following topics are
addressed and their findings are summarized:
2
Trends in Survey Data
IEM’s Trust and Control Model
Focus Issues for Sites
Future Survey Efforts
Conclusion
Evaluating Effectiveness of the Survey Tool
Tr e n d s i n S u r v e y D a ta
Trends in the survey data highlight the progress made through public outreach
over the course of several years. In order to observe trends and note areas of
improvement as well as areas that still need improvement, results from the first
survey are compared to the most recent survey results for each site. The
methodology to track improvement addresses the following questions:
1. Where were we before? (looking at the baseline survey data for the given site)
2. Where are we now? (comparing the current situation at the participating sites
with the baseline data)
3. How did we get here? (considering the recommended outreach strategies)
4. What are the challenges for future outreach efforts? (suggesting areas that
could be targeted for future outreach based on survey findings)
5. How can the survey tool/report be modified to help focus these challenges?
The survey questions can be divided into groups based on general areas of
concern such as awareness of chemical weapons, specific knowledge of protective
actions, and so on as presented in Table 1. For each participating site,
improvements have been tracked over the course of the surveys and have been
reported under each subsection in this report. Equivalent comparisons are not
provided for Newport because the two surveys for the site were conducted
through different mediums. More specifically, the most recent 2005 mail survey
cannot be compared to the baseline 2002 telephone survey in the same fashion as
the other sites. However, results from Newport are noted where applicable.
Table 1: Focus Areas
CSEPP Emergency Preparedness
Focus Areas
Awareness
Protective Actions
Evacuation
Shelter-in-place
Family Emergency Plans
School Preparedness
CSEPP information
Levels of Trust and Control
3
Evaluating Effectiveness of the Survey Tool
Awareness
Throughout the surveys, the proportion of respondents aware of the chemical
agents being stored at each CSEPP site has remained high, usually near 90%. It is
encouraging to note from the survey results that the awareness levels among the
sites have remained relatively constant over the specified period.
100%
95%
92%
91% 92%
85%
89%
94% 92%
96% 95%
2003 - 2006
Pueblo
2002 - 2006
Umatilla
80%
60%
40%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
Figure 1: Awareness of Chemical Agents
Follow-Up Actions
It appears that the initial goal of CSEPP public awareness has been met. Due to
the high level of awareness of the chemical agents, there has been limited room
for improvement in this area. The survey tool can be used as an instrument to
gauge public knowledge from time-to-time to help sites maintain these high
awareness levels.
Protective Actions
Significant improvements have been made in residents’ knowledge of shelter-inplace in the event of a chemical emergency, which contributes to a greater number
of people who are confident in their ability to shelter-in-place. As illustrated in
Figure 2, there have been increases up to 36 percentage points over three years for
Pueblo.
4
Evaluating Effectiveness of the Survey Tool
100%
80%
69%
72%
68%
66%
61%
60%
63%
57%
53%
40%
29%
30%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
2003 - 2006
Pueblo
2000 - 2006
Umatilla
Figure 2: Confident in Ability to Shelter in Place
In 2002, 29% of the Pine Bluff EPZ residents surveyed indicated they were
confident they could shelter in place properly. Because survey results indicated
that the awareness level in this community was relatively high, it was
recommended that outreach in the community focus on increasing the protective
action knowledge of the residents. Following subsequent surveys, residents’
knowledge of shelter-in-place increased to 57% in 2007. This is an increase of 28
percentage points over a period of five years.
The Pueblo site also experienced noticeable progress in this area. The 2003
survey report recommended that the community define and target knowledge
deficiencies specific to sheltering-in-place. The report also noted that outreach
should emphasize key steps to sheltering-in-place and employ a mnemonic
device, if possible, to help people remember the steps. By 2005, the percent of
respondents who were confident they could shelter-in-place increased from 43%
to 61%. At that time, it was recommended that shelter-in-place outreach continue
with an emphasis on targeting IRZ residents.
Umatilla and Anniston started out with relatively high knowledge levels about
shelter-in-place. A pilot public outreach program was already developed for
Umatilla in 2000, which included a mass media campaign. IEM started
conducting telephone surveys for the Anniston site in 2004. Prior to this, surveys
were conducted by the advertisement agency Benton and Newton with proper
outreach strategies already in place.
5
Evaluating Effectiveness of the Survey Tool
Follow-Up Actions
Although the number of people who say they would take steps to effectively
shelter-in-place is relatively high, only a small percentage of the respondents at all
the sites actually indicated they would take all the basic shelter-in-place steps.
These percentages are as low as 1% for some of the sites. It is important that
people understand the proper basic steps for shelter-in-place in the event of an
emergency. The low percentages indicated by the surveys are in need of attention
for focused public outreach efforts. To address these gaps, the survey tool can be
modified to ask more specific questions about shelter-in-place steps.
Family Emergency Plans
The proportion of respondents in each site who have a family emergency plan for
each site is shown in Figure 3. Some sites have made significant progress, but
overall numbers are still low. Results from the most recent survey in each area
indicate that less than half of respondents have a family emergency plan.
100%
80%
60%
42%
49%
46%
36% 37%
40%
33%
37%
20%
20%
0%
2004 - 2007
Anniston
2005 - 2007 Pine
Bluff
2003 - 2006
Pueblo
2000 - 2006
Umatilla
Figure 3: Family Emergency Plan
In 2003, it was recommended that the Pueblo site develop a family emergency
plan initiative for their outreach program. The CSEPP community responded by
providing information on family emergency planning in their annual emergency
preparedness calendar, in specific issues of the CSEPP Update (a quarterly
newsletter), and on their Web site. By 2006, the proportion of respondents with a
family emergency plan increased from 20% to 33%. Also, improvement at
Newport can be noted, with an increase from 17% in 2002 (telephone) to 31% in
2005 (mail).
6
Evaluating Effectiveness of the Survey Tool
Follow-Up Actions
The survey result demonstrates a direct correlation between having a family
emergency plan and higher confidence in all other areas of disaster readiness.
Family emergency preparedness and planning are addressed in current outreach
materials. Based on the survey findings, it is recommended that outreach
education continue to include information on the basic components of a family
emergency plan, with an emphasis on the components that need the most
improvement. For future survey efforts, IEM proposes to modify the questions on
family emergency plans depending on the CSEPP site area. This would help us
identify the gap between the numbers of people who actually have a shelter-inplace kit in their possession and those who should have one.
School Preparedness
Trends in parental knowledge of school emergency plans are inconsistent across
sites, with some sites that show improvement and others that do not. According to
the most recent surveys at each site, only 37–52% of parents are familiar with
their child’s school plan as displayed in Figure 4.
100%
80%
60%
53%
59%
47%
52%
42%
46%
37%
40%
29%
20%
0%
2004 - 2007
Anniston
2003 - 2005
Deseret
2005 - 2007
Pine Bluff
2004 - 2006
Pueblo
Figure 4: Parents Familiar with Child’s School Plan
Deseret has made notable improvements in awareness of school emergency plans.
In 2003, 42% of parents surveyed were aware of their child’s school emergency
plan. The 2003 survey report recommended that the site verify each school in the
EPZ had a detailed chemical emergency plan and develop a programmatic
outreach plan for the community. Suggestions for the outreach plan included
linking CSEPP to an all-hazards approach and providing coloring books, shelterin-place materials, and family awareness activities throughout the year. As a
result, school plan awareness rose to 52% by 2006.
7
Evaluating Effectiveness of the Survey Tool
Figure 5 shows improvement in parents’ sense of school safety over the course of
the CSEPP surveys. Newport shows an increase as well, from 53% in 2002
(telephone) to 64% in 2005 (mail). The most recent survey results for each site
show relatively large proportions of parents that feel their children are safe at
school. These parents would likely cooperate with school plans in the event of an
emergency.
100%
83%
80%
70%
84% 84%
82%
72%
70%
60%
45%
40%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
2004 - 2006
Pueblo
Figure 5: Parents’ Confidence in Child’s Safety at School
A common trend among parents in CSEPP sites is that even though a large
proportion of parents believe their children will be safe at school in the event of a
chemical emergency, they are still likely to pick up their children from school.
This trend can be observed in Figure 6. This could be indicative of the fact that
parents are not familiar with their child’s school emergency plan and the fact that
in an attempt to pick up their child during a chemical emergency they could
endanger themselves, their child, and others. However, from the verbatim
responses recorded for this question, it appears that this action is driven at large
by parental instinct.
8
Evaluating Effectiveness of the Survey Tool
100%
79%
73%
80%
60%
80%
75%
76%74%
59%61%
56%
41%
40%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
2003 - 2006
Pueblo
2000-2006
Umatilla
Figure 6: Parents’ Likelihood of Getting their Child from School in
the Event of a Chemical Emergency
Follow-Up Actions
Although most parents are confident that the school can take care of their children
in a chemical emergency, there are still parents who plan to pick up their children
right away. Education in the schools must coincide with additional general public
information to bridge this gap. The survey tool can be used as an assessment tool
to track effectiveness of the outreach messages. Future recommendations from
IEM will look into strategies that would specifically target this behavior and
“parental protection” emotion.
CSEPP Information
In order to increase public awareness and preparedness in CSEPP communities, a
top priority of public outreach has been to increase the number of people who
receive CSEPP information. Figure 7 shows the percentage of respondents who
have received CSEPP information for each site. Although progress has been made
at all sites, some sites still exhibit a need for improvement in this area.
9
Evaluating Effectiveness of the Survey Tool
100%
93% 94%
85%
84%
76%
80%
69%
75%
56%
60%
40%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
2003 - 2006
Pueblo
Figure 7: Respondents Who Have Received CSEPP Information
Notable progress in the number of residents receiving CSEPP information was
made at Deseret and Pueblo by targeting new residents and increasing the amount
of information available on the Internet. A suggestion for targeting new residents
was distributing “newcomer” packets that included preparedness information to
new residents. Overall, the proportion of residents who have received CSEPP
information rose from 76% to 84% at Deseret and from 75% to 85% at Pueblo.
In 2002, the Pine Bluff survey report recommended that Pine Bluff conduct a
detailed review of the area’s media to assist the public affairs personnel in
selecting the best medium for outreach. It was also recommended that outreach
target depot employees and first responders. The last survey conducted at this site
indicated that 69% of residents have received CSEPP information.
Follow-Up Actions
There is no doubt that all the means by which information is disseminated across
the communities appears to be working. Significant changes have been observed
in the number of people receiving CSEPP information at most of the sites. Future
survey reports would investigate the possibility of using different sources to reach
the community. For example, a site more involved with media campaigns could
try to send out written materials in the mail. This could help in assessing the
effectiveness of the sources for disseminating CSEPP-related information.
10
Evaluating Effectiveness of the Survey Tool
Trust and Control Levels
Level of Trust
Trust in the CSEPP survey is measured in terms of how confident the public is
that they will be notified quickly in the event of a chemical accident at their local
depot. A high level of confidence indicates that the survey respondent expects that
the people charged with notifying them about a chemical accident will do so in a
timely manner, indicating a “High Trust” level. Similarly, a low level of
confidence indicates that the respondent is skeptical that appropriate actions will
be taken by the responsible authorities.
As shown in Figure 8, public outreach has helped to significantly raise trust levels
of residents in each CSEPP community. Results from the most recent survey in
each area indicate 77–85% of residents believe they will be alerted quickly in the
event of a chemical emergency. However, the mail survey for Newport in 2005
produced lower numbers compared to other sites, with only 61% of Newport
residents indicating they trust that they would be alerted quickly in the event of a
chemical emergency.
100%
85%
78%
80%
78%
80%
77%
71%
69%
80%
72%
61%
60%
40%
20%
0%
2004 - 2007
Anniston
2002 - 2005
Deseret
2002 - 2007
Pine Bluff
2003 - 2006
Pueblo
2002 - 2006
Umatilla
Figure 8: Confidence in the Ability that Public will be Notified
Quickly in the Event of an Emergency
11
Evaluating Effectiveness of the Survey Tool
Level of Control
While the CSEPP public outreach programs aim to provide residents with
instruction and knowledge of appropriate protective actions, it is important to for
residents to be confident in their ability to protect themselves and their families
(i.e., having control). The community survey measures control within a CSEPP
community by asking whether, “I am able to protect myself and my family in the
event of a chemical emergency.” Survey reports indicated that while some sites
have shown significant increases in this area, several sites have had little change.
This has been illustrated in Figure 9.
100%
80%
72%
65%
68%
77%
72%
63% 65%
60%
74% 76%
60%
40%
20%
0%
2004 - 2007
Anniston
2003 - 2005
Deseret
2005 - 2007
Pine Bluff
2004 - 2006
Pueblo
2003 - 2006
Umatilla
Figure 9: Confidence in the Ability to Protect Themselves and Their
Family in the Event of an Emergency
Follow-Up Actions
Overall, the level of trust seems to be higher than the level of control for the
respondents living in the area surrounding the CSEPP sites. The low level of
control is directly associated with the preparedness level. If the individuals are
prepared with a contingency plan, then they will feel protected and will have a
greater sense of personal control. Efforts in this direction relates to previouslyrecommended actions for increasing knowledge of protective actions.
I E M ’s Tr u s t a n d C o n t r o l M o d e l
The knowledge of each respondent’s trust level and perceived level of control has
been made useful for public outreach. This information is the backbone of IEM’s
Trust and Control Model, an integral part of survey reports since 2002. By
combining the responses of each measure into a matrix, four groups of
respondents emerge as shown in Table 2.
12
Evaluating Effectiveness of the Survey Tool
Table 2: Trust and Control Matrix
High Trust
Low Trust
High Control
Participants
Watchdogs
Low Control
Believers
Cynics
Trust and Control groups generally have the following characteristics:
Participants – Trust CSEPP/Army and feel that they have personal control in
the event of a chemical emergency
Believers – Trust CSEPP/Army but feel that they have little, if any, personal
control in the event of chemical emergency
Watchdogs – Do not trust CSEPP/Army but feel that they have some personal
control in the event of a chemical emergency
Cynics – Do not trust CSEPP/Army and feel that they have little, if any,
personal control in the event of a chemical emergency
Trust and Control groups are important in segmenting a community and
identifying knowledge gaps and effective ways of filling those gaps. IEM’s Trust
and Control Model can allow public outreach officials to use the variables
measuring trust and control along with other data collected in the survey to
predict the actions that a given respondent may take in the event of an emergency
based on the Trust and Control group to which the respondent belongs. For
example, if a respondent falls into the Believer category (with high trust and low
control), the data may support that he or she will likely not act on their own if
warned of an emergency, but will first listen for instructions from officials.
The survey tool uses the Trust and Control Model to distinguish between
behavioral traits (as opposed to physical characteristics used by common
demographics) and gives insight into the motivations and beliefs of both the
community as a whole and identified subsets within the community. In addition,
the responses in the Trust and Control groups can be compared to other data
collected in order to identify common trends that can be tracked over time to
evaluate shifts in these trends. The model has also been helpful for providing
recommendations for targeted public outreach messaging. This application also
acknowledges associated demographic variables by using them in conjunction
with the model to identify knowledge gaps.
Application of the Trust and Control Model
As part of the survey results, cross tabulations are provided for the response
variables and certain demographic variables such as age, gender, and ethnicity.
Investigating the composition of trust and control groups for different
demographics helps to determine target populations for future outreach. The most
recent cross tabulation results for trust and control groups in each CSEPP
13
Evaluating Effectiveness of the Survey Tool
community were examined to obtain a snapshot of the sites’ current standing and
determine trends and possible populations to target.
The following is an example of how the model can be used for targeted outreach:
Suppose the survey data indicates that males tend to belong to the Participants
category, while females are more often Cynics or Believers. This suggests
females tend to have a slightly lower sense of control and in some cases a lower
sense of trust. For sites that have gender differences, outreach materials,
especially those dealing with protective actions, should target the female
population to improve trust and control.
Table 3 shows the progression of the trust and control groups over time for each
CSEPP community. The proportion of Participants at each site has increased in
the last few years. Improving the number of Participants in each community is
vital, as Participants have generally been found to be the most prepared for a
chemical emergency and have the highest knowledge of protective actions when
compared to the other Trust and Control groups.
Table 3: Trust and Control Groups at the CSEPP Sites
Base-line survey
Last survey
Anniston
Participants
51%
61%
Believers
15%
17%
Watchdogs
14%
11%
Cynics
15%
11%
Deseret
Participants
58%
66%
Believers
18%
18%
Watchdogs
16%
6%
Cynics
3%
10%
Pine Bluff
14
Participants
47%
54%
Believers
14%
23%
Watchdogs
25%
11%
Cynics
6%
12%
Evaluating Effectiveness of the Survey Tool
Base-line survey
Last survey
Pueblo
Participants
47%
67%
Believers
23%
13%
Watchdogs
17%
10%
Cynics
6%
10%
Umatilla
Participants
47%
68%
Believers
24%
14%
Watchdogs
16%
9%
Cynics
8%
9%
Following the recommendations to use trust and control messages to formulate
outreach in the 2004 survey reports, Anniston and Pueblo showed notable
improvements pertaining to the trust and control of residents. Suggestions for
doing so included tying CSEPP protective actions and preparedness measures to
an all-hazards approach which may encourage a greater degree of participation
from those who did not have a sense of empowerment regarding their ability to
protect themselves. By 2005, Anniston had increased trust levels from 69% to
79% and control levels from 65% to 74%. Pueblo was able to increase the
percentage of residents who feel they are in control from 60% to 72%.
The model can be used effectively for public outreach decision-making by
comparing it with the other demographic variables. Comparatively fewer
Participants and more Cynics can be found in younger age groups as observed
from the past survey data. For example, 52% of 18–29 year olds in Anniston are
Participants compared to 58–66% of those 30 and older; 20% of Pueblo residents
ages 30–34 are classified as Cynics, compared to only 6–13% of those 35 and
older. A notable exception to this trend is Pine Bluff, where the lowest rates of
Participants are found in older age groups (60 and older). This information can be
utilized in targeting different segments of the population with different sets of
goals.
Also, surveys have shown that for Anniston, Pine Bluff, and Pueblo, smaller
proportions of Participants are reported for residents that have been in the area for
five years or less. These residents are relatively new to the area, and in many
cases have not received CSEPP information or outreach materials. In the past,
based on the recommendations in the survey reports, some sites have targeted
newer residents with newcomer packets to improve performance in this area.
The Trust and Control Model used in the survey analyses has proven to be a
helpful and adaptable tool in identifying and shaping behavioral components in
15
Evaluating Effectiveness of the Survey Tool
the CSEPP communities. It has also assisted in providing recommendations for
targeted public outreach messaging. Along with other demographic variables of
interest, this model can prove to be beneficial in future efforts to address the
concerns of individual sites.
Focus Issues for Sites
The key focus of outreach for CSEPP communities in the past has been
distributing appropriate CSEPP information to the public. This information
includes a description of CSEPP, literature on chemical hazards, and instructions
on what to do in the event of a chemical emergency. Since its inception, many
strides have been made in the program. Based on real-life emergencies, lessons
learned, information sharing, and best practices, the program has been able to
adjust accordingly to enhance response to emergencies. This is largely due to
strategic planning and coordinated response among multiple agencies, along with
proactive media campaigns focused on demographics and locality to the chemical
sites.
As the program has evolved, sites have encountered issues specific and relevant to
their individual locales. To facilitate an understanding of key issues among sites
at the present time, an online tool was designed to gather feedback on the current
survey tool. Table 4 documents the responses from the emergency management
personnel at the participating sites.
Table 4: Feedback from Sites
Question
Do you think that CSEPPrelated public outreach is
necessary in your area?
Why or why not?
How do you assess the
effectiveness of your
public outreach
campaigns?
16
Survey Responses
Yes, in order to inform new residents while re-informing current,
as to our program to increase levels of preparedness.
Yes, to educate residents as to their impact / role in
preparedness as it relates to CSEPP.
Yes, to inform the public of protective actions they can take in a
chemical emergency.
Yes. As long as there are chemical munitions in a given area,
there should be public outreach. The degree of outreach may
vary depending on the level of public awareness and
acceptance.
Yearly public awareness surveys.
Word of mouth and contacts made because of the media
campaign.
The number and types of public inquiries
Through the frequency of (or lack of) activist activities.
Through media articles and stories.
Evaluating Effectiveness of the Survey Tool
Question
Survey Responses
Yes, very much so, but we need to constantly reassess the
assessment tool as well to make sure we're measuring what we
feel counts.
Yes, we may base the focus of our public outreach on certain
areas of need as identified by the survey results.
Yes, it is one of only a few ways of measuring the effectiveness
of our public outreach, along with unsolicited public and media
inquiries.
Yes, previous year survey results can be compared to current
year survey results to measure the success of public outreach
over time.
No, there is always a need for public outreach regardless of
survey results.
Trust building
Confidence building
Family emergency preparedness kits, continued emphasis on
family planning
Would like to see the reports somehow break down the
percentage of change from the first survey to the most current.
Do the survey questions
adequately cover your
CSEPP public outreach
program?
Very useful if questions match objectives, validity
Adequate, especially since we were allowed to add our own
site-specific questions.
Which of the survey
deliverables are most
useful to you (tabulations,
cross-tabulations, final
report,
recommendations)?
Yes, all, but recommendations seem to lack a considerable
amount of creative thought
final report and recommendations are the most useful
Tabulations, cross-tabulations, final report, and the
recommendations. Especially the summary.
What information do you
find most beneficial from
the survey?
Public confidence levels
Understanding of school plans
Percentage of families with emergency plans and response kits
Protective actions. The public cannot protect itself if it doesn't
know how.
All of the off-post question results.
Idea joggers
Yes, they are helpful. The last two surveys tell us the public
knows what CSEPP is but it showed we need to give them more
specific information on protective actions.
Tried to give out more information about the proper steps to
shelter-in-place, following recommendations from the survey.
Do you use the survey
results in your public
outreach decisions? How?
What are the current
needs of your community
that could be
accomplished by the
surveys?
Are the survey report
recommendations useful?
If so, are you able to
implement them? Can you
please give us an
example?
17
Evaluating Effectiveness of the Survey Tool
Question
Survey Responses
Local political climate, what's important to my public RIGHT
NOW...
Word of mouth and contacts made through the media campaign.
Public inquiries and media coverage.
What other information
tool(s) do you use to
assess public outreach
efficiency?
No other
Word of mouth and contacts made through the media campaign.
Public inquiries and media coverage.
What changes to the
survey tool would you
recommend to help you
accomplish your goals?
Dynamic evolution of questions to match current objectives
better
More creative recommendations
Is the demographic
information (age, gender,
income, etc.) provided by
the survey helpful in
directing public outreach?
Somewhat
Yes
What other information
tool(s) do you use to
determine where to focus
public outreach?
Site responses provided above support the use of annual public awareness surveys
as a means to assess the impact and validity of CSEPP. The survey tool provides
true measurement of public awareness and knowledge of appropriate protective
actions citizens will take during an emergency. Yearly surveys allow CSEPP sites
to assess the effectiveness of ongoing outreach campaigns and note areas of
improvement. However, IEM can focus on the following two areas to improve the
survey tool and achieve the goals of the survey task. They are:
1. Questionnaire development
2. Recommendations for future outreach effort
The research team at IEM can continue to develop new tools and processes to
make improvements in these suggested areas. In an attempt to do so, we have reevaluated our process and have documented some new ideas in the Future Survey
Efforts section of this document.
18
Evaluating Effectiveness of the Survey Tool
F u t u r e S u r v e y E f f o r ts
Data Collection
Response Rates
Table 5 presents the average response rate and refusal rate across CSEPP sites for
different years. The response rates are calculated using final disposition codes and
response rate formulas published by the American Association for Public Opinion
Research (AAPOR). 1 The average response rate across CSEPP sites has come
down to 17.33% in 2007 compared to 20.65% in 2003–2004. A large portion of
this decline is due to the fact that refusal rate has increased from 13.66% to
25.25%. More effort is required to achieve desired response rates because people
are becoming less willing to respond.
Table 5: Response Rates
2003–2004
2006–2007
Response rate
20.65%
17.33%
Refusal rate
13.66%
25.25%
Non-Contact (e.g. answering
machine, callback, and language
barrier)
26.84%
23.65%
Unknown Eligibility (e.g. no
answer, always busy)
37.21%
36.23%
Partially completed questionnaire
1.63%
1.06%
Total
100%
100%
When conducting the CSEPP telephone surveys, we make the assumption that
non-response is independent of answers to questions on the questionnaires.
Essentially we assume non-response is missing at random. We have checked this
assumption by comparing the demographic percentages in the survey against U.S.
Census data and past survey results. However, this decline in response rate
requires attention and needs to be addressed appropriately. Here are some
suggested techniques that could be adopted in conducting future surveys to
increase the response rate by a certain amount:
1. Send a Pre-Notification Letter – Research indicates that sending a prenotification letter generates the highest effect in increasing the response rates.
This method has already been adopted by some sites such as Umatilla, and a
difference in the response pattern is observed for Umatilla compared to the
other sites.
1
http://www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/
standard_definitions
19
Evaluating Effectiveness of the Survey Tool
2. Number of Call Attempts – CR Dynamics’ Computer Assisted Telephone
Interviewing (CATI) system allows for the scheduling of callbacks either
automatically or for a time determined by the interviewer’s contact with a
survey respondent. Telephone numbers are generally attempted up to three
times. This number can be increased as the refusal rates have increased.
Research 2 on the optimal number of call attempts indicate that the majority of
completed interviews, refusals (initial refusals), and ineligible cases are
established by the sixth or seventh call attempt.
Questionnaire Development
Updates to Questionnaire to Address Appropriate Site Issues
Questionnaire development is the most crucial step in any survey process. The
CSEPP Public Affairs IPT developed a set of core questions that are used in
residential EPZ surveys at participating CSEPP sites. The site PIOs have been
intrumental in the questionnaire design and have assisted IEM with the design of
the site-specific questions. The inclusion of site-specific questions is to meet the
specific needs of the individual sites and help capture the breadth of information
needed to measure their program’s impact. Many sites are interested in reviewing
and modifying the questionnaire to match the current objectives of the program.
As in the past, CSEPP sites should be able to add site-specific questions to the
questionnaire. These questions must go through a review process before being
incorporated into the survey. The IEM project personnel will review the sitespecific questions for validity, reliability, clarity of content, and question bias.
Questions will be modified as necessary, and final versions of the site-specific
questions will need to be approved by CSEPP headquarters and then incorporated
into the survey.
Short and Effective Questionnaire
As demilitarization of the chemical weapons stockpile progresses and some of the
sites are nearing closeout, there has been a reduction in the availability of funds to
aid public outreach efforts at the sites. This development makes it necessary to
analyze the cost and associated benefits of the survey task as an assessment tool.
The aim would be to minimize the cost of the survey tool without losing its ability
to assess the public outreach effort efficiently. One way to achieve such cost
reduction may be to identify the critical questions from the surveys by
concentrating on specific areas of preparedness that need to be emphasized or
improved. This will shorten the length of the survey questionnaire. Depending
upon the length of new survey questionnaire, the reduction in the overall survey
cost can be estimated by comparing it with the existing survey costs. For example,
there is a savings of approximately $5,000 when comparing a 15-minute survey to
a 10-minute survey.
2
Do current methods used to improve response to telephone surveys reduce nonresponse bias? Roberta L. Sangster, Bureau of Labor
Statistics, Office of Survey Methods Research, Washington DC 20212 USA.
20
Evaluating Effectiveness of the Survey Tool
A more streamlined option is the creation of a “focused survey” questionnaire that
targets a site-specific, critical preparedness area. The question is how effective a
“focused survey” would be. The main purpose of annually conducting the surveys
is to track changes in the awareness and preparedness level over a given time
span. If, for example, a “focused survey” was created on school preparedness at a
given CSEPP site, to measure improvements, data for consecutive years are
required in school preparedness and, hence, the focused survey cannot be targeted
for only one year. Focusing on only one specific area would not provide an
overall picture of preparedness level, even if the focus area is rotated every year.
This solution would also be inefficient from a cost perspective, because a new
survey will need to be created for each focus area.
There is a need to create a balance between the length of the survey and the goal
for cost reduction. Hence, to meet the objective of CSEPP in general, it is
recommended that the sites conducting the surveys should continue asking the
important questions that will provide data relevant to preparedness level and will
facilitate the preparation or modification of public outreach materials. The aim
would be to achieve the desired objectives of CSEPP by creating and
implementing a short survey questionnaire that can be cost-effective as well as
efficient.
It is always a good practice to keep the survey objective in mind while designing
the survey questions. One of the means to increase the response rate on the
surveys is to keep the questionnaire short and effective. Care should be taken to
avoid asking questions that we think might provide us with some ”interesting
data” but do not tie up directly to the objective. For example, in the CSEPP
surveys, the demographic questions are used to identify characteristics such as age
group, gender, income group, race, time living in the vicinity of the CSEPP site
and so forth. One needs to ask the question, “Does this information really help
targeting population groups?” Further, it should be determined if there are
outreach materials available that are designed to address specific groups of the
populations based on the demographic variables of interest.
As part of the quality assurance plan of the survey process, IEM monitors live
calls on the first two nights the surveys are fielded. Team members at IEM have
observed a significant increase in the time for conducting a single survey, with the
increase in the number of options on the multiple choice questions. Call takers
find it difficult to locate the exact response option at times (e.g., when there are
more than 20 options). Reducing the number of options can also save time
conducting a single survey. Again, with the objective in mind, each option on the
response item should be weighed to see if it is an added benefit.
21
Evaluating Effectiveness of the Survey Tool
Report Writing and Recommendations
Deliverables
The survey reports have taken a predetermined format in the course of the survey
work. Based on the feedback received from the sites, it is noted that they would
be interested in all of the three reports that are provided to them at this time,
namely (a) the Tabulations Report, (b) the Cross Tabulations Report, and (c) the
Final Report with Recommendations. However, as has been done in the past, IEM
will continue to work with the sites to provide them with customized information
to meet their specific requirements.
Recommendations
In the past, there has been a focus on disseminating basic CSEPP information
(i.e., a description of CSEPP, literature on chemical hazards, and instructions on
what to do in the event of a chemical emergency).
Many strides have been made, and one of the main areas of accomplishment is
public education and outreach. Based on real-life emergencies, lessons learned,
information sharing, and best practices, the program has enhanced its response
process in the event of an emergency. This is largely due to strategic planning and
coordinated response among multiple agencies, along with proactive public
education/media campaigns focused on demographics and locality to the chemical
sites.
To gauge public perception and knowledge, as well as provide corrective action
strategy, the surveys provide public outreach recommendations designed to
measure buy-in, information retention, and protective actions. In doing so, it has
been routine to note discrepancies from the previous survey, provide areas of
focus, and offer suggestions for improvements to the process.
Although this process has been generally beneficial, given the stage of the
program as sites approach closeout, re-evaluation of the survey is timely. The
survey and recommendations should more reflect changing issues, the program’s
maturation, and site staff feedback.
A major improvement in the process would be the addition of regular and
consistent feedback from the CSEPP site staff. To make the survey and public
education/outreach recommendations more enhanced and comprehensive tool, it
would be very helpful to get information from the sites on the following:
22
Annual Site Objectives and Goals – Outreach objectives and tactics should
tie into the site’s strategic plan as well as survey data to have the most impact.
Feedback on Previous Recommendations – Were they implemented? If, so
how were they received? If not implemented, why not?
Evaluating Effectiveness of the Survey Tool
Information on How the Surveys can be More Helpful – Because the sites
have different demographics and issues, the surveys can be enhanced by
incorporating site-specific information or perspectives.
To provide measurable results, valid comparisons, and pertinent
recommendations, we propose modifying the format from its current structure.
For example, instead of listing significant increases in an area and notable
decreases in another and listing generic means for reaching audiences, the focus
may shift to minor alterations within the organization that will help bridge the gap
between what the public needs to do and what they will actually do in an
emergency. This will look different in the recommendations section than in
previous surveys. Instead of a bulleted positive and room for improvement
section, an overall evaluation of the program with big picture recommendations as
well as a focus on innovative means of enhancing current efforts will serve as
recommendations.
Most planning efforts tend to be based on how the emergency response
community will respond and what the public needs to know and do to ensure
emergency response is most effective. This is a necessary planning component to
ensure proper coordination and collaboration among disciplines, agencies, and
jurisdictions. However, rather than providing a strategy (i.e., to increase the
number of people surveyed about evacuation procedures from 43% confidence in
procedures to 55% confidence, which shows a significant increase, yet it is not
enough to ensure total compliance of a community), it may be beneficial to
consider a contingency plan for the other half of the population who are not
confident that they know the proper evacuation procedures in a chemical
emergency.
Additional recommendations may include more emphasis on message mapping
(i.e., the ability to gauge audience ability to receive messages) and/or in
combination with the area demographics. For example, throughout the program,
there has been heavy emphasis on CSEPP education in schools. However, in a
community where only 20% of the population has school-aged children, this is
not necessarily the most effective means for reaching the majority of the
population. Similarly, a community that indicates it prefers to receive information
via TV does not necessarily need an abundance of mail or paper inserts.
Conclusion
In recent years, CSEPP has evolved as it continues the mission to educate and
protect citizens in CSEPP communities. Public awareness surveys have proved to
be an integral resource throughout the program. The survey tool is able to
measure preparedness levels of the public, which allows sites to address
knowledge gaps and judge the impact of public outreach efforts. In order to best
support CSEPP, there is a need to actively pursue the most effective and dynamic
survey tool possible. Therefore, it is necessary to examine the role of the survey
23
Evaluating Effectiveness of the Survey Tool
tool in the program and continually enhance and modify the tool where
applicable.
The approach for evaluating the survey tool included a site-level analysis of past
survey data to observe trends and address knowledge gaps. Repeated measures of
preparedness levels provided in survey results were used to establish relationships
between focused public outreach efforts and public performance. Also, individual
sites were consulted to obtain their input on the use of the survey tool at their site
and enhancements for future surveys. Through this approach, the current needs of
sites and objectives for future survey work were able to be re-evaluated.
The survey evaluation has lead to the development of several suggestions for
improving future surveys. The goal to develop the most efficient survey tool
while still addressing the current needs of the different CSEPP sites may be
achieved through various efforts. One key issue is to ensure a short but effective
survey. To shorten the length of future surveys, only survey questions and
response options that work toward current program and site objectives shall be
included in the questionnaire. A shorter questionnaire may also improve response
rates, as it will take less effort for residents to participate. Other suggestions for
improving response rates are providing pre-notification letters to residents and
increasing the number of callback attempts in the survey process. Finally, to
continue to address the current needs of participating sites, recommendations for
future outreach can strive to take a different approach which will provide a
comprehensive, big picture evaluation of the public outreach program for each
site.
P o i n ts o f C o n ta c t
The primary IEM point of contact for this project is Sangeeta Singh. She may be
reached at:
IEM
8550 United Plaza Blvd., Suite 501
Baton Rouge, LA 70809
(225) 368-6765
(225) 952-8122 (fax)
sangeeta.singh@iem.com
The alternate point of contact is:
Jack Long
IEM
2014 Tollgate Road, Suite 208
Bel Air, Maryland 21015
(410) 569-8191
(410) 569-9553 (fax)
jack.long@iem.com
24
| File Type | application/pdf |
| File Title | Microsoft Word - Survey_Evaluation_Report_FINAL_1Oct2007.doc |
| File Modified | 2007-10-01 |
| File Created | 2007-10-01 |