Hazard Simplification Case Study Supp and Part B questions

OMB Control No. 0648-0342

Expiration Date: 05/31/2018

Hazardous Weather Messaging Case Study Survey

for NOAA’s National Weather Service Hazard Simplification Project

Amendment to NOAA Watch, Warning, and Advisory Focus Groups

(ICR Reference Number 201103-0690-001, approved 3/14/14)

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The survey will be conducted by NOAA’s National Weather Service (NWS). The NWS forecasts hazardous weather situations and issues warnings, watches, advisories (WWA), and other information products to convey the threats posed by these events. These products are intended to help communities prepare for and respond to hazardous weather in order to protect people’s lives and property. The products are communicated to the public through websites, smart phones, television programs, radio broadcasts, and NOAA Weather radio. NWS customers include weather professionals, transportation and aviation officials, emergency management personnel, public works departments, broadcast meteorologists and other media, and the public.

The NWS has embarked on an effort to simplify its WWA products, since both prior social science research and NWS service assessments have demonstrated that many members of the public, and even some NWS partners, don’t understand the distinctions among the terms used in the different WWA products or their intent. However, any change to the current WWA system must happen deliberately, gradually, and with transparency since the terms are recognized and widely used by NWS partners, as well as institutionalized into some kinds of societal decision-making.

This survey builds on and furthers the social science research conducted in the summer of 2014 that involved focus groups with emergency managers, broadcast meteorologists, NWS Weather Forecast Office staff, and the public. The focus groups explored the current understanding and utility of the WWA system and possible enhancements to a new or modified system (ICR Reference Number 201103-0690-001, 3/14/14). This survey asks for respondents to provide specific examples of the strengths and weaknesses of the current system to help NWS understand what kinds of potential changes would be beneficial to make to the present WWA system.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

The NWS contracted with Eastern Research Group, Inc. (ERG) to develop this survey. ERG has significant experience in planning and conducting social science research, including interviews, focus groups, and surveys that focus on customer satisfaction and outcome attainment.

To develop the survey, ERG’s Dr. Gina Eosco, a well-known and respected social scientist in the weather community, worked closely with a team of NOAA social scientists and contractors, including Dr. Kim Klockow, a Post-Doctoral Fellow at the University Corporation for Atmospheric Research (UCAR), who is currently working with the NWS in the Office of Atmospheric Research and Office of Weather and Air Quality; Dr. Danielle Nagele, a risk communication expert in NOAA’s National Ocean Service (NOS), Dr. Vankita Brown, a social scientists with NOAA’s NWS Center For Advanced Public Safety, and Dr. Chris Ellis, a social scientist with the NOS Office for Coastal Management. ERG also worked closely with NOAA and NWS leadership on the project, including Elliott Jacks, Acting Chief, Forecast Services Division for the NWS Analyze, Forecast and Support Office, and Jennifer Sprague, NOS Acting Chief of Staff. Additionally, ERG consulted with Rebecca Jenning, a meteorologist with the Federal Emergency Management Agency (FEMA), and Nate Johnson, a broadcast meteorologist with WRAL-TV in North Carolina. Suggestions for improving the survey included changing wording to address the responders’ frame of reference, avoiding acronyms and NWS jargon, and shortening and simplifying the questions being asked.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The on line survey link will be distributed to partners in the emergency management/response communities in each NWS Weather Forecast Office (WFO) via each of the NWS’s 122 WFOs, as well as to broadcast meteorologists and members of the media who cover weather. Each WFO keeps detailed email list of these entities. These lists vary from office to office but typically contain a few hundred names. The survey will also be distributed to broadcasters and other members of the media via the American Meteorological Society (AMS), which has a total membership of 14,000 of which 558 are certified broadcast meteorologists.

NWS anticipates that approximately 10 percent of the customers will respond. The survey contains a fair number of open-ended responses, which could limit response. However, respondents will be individuals who know, use, and care about NWS products; many of these individuals will also have worked closely with WFO staff and will understand the value of the survey. These close working relationships may help to boost survey response. NWS also plans to increase the sample size by using a chain referral approach, whereby each respondent will be encouraged to share the survey link with colleagues. NWS will also send several reminders to potential survey takers to encourage them to fill out the survey.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The data gained from this survey will be reported to the NWS project leadership with the goal of further consideration of possible changes or modifications to the WWA system. NWS will not be sampling from a customer population for this data collection. The data are intended to provide guidance from interested stakeholders in the way in which information should best be conveyed. Thus, no extrapolation to a population will be performed.

The data do not directly contribute to a GPRA measure.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe is approximately 27,454 individuals, representing agencies and organizations typically involved in emergency planning and response. This includes federal agency representatives (Coast Guard, FAA, FEMA, DOT, USGS, EPA, DOD, etc.) state/tribal agency representatives (state departments of environmental protection, state emergency management agencies, state departments of transportation, etc.), and local government employees (fire and rescue, emergency managers, utilities, department of public works, etc.), as well as private sector individuals (e.g., hospital consortia, college/university emergency managers, broadcasters, media, and others). No statistical methods are being used in the participant selection. The expected response rate is 2,745 (10 percent).

Respondent Source	Federal Agency	State Agency	Local/Tribal Agency	Private
WFO Partner List 122 x 200 =24,400*	6,100	6,100	6,100	6,100
AMS				558
Referrals**	610	610	610	610 + 56 = 666
Subtotals	6,710	6,710	6,710	7,324
Total	27,454
Expected Response (10 percent of Total)	2,745

* There are 122 NWS WFOs—each with its own partner list. Based on information from the WFOs, we assume the average WFO list contains about 200 partner names. Also based on information from the WFOs, we presume the partners are split fairly evenly among federal agencies, state agencies, local agencies, and the private sector.

** Survey respondents will be encouraged to share the survey link. The number of referrals assumes 10 percent of each category of potential respondents will complete the survey (610) and all share the link. For the AMS referrals, this assumes 10 percent of the broadcasters (56) complete the survey and all share the link.

No statistical methods are being used in the participant selection.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification and Sample Selection

NWS is not using statistical methods for collecting these data.

Estimation Procedure and Accuracy

NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The WFOs and AMS will send a personalized letter with the survey link to the potential respondents; this letter should help to encourage response since it will come from an entity the potential respondents will recognize. Additionally, the potential respondents will be individuals who know and use NWS products; many of these individuals will also have worked closely with WFO staff and will understand the value of the survey. These close working relationships may also help to boost survey response. The NWS will also promote the survey at several upcoming AMS and National Weather Association conferences to help boost response, and reminders will also be sent to potential respondents to encourage their participation.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The NWS distributed a slightly different version of this survey, tailored to forecasters, internally within the NWS in early April. The development of that survey provided the basis for this external survey. NWS social scientists and NWS leadership provided feedback on that survey before it was distributed. Additionally, a series of focus groups conducted under ICR Reference Number 201103-0690-001 (approved on 3/14/14) provided valuable input to develop this survey about the current strengths and weaknesses in the watch, warning, and advisory system.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, to design the survey instrument, develop the sampling approach, implement the survey, and analyze the resulting data collected. The survey design team included the following individuals:

Dr. Lou Nadeau (781) 674-7316; lou.nadeau@erg.com

Dr. Gina Eosco (781) 704-4458;

gina.eosco@erg.com

Linda Girardi (703) 841-0501; linda.girardi@erg.com

5