Supplemental and Part B questions

A. Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB Control Number 0648-0342)

PROPOSED WEA PARTNER SURVEY

1. Explain who will be conducting this survey. What program office will be

conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This survey will be conducted jointly by the National Weather Service’s (NWS) Forecast Services Division (FSD) and the NWS Office of Dissemination (ODIS) using the online interface Survey Monkey. The FSD’s primary responsibility is the development of requirements, policy, and procedures for all NWS forecast and warning services. The ODIS is responsible for integration and planning of new dissemination requirements-driven technologies for performance enhancement. The audience for this survey is specific NWS partners including emergency management, first responder, and government officials. Most targeted by this survey are partners that have now, or could have in the future, the capability to issue a Wireless Emergency Alert (WEA). According to the Federal Emergency Management Agency website:

“Federal, state, and local laws determine which public safety officials are granted the authority to alert the public of emergency situations. Specific authorities may be designated in a state’s emergency communications plans, such as the state’s Emergency Alert System plan and America's Missing: Broadcasting Emergency Response (AMBER) Alert plan. Generally, eligible organizations are:

• Federal Agencies

• State Governments

• Territorial Governments

• Tribal Governments

• Local Governments

Other public or private sector organizations may be eligible depending on their public safety mission.”¹

WEA is activated by the NWS for life-threatening warnings. These partners are the most appropriate audience for this survey as they (a) must be prepared to directly interface with those in their community that receive an NWS WEA, and (b) they have likely spent time considering the appropriate content of WEA messages relative to the hazard being alerted.

________

¹ https://www.fema.gov/alerting-authorities - Text taken from the section ”Who can sign up for FEMA IPAWS” (Integrated Public Alert and Warning System)

2. Explain how this survey was developed. With whom did you consult during the

development of this survey on content? statistics? What suggestions did you get about improving the survey?

This survey was created by the NWS WEA Policy Team which has members from the NWS Regions, the National Service Programs (i.e., tsunami, tropical, etc.) that alert using WEA, and representatives from ODIS. The team was formed to address new Federal Communications Commission regulations and recommend policy to NWS senior leadership for the provision of WEAs. The survey is specifically intended to support NWS’ effort to adopt FCC 16-127 which directs 4G wireless networks and greater to be able to support 360 character WEA messages by May 2019. The team consulted with contracted NWS social science staff to determine the structure of the questions that would be most appropriate to gather feedback on both the current 90 character and proposed 360 character WEA messages.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate?

The NWS will conduct the survey using Survey Monkey, an online software that partners will access via a web link. The NWS will use both NWS-wide partner lists and email lists specific to the hazards addressed in the survey and alerted for via WEA. The distribution of the survey will be based on, but not be limited to, those partners with public alerting authority that has either been approved or is in the process of being approved by the Federal Emergency Management Agency (FEMA). 

https://www.fema.gov/media-library/assets/documents/117152

These lists do not include federal partners such as public safety specialists within the Federal Emergency Management Agency or the Department of Defense, nor do they include private emergency management companies that have subscription services for emergency alerts. The potential respondent universe was rounded up to 1500 to accommodate these other groups. NWS will use their network of Warning Coordination Meteorologist (WCM) to personally send partners with responsibility for public safety and alerting to the survey. The NWS will direct their WCMs taking part in partner conferences and other meetings to direct partners to the survey while responses are being solicited. In addition, the NWS will use social media and advertising on NWS websites to direct partners to the survey. The first question of the survey will determine if the respondent fits within the targeted professional affiliation; if they do not the survey will end. It is estimated that approximately 40% of those solicited will respond.

It is estimated that it will take the average respondent approximately 15 minutes to complete this voluntary survey.

4. Describe how the results of this survey will be analyzed and used. If the customer

population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The feedback from this survey will be used to inform NWS on the content and language used within its WEA messages during life-threatening situations. We do not expect our results will generalize to the US population nor to the full emergency management/first responder/government population. This work alone will not be used to premise any changes to products and services. Instead, results will inform potential changes to NWS WEA alerting.

Basic statistical analysis of the responses will provide information on elements of the WEA messages that are the most and least useful as well as expected effectiveness of the messages.

This survey is not intended to inform or assess a GPRA measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any

sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Those most likely to take the survey are partners who regularly respond to requests from NWS Warning Coordination Meteorologists (WCMs). The NWS is charged by the The Weather Research and Forecasting Innovation Act of 2017 (Pub.L. 115–25) (in Section 405) to employ at least one warning coordination meteorologist at each weather forecast office of the NWS. The job of the warning coordination meteorologist is to serve as a liaison with local and regional users of NWS products and weather information, to ensure that products are meeting user needs and reaching potential in improving public response and other outcomes to weather events.

Table 1 : Potential Respondent Universe and Expected Response Rate

Category of Respondent	Estimated Number of Qualified Respondents *	Expected Response Rate #
Emergency Management	1,000	400
First Responder (i.e., police, fire, etc.)	100	40
Government	400	160
TOTAL	1,500	600
* Based on a rounded number of emergency management offices, first responders, and governments that have approval or have asked for approval from FEMA for IPAWS alerting authority # Assuming a 40% response rate

Demographics for the expected respondent universe are primarily male and well-educated. Statistical analysis of the results will primarily be performed by running data queries using the Survey Monkey software.

2. Describe the procedures for the collection, including: the statistical methodology for

stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

It is not the intent of this research to generalize to a broader population but rather, to obtain initial feedback from a set of NWS core partners to inform potential future changes to NWS WEA messaging. This survey, alone, will not premise any changes to products and services.

The first question of the survey will determine if the respondent fits within the targeted professional affiliation; if they do not, the survey will end. Then, respondents will be asked to identify the most-concerning and most-frequent hazards in their area of responsibility. Skip logic will direct them to a corresponding WEA messages and subsequent questions related to those messages. This technique allows multiple hazards to be examined without burdening each respondent for a response on every hazard NWS alerts via WEA. All feedback will be useful.

Survey data will be stored on an NWS headquarters password-protected workspace, accessible only by staff particular to this project.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield “reliable” data that can be generalized to the universe studied.

In order to maximize response rates, the WEA Policy Team has limited the amount of proposed questions to 14. Aside from a final question that asks for general feedback, none of the proposed questions will be open-ended. This will cut down on the time it takes the user to complete the survey.  Partial and non-submitted surveys will not be factored into the data analysis.   

We plan to maximize response rates and minimize non-response by working with our WCMs to directly engage their local partners. We will monitor responses during the period when the survey is active and send reminders to those groups who have shown a lower response to encourage increased response rates as needed.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved, OMB must give prior approval.

The internal social scientists will be conducting a limited participant test (fewer than ten) before the advisory examples/questions go “live” for survey completion.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Danielle Nagele, Ph.D.

NOAA-Affiliate (INNOVIM)

NWS Analyze, Forecast, and Support Office, Forecast Services Division

danielle.nagele@noaa.gov

301.427.6919

Tyra Brown Harris, Ph.D.

NOAA-Federal

NWS Office of Dissemination

tyra.brown@noaa.gov

202.379.6676

Jessica Schauer

NOAA-Federal

NWS Analyze, Forecast, and Support Office, Forecast Services Division

Tropical Services Program Lead

jessica.schauer@noaa.gov

305.229.4476

8