Supplementary and Part B questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB Control Number 0648-0342; expiration date 6/30/2021)

PROPOSED TROPICAL CYCLONE FORCAST/ADVISORY (TCM) SURVEY

Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This survey will be conducted by the National Weather Service’s (NWS) Tropical Services Program using the online interface Survey Monkey. The Tropical Services Program provides

“programmatic leadership for the nation´s tropical weather program, which includes operations at the National Hurricane Center (NHC) and the Central Pacific Hurricane Center (CPHC). These centers are responsible for issuing watches, warnings and other supporting products linked to the evolution, track and impacts of tropical storms and hurricanes in the Atlantic, Caribbean, Gulf of Mexico and central and eastern Pacific Oceans.”¹ NHC and CPHC along with NWS social scientists have provided critical guidance on the composition and distribution strategy for the survey. In addition, the Tropical Services Program was granted a request through NOAA Educational Partnership Program (EPP) for a graduate student specializing in meteorology and social science to assist with this survey.

This survey will be used to gather information on how the NWS’s Tropical Cyclone Forecast/Advisory (TCM) product is used and collect feedback on the impact of potential format changes to the TCM. The TCM product “contains current and forecast storm information in a fixed format suitable for parsing by computer software. It contains a list of all current coastal watches and warnings, cyclone position, intensity, and direction and speed of motion.”² The TCM is used by a variety of NWS partners including the media, weather software designers, mariners, and public safety officials such as emergency managers and the Department of Defense, to make the decisions necessary to prepare for a tropical cyclone. These partners receive the information provided in the TCM through the web, RSS feeds, FTP, the NWS Gateway, and satellite dissemination services.

________

¹ https://www.weather.gov/organization/afs - Text taken from the section “Forecast Services Division”, subsection “Marine, Tropical, and Tsunami Services Branch

² https://www.nhc.noaa.gov/pdf/NHC_Product_Description.pdf - Text taken from the section ”Tropical Cyclone Forecast/Advisory”

Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics? What suggestions did you get about improving the survey?

This survey was developed by a cross-NWS team, led by the NWS Tropical Services Program Leader, Jessica Schauer, and the aforementioned EPP intern, Ayesha Wilkinson. NHC Warning Coordination Meteorologist (WCM) Daniel Brown served as the chief technical advisor. Feedback on the content of the survey was gathered from NHC, CPHC, and forecasters at the NWS Weather Forecast Office in Miami. This group ensured that the survey questions accurately represented current and potential future operations. In addition, both this group and the NOAA Public Affairs Officer at NHC provided guidance on the target audience for the survey and how best to present the information for the audience. For example, the Tropical Analysis and Forecast Branch (TAFB) within NHC provided guidance on the proper terms to describe the marine community (i.e., cruisers and commercial ships).

NWS Office of Dissemination social scientist Tyra Brown Harris served as the chief advisor on the format of the survey, with a number of other NOAA social scientists also serving as consultants. The cross-NWS team was advised on the survey structure that would be most appropriate to gather feedback from TCM users who both read and parse the product as well as on improving questions and answer choices. Suggestions included, but were not limited to, changing the form of multiple choice questions into a ranking scheme so results can show the relative importance of the various facets of the TCM compared to a simple “yes or no.”

Furthermore, the survey is provided in two languages, English and Spanish. These are the primary languages of the users within the Atlantic and eastern and central North Pacific basins where the TCM product is valid. The translation to Spanish was led by Maria Torres of the NWS Multimedia Assistance in Spanish (MAS) Team. This team regularly translates NWS messaging for the Spanish-speaking audience.

In addition, the survey was approved by the NWS Tropical Services Program Team (SPT) composed of representatives from all operational and support services branches of the NWS, including all NWS regional headquarters that issue tropical products.

Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate?

The survey will be conducted through a web-based link via Survey Monkey, which is an online service that can be accessed through computer, tablet, or mobile device. No password will be required to access the survey. Survey respondents may choose to take the survey in English or Spanish. The survey questions will be in the form of multiple choice and ranking answers. The end of the survey will include an optional comment box. Prior knowledge of the TCM product is a prerequisite for taking the survey. The survey will end for all respondents who indicate that they do not use the TCM product. Partners will be provided with the web-based link via internal email partner lists provided by NHC, CPHC, NWS National Service Programs, and the network of NWS Warning Coordination Meteorologists (WCMs) who serve in hurricane-prone areas. If a respondent does not identify their self as a member of one of the professional affiliation options presented in the survey to identify the target audience, the survey will end for that respondent. The estimated audience size for the survey is approximately 2,000. This estimate includes those on partner lists that are highly familiar with products issued by NHC and CPHC. The NWS will direct their WCMs taking part in partner conferences and other meetings to direct partners to the survey while responses are being solicited. It is estimated that approximately 40% of those solicited will respond. It will take the average respondent approximately 15 minutes to complete this voluntary survey.

Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

While the results will be segregated by the participants’ profession to gain a general understanding of how each partner group uses and accesses the TCM, we do not expect our results will generalize to the U.S. population nor to the population of any specific partner group. This work alone will not be used to premise any changes to products and services.

Skip logic will be applied in Survey Monkey such that those who do not self-identify as a user of the TCM product will not be able to proceed with the survey. Basic statistical analysis of the responses will provide information on elements of the TCM that are the most and least useful as well as the utility and effectiveness of the product.

This survey is not intended to inform or assess a GPRA measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The estimated number of partners that will be asked to take this survey is around ~2,000. This number was derived by culling partner contact lists from NWS offices with operational responsibility during tropical cyclone events. The partner lists include both international partners and partners located in the United States. Those most likely to take the survey are partners who regularly respond to requests from NWS WCMs. The NWS is charged by The Weather Research and Forecasting Innovation Act of 2017 (Pub.L. 115–25) (in Section 405) to employ at least one warning coordination meteorologist at each weather forecast office of the NWS. The job of the WCM is to serve as a liaison to local and regional users of NWS products and weather information, to ensure that products are meeting user needs and improving public response to weather events. WCMs at operational NWS offices maintain up-to-date partner lists as they are responsible for educating and communicating with users about NWS products. The primary partner groups are public safety and government officials, information technologists, media, and the marine industry. NWS estimates a partner response rate of 40%. Demographics for those partners who participate are likely to be heavily male and well educated compared to the U.S. population based on previous surveys conducted with similar audience.

This survey will provide guidance to how partners are using the TCM and accessing it. There are three groups of interest for this survey: (1) users who communicate the information from the TCM product directly to the public, (2) end users who read the TCM text file directly, and (3) those who repackage the information from the TCM primarily through parsing. Those who communicate the information, such as broadcasters and other media, may read the TCM as a text product or use graphics created by information technologists by parsing the data from the product. End users such as the marine community are believed to primarily read the TCM product as posted. This survey is intended to gain understanding on how each respondent group uses the TCM product.

Table 1: Potential Respondent Universe and Expected Response Rate

Category of Respondent	Estimated Number of Qualified Respondents *	Expected Response Rate #
Information Technologists (Software / App / Programmer)	150	60
Media	900	360
Marine Industry	250	100
Public Safety/Government	700	280
TOTAL	2,000	800
* Based on information obtained from multiple NWS partner email lists. # Assuming a 40% response rate from partners invited to partake in the survey.

Statistical analysis of the results will primarily be performed by running data queries using the Survey Monkey software.

Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The link to the web-based Survey Monkey survey will be provided directly to partners through an internal email list from NHC, CPHC, NWS National Service Programs, and the network of NWS Warning Coordination Meteorologists (WCMs) who serve in hurricane-prone areas. If a respondent does not identify their self as a member of one of the professional affiliation options determined the target audience, the survey will end for that respondent. The next question in the survey will determine if the respondent is familiar with the TCM; if they are not, the survey will end. This will ensure the responses are pertinent to the target audience. Responses will be statistically analyzed based on professional affiliation. These results are not expected to be a generalization of an entire population. This survey will reduce any burden of data collection through using a web-link based survey which collects all responses online. The results of this survey alone will not directly dictate changes to the format or content of the TCM.

Survey data will be stored on an NWS headquarters password-protected workspace, accessible only by staff specifically assigned to this project.

Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

In order to maximize the response rates, they survey is short. The survey completion time is around 15 minutes based on the question format. The question format was taken into consideration to maximize response rate by providing multiple choice, drop down, and ranking questions compared to open-ended responses, so the participants can quickly take the survey. Partial and non-submitted surveys will not be incorporated into the data analysis.

The survey link will be sent to partners who are familiar with the products of the NWS and are more likely to engage in the survey. The web link for the partner survey will be provided only through an email list from NHC, CPHC, NWS National Service Programs, and the network of NWS Warning Coordination Meteorologists (WCMs) who serve in hurricane-prone areas. When the survey is active, we plan to monitor responses and send a follow-up email to the partner group(s) whose response rate may not be as expected to remind them to participate in the survey. In addition, NWS employees participating in applicable conferences and meetings during the period that the survey is open will encourage partners to take the survey.

This survey is not intended to be generalizable, as it pertains to a product that is used by specific sectors, rather than the general public. All relevant sectors will be represented in the survey audience.

Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB give prior approval.

The team conducted a small participation survey (fewer than five respondents) to find any bugs or corrections needed before the advisory examples/questions go “live” for survey completion.

Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Jessica Schauer

NOAA-Federal

NWS Analyze, Forecast, and Support Office, Forecast Services Division

Tropical Services Program Lead

jessica.schauer@noaa.gov

305.229.4476

Ayesha Wilkinson

NOAA-Affiliate (NCAS)

Howard University

Ayesha.Wilkinson@noaa.gov

678.860.2294

Dan Brown

NOAA-Federal

National Hurricane Center Hurricane Specialist Unit (HSU)

Warning Coordination Meteorologist (WCM)

daniel.p.brown@noaa.gov

305.229.4441

Jennifer Sprague-Hilderbrand
National Weather Service Office of Programming, Planning and Service Delivery
Senior Advisor

Jennifer.Sprague@noaa.gov

301.427.9065