Supp and Part B questions with correspondence included

NOAA National Weather Service

Probabilistic Snow Forecast Products Project

Part A and B DOC/NOAA Customer Survey Clearance

OMB Control No. 0648-0342

Expiration Date: 05/31/2018

Communicating Probabilistic Information for Decision-Makers: A Case Study Using Experimental Snow Forecast Products

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

For decades, NOAA’s National Weather Service (NWS) has provided snow forecast predictions based upon a deterministic solution. With improvements in high performance computing and numerical models, NWS forecasters can now take advantage of bundling numerous solutions, called an ensemble, for a winter weather event providing possible snow ranges and probabilities. Ensemble forecasting can aid in decision making by alerting individuals to a “most likely fit” along with the potential for predicting extremes, both high and low.

As the NWS transitions from historical deterministic products to probabilistic decision support services, the question focuses on how to provide this information in an optimal and consistent way that better enables the understanding of risk and improves decision-making. The NWS is experimenting with different methods and visualizations of showing probabilistic information, using different colors, categories and other visualizations. As such, the NWS is looking for input on how to optimize the design and communication of NWS experimental probabilistic snowfall accumulation products. This input is intended to feed into the larger NWS effort to communicate uncertainty and impacts across NWS service areas for the protection of life and property.

To accomplish this, the NWS seeks to conduct focus groups with a purposive sample of WFO partners, including emergency managers (BMs), state and local officials (e.g., school superintendents, state departments of transportation, public works), and a few broadcast meteorologists. The purpose of the focus groups is to gather partner feedback on the visual design of the graphics and overall snowfall probability communication to see how it improves NWS partner decision making.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the focus group guide. ERG has significant experience in organizing and facilitating focus groups for federal agencies that focus on customer satisfaction and outcome attainment. To develop the focus group guide, ERG worked with the NWS team including representatives from NWS Headquarters as well as local weather forecast offices (WFOs). Additionally, materials were also reviewed by Dr. Kim Klockow, a social scientist working for NOAA’s Office of Atmospheric Research, as well as Dr. Susan Joslyn, an expert in probability and decision science.

The focus group questions are divided into three sections. The first section focuses on general decision-making processes that depend on winter weather forecasts. ERG will explore participants’ information needs and what type of decisions they make at certain thresholds. The second section focuses on the actual NWS experimental probabilistic graphics. ERG will determine if participants understand the current products, probe for how useful they are for decision-making, and ask about graphical items such as labels, timing, colors, etc. The third section focuses on final graphical details and changes, including format needs, accessibility, and any additional information that would improve the decision-making process.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

ERG will conduct a combination of in-person and virtual focus groups (via GoToMeeting format) with a diverse group of NWS partners.

The onsite focus group locations include working with four WFOs in Denver, Colorado; Riverton, Wyoming; Sterling, VA; and Taunton, MA. Each of these areas was chosen for their varying levels of experience with winter weather events. ERG will conduct 1-2 focus groups with 8-12 participants each, for an approximate maximum of 96 participants. The focus groups will last approximately one hour. ERG will contact the Warning Coordination Meteorologist (WCM) at each WFO in these locations to organize the focus groups.

For the virtual focus groups, we will work with the WCMs to send out an invite to their partners to participate. Locations include Atlanta, GA; Louisville, KY; Detroit, MI; St. Louis, MO. ERG will conduct small group webinar interviews with up to 6 participants for an approximate maximum of 24 participants.

The WCMs will invite their core partners, including emergency managers, school superintendents, state departments of transportation (DOTs), public works and media using their respective decision support list serve. If the amount of interested participants exceeds 12, we will plan more than one focus group time. The list serves usually have approximately 500 -1500 individuals on it, of which emergency managers, schools, DOTs, public works, and media are a subset of this list. We aim to include up to 96 participants for onsite focus groups and 24 participants for virtual focus groups, but the exact number of participants will vary depending upon interest. We estimate that we will have about 60 emergency managers, 20 media, 16 schools, 16 public works, 8 State DOTs. Broadcasters may include local affiliates (ABC, NBC, CBS and FOX).

ERG will contact all of the WCMs to introduce the project, our role in the project, and request that focus groups be scheduled at a convenient time. For onsite focus groups, the NWS local office will be responsible for inviting their partners. The NWS expects that 75- 100 percent of the invitees will participate in the focus group with ERG, given the importance and level of interest in this project. Dates will be chosen to maximize participation. For virtual focus groups, the WCM will send the same invite to their partners, but ask them instead to contact us for a follow-up webinar date. ERG will coordinate the logistics at that point.

In scheduling each onsite focus group, ERG will introduce ourselves, explain our affiliation, describe what information we are seeking, and how long the interview will take:

Dear XX,

My name is XX. I am with Eastern Research Group (ERG), and I am emailing to schedule focus groups on behalf of NOAA’s National Weather Service. The focus of this research is on developing improved probabilistic snow products and information that will increase the likelihood of public safety officials making better decisions in the face of winter storms. We’re looking at 1-2 focus groups with emergency managers, DOTs, public works, and broadcast meteorologists.

Is this something you could help us organize? We’d be looking for your help in locating a venue and identifying and inviting the participants. Sometime in April and May would work well with our schedules.

Let me know if you are interested in helping us host these focus groups and if a conversation would be helpful to talk more about the specifics.

We will also share with them the following email invitation they can send to their partners once we have a focus group date:

The National Weather Service cordially invites you to participate in a focus group on "Communicating Winter Weather." This is a wonderful opportunity for our partners to not only provide feedback on how we conduct winter weather communication now, but also an opportunity to influence our future efforts. Please take advantage of this opportunity to provide feedback related to winter weather communication.

[Insert facilitator] and [Insert who is notetaking], with Eastern Research Group will conduct the focus groups.

Date: 

Location:  

The National Weather Service looks forward to your participation. Please save this date and time on your calendars and respond via email if you can or can not attend.  Your attendance is greatly appreciated.

If you have any questions or need additional information, please let us know.

For virtual focus groups, ERG will send the following introduction to the WCMs:

Dear XX,

My name is XX. I am with Eastern Research Group (ERG), and I am emailing to coordinate virtual focus groups with your partners on behalf of NOAA’s National Weather Service. The focus of this research is on developing improved probabilistic snow products and information that will increase the likelihood of public safety officials making better decisions in the face of winter storms. We were wondering if you could send the following message [See below] to your partners, and ask them to follow up with us.

Let me know if you are willing to help us connect with your partners. If a conversation would be helpful to talk more about the specifics, please let me know.

Message to your partners:

The National Weather Service cordially invites you to participate in virtual focus group on "Communicating Winter Weather." This is a wonderful opportunity for our partners to not only provide feedback on how we conduct winter weather communication now, but also an opportunity to influence our future efforts. Please take advantage of this opportunity to provide feedback related to winter weather communication.

Eastern Research Group is conducting the virtual focus groups through Go-to-Meeting conference calls. If interested and willing to participate, let us know, and we will pass along your interest to ERG.

ERG will generate detailed notes from the focus groups.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The NWS will use the information resulting from this data collection to help guide refinements in how they visualize winter weather probabilistic ensemble information. This information will also help the NWS determine if more explanation of products is necessary during DSS with core partners.

NWS will have ERG perform thematic analyses on the detailed focus group notes to identify key findings and recommendations. NWS is not using any statistical methods to select participants from the population and will select all population members.

The data do not directly contribute to a GPRA measure.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe is based on the size of the emergency management, school, DOT, public work, and broadcast meteorology community for the chosen NWS offices. The entire respondent universe within that NWS local office warning area will be invited to participate in the focus groups. For similarly formatted focus groups in Impact Based Warnings, ERG estimated approximately 45 participants in each of the 4 onsite locations for a total of 180. Table 1 below shows the actual participation for 10 onsite focus groups was 126 emergency managers and 27 media for a total of 153 participants. We expect a similar turnout, but with only 8 onsite focus groups, and 4 sets of virtual focus groups.

Table 1. Focus Group Participant (response rate) Information for Impact Based Warnings Focus Groups

Location	Venue	Date	Stakeholders	# Participants
Raleigh, NC	Raleigh Emergency Operations Center (EOC)	April 4, 2016	EMs Media	33 6
Winston Salem, NC	Wake Forest University	April 5, 2016	EMs Media	17 3
Benton County, IA	Benton County Sheriff’s Department	April 12, 2016	EMs Media	11 3
Des Moines, IA	Polk County Emergency Management	April 13, 2016	EMs Media	7 1
Oklahoma City, OK	Edmond EOC	April 14, 2016	Media	6
Norman, OK	National Weather Center	April 14, 2016	EMs	12
Norman, OK	National Weather Center	April 15, 2016	EMs	11
Montgomery, AL	Auburn University at Montgomery	May 19, 2016	EMs Media	13 5
Tuscaloosa, AL	National Water Center	May 20, 2016	EMs Media	8 1
Birmingham, AL	Birmingham EOC	May 20, 2016	EMs Media	14 2
Total	-	-	EMs Media	126 27

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Method for Stratification and Sample Selection

For the focus groups, the NWS is not using statistical methods for collecting this data.

Estimation Procedure and Accuracy

For the focus groups, the NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The NWS office in the region will send out the invitation for the focus groups. This invitation should help to encourage response since it will come from an entity the potential attendees will recognize. Additionally, the potential attendees have all previously worked closely with the NWS. The NWS will send out reminder invitations as well.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The NWS consulted with Eastern Research Group, Inc. (ERG) on the development of the focus group guide. ERG has significant experience assessing technical assistance provided by federal agencies through detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The focus group guide development process was informed through discussions with NWS staff.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc (ERG) of Lexington, Massachusetts, to design the focus group guide and implement the data collections. ERG’s project manager for this work is Gina Eosco (781-704-4458; gina.eosco@erg.com).

Probabilistic Snow Forecast Products Project 6