NOAA National Weather Service
National Weather Service Institutionalization Survey
Part A and B DOC/NOAA Customer Survey Clearance
OMB Control No. 0648-0342
Expiration Date: 05/31/2018
NATIONAL WEATHER SERVICE (NWS) Watch, Warning, and Advisory Program
Institutionalization study survey
Supplemental
Questions for DOC/NOAA Customer Survey Clearance
(OMB Control
Number 0648-0342)
Explain who
will be conducting this survey. What program office will be
conducting the survey? What services does this program
provide? Who are the customers? How are these services provided to
the customer?
NOAA’s National Weather Service (NWS) issues watch, warning, and advisory (WWA) products to help communities prepare for and respond to hazardous weather in order to protect people’s lives and property. These various WWA products (e.g., flash flood watch, tornado warning, heat advisory) are communicated to the public through websites, smart phones, television programs, radio broadcasts, and NOAA Weather radio. NWS customers include weather professionals, transportation and aviation officials, emergency management personnel, public works officials, broadcast meteorologists and other media, the public, and others.
The NWS seeks to conduct a survey to gather feedback from
organizations that use hazardous weather warning information to
discern the degree to which any specific types of watches, warnings,
and advisories (or the actual terms “watch,” “warning,”
and “advisory”) are embedded in their decision-making,
laws, policies, operating procedures, technologies, by-laws, or other
activities or processes. The goal of the data gathering is to gauge
the level of institutionalization of these products and terms in
different industry sectors as the NWS contemplates potential
improvements to the present WWA system.
The target audience for the effort is individuals who use or have knowledge of how NWS weather information is used within their organizations to make decisions, develop laws or policies, create standard operating procedures, or conduct other tasks. These individuals may include representatives from emergency management agencies (at the local, state, and federal levels), departments of transportation, utilities, insurance companies, schools and universities, and others.
This effort builds on and furthers social science research conducted in the summer of 2014 that involved focus groups with emergency managers, broadcast meteorologists, NWS Weather Forecast Office staff, and the public. The focus groups explored the current understanding and utility of the WWA system and possible enhancements to a new or modified system (ICR Reference Number 201103-0690-001, 3/14/14). This work indicated that it would be beneficial to gain a better understanding of the degree to which the WWA products and terms are potentially institutionalized in different sectors of the society so that the NWS can factor this information into any consideration of potential changes to the present WWA system.
Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?
NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the survey. ERG has significant experience in conducting detailed interviews, focus groups, usability tests, and surveys for federal agencies that focus on customer satisfaction and outcome attainment. To develop the survey, ERG first conducted interviews with individuals representing agencies and organizations in different sectors that use weather information to make decisions:
Federal Sector/Transportation: Department of Transportation Federal Highway Administration
Federal Sector/Emergency Management: Federal Emergency Management Administration
State Sector/Transportation: American Association of State Highway and Transportation Officials
Local Government Sector/Emergency Management: Morgan County, AL, Emergency Management Division
Private Sector/Energy: Duke Energy
Private Sector/Insurance: Insurance Council of Texas
Private Sector K-12 Schools: Pennsylvania School Boards representing K–12
Private Sector Colleges/Universities: University of Santa Barbara, CA
ERG used the information from these interviews to develop the survey. The survey asks a series of questions to discern if WWA products or terms are institutionalized in organizations’ policies, procedures, guidance, technologies, or by-laws, as well as potential impacts that any change in the WWA present system would have on these organizations. Overall, we found through these interviews, that sectors are more concerned with the information embedded within the terms and not the terms themselves, so we included questions in the survey reflecting this distinction. Additionally, we learned through the interviews that WWA terms and information can be embedded not only in policies and by-laws, but also in the operating procedures (e.g., nuclear power plants) and software used by certain agencies, so we broadened the scope and type of questions asked in the survey to reflect these items.
Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)
The NWS will format the survey using the software package SurveyMonkey. The NWS will then distribute a link (generated by this software) to the survey through an invitation email sent to a number of email lists, as described below.
There are three main ways in which the survey link will be distributed:
Each of the NWS’s 122 Weather Forecast Offices (WFOs) maintains detailed email lists of entities in their jurisdictions that rely on NWS information for decision-making. These lists vary from office to office but typically contain a few hundred names. The individuals on the lists represent various agencies and organizations in the private and public sectors, including transportation officials, emergency managers, first responders, aviation officials, parks and recreation officials, public works officials, health officials, and many others. The survey will be distributed to the individuals whose names appear on these WFO lists.
The survey link will also be distributed to a number of federal agency officials through the coordination of the Office of the Federal Coordinator for Meteorology (OFCM), an interdepartmental office that facilitates the systematic coordination of operational weather services and supporting research among the federal agencies.
Finally, the survey will be shared with the members of various public and private membership-based organizations representing sectors that use meteorological information.
The NWS expects that 10 percent of the survey respondents will complete the survey, and that another 10 percent of the respondents will share the link with a colleague or other member of their profession. The NWS expects that approximately half of the respondents who received a shared link will complete the survey. While the survey will be widely distributed to membership-based organizations and government agencies at local, state, tribal, and federal levels, it is likely that many of the individuals who receive the survey link will not be the “right” person (i.e., the person within the agency who knows if WWA terms or products are codified or institutionalized in their organization), which could limit response. However, many of the potential universe of respondents will also be individuals who know, use, and care about NWS products and recognize the importance of the survey.
The NWS plans to increase the sample size by including the survey link on its websites and social media. It will also include instructions in the survey encouraging potential respondents to share the survey link with the appropriate members of their organizations. Using this referral approach will help to appropriately target the survey and increase response. The NWS will also personally call relevant membership organizations and federal agencies that it will ask to distribute the survey to ensure they understand the scope and purpose of the survey and can direct it appropriately within their organizations. Finally, the NWS will send several reminders to potential survey takers to encourage them to fill out the survey.
Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)
The data gathered from this survey will be reported to the NWS project leadership with the goal of further consideration of possible changes or modifications to the WWA system. NWS will not be sampling from a customer population for this data collection. The data are intended to provide insights from relevant stakeholders on the degree to which present terminology is embedded or institutionalized within their organization. Thus, no extrapolation to a population will be performed.
The data do not directly contribute to a GPRA measure.
Collections of Information Employing Statistical Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.
The table on the following page lists the number and categories of potential respondents. These respondents can be broken into three broad categories:
Respondents on WFO lists: Each NWS WFO will send a personalized letter with the survey link to the individuals on its email list. While the number of people on each WFO list varies from one office to another, we are assuming an average of 200 names are on each WFO list (200 names/list x 122 WFOs = 24,400 respondents).
Respondents on OFCM mail lists: The NWS will contact each of the 15 agencies that comprise the OFCM to ask if they will distribute the survey link; based on discussions with the OFCM, we estimate approximately 150 potential respondents will receive the survey.
Respondents from membership-based organizations: The NWS will contact membership-based organizations that represent key pertinent industry sectors (e.g., aviation, energy, water, transportation) to ask if the organization would distribute the survey link to its members. Based on membership totals provided by their websites, this portion of the sample is estimated at 12,920 individuals and/or organizations.
The NWS expects that 10 percent (3,747) of the total survey respondents (37,470) will complete the survey, and that another 10 percent of the respondents will share the link with a colleague or other member of their profession. The NWS expects that approximately half of the respondents who received a shared link (1,874) will complete the survey. Therefore, the total potential sample is 5,614.
Table I. Potential Survey Sample
Respondent Source |
Federal Agency |
State Agency |
Local/Tribal Agency |
Private |
WFO Partner List 122 x 200 =24,400* |
6,100 |
6,100 |
6,100 |
6,100 |
Office of the Federal Coordinator for Meteorology (15 federal agencies) |
150 |
|
|
|
Membership-Based Organizations |
||||
|
|
3,000 |
3,000 |
3,000 |
|
|
|
|
900 |
|
|
|
|
400 |
|
|
|
350 |
|
|
|
|
|
200 |
|
|
300 |
|
|
|
|
|
|
650 |
|
|
|
|
130 |
|
|
|
|
140 |
|
|
|
|
850 |
Subtotals |
6,250 |
9,400 |
9,450 |
12,370 |
Total Potential Sample |
37,470 |
|||
Expected Response (10 percent of Total) |
3,740 |
|||
Referrals (5 percent of Total) |
1,874 |
|||
Totals (Expected Responses + Referrals) |
5,614 |
|||
1 There are 122 NWS WFOs—each with its own partner list. Based on information from the WFOs, we assume the average WFO list contains about 200 partner names. Also based on information from the WFOs, we presume the partners are split fairly evenly among federal agencies, state agencies, local agencies, and the private sector.
2Available membership information does not signify federal, state, local, or corporate levels; for this table, the total membership number that was available was divided equally among the likely members’ levels based on the organization’s mission.
3Available membership information is at a corporate, organization, or agency level—not an individual membership level.
4 Survey respondents will be encouraged to share the survey link. The number of referrals assumes 10 percent of each category of potential respondents will share the link and that half of these referrals will complete a survey.
Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Statistical Method for Stratification and Sample Selection
NWS is not using statistical methods for collecting these data.
Estimation Procedure and Accuracy
NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates is not meaningful to calculate
Unusual Problems Requiring Specialized Sampling Procedures
None are required.
Periodic Data Collection Cycles
This request is for a one-time data collection.
Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.
The survey will be widely distributed through NWS email lists, through the OFCM, and through membership-based organizations and government agencies at local, state, tribal, and federal levels. The NWS plans to maximize response rate by including the survey link on its websites and social media. It will also include instructions in the survey encouraging potential respondents to share the survey link with the appropriate members of their organizations. Using this referral approach should help to appropriately target the survey and increase response. The NWS will also personally call the membership organizations and federal agencies that it will ask to distribute the survey to ensure they understand the scope and purpose of the survey and can direct it appropriately within their organizations. Finally, the NWS will send several reminders to potential survey takers to encourage them to fill out the survey.
Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.
Before developing the survey, ERG (under contract to NOAA) first conducted interviews with individuals representing agencies and organizations in different sectors that use weather information to make decisions:
Federal Sector/Transportation: Department of Transportation Federal Highway Administration
Federal Sector/Emergency Management: Federal Emergency Management Administration
State Sector/Transportation: American Association of State Highway and Transportation Officials
Local Government Sector/Emergency Management: Morgan County, AL, Emergency Management Division
Private Sector/Energy: Duke Energy
Private Sector/Insurance: Insurance Council of Texas
Private Sector K-12 Schools: Pennsylvania School Boards representing K–12
Private Sector Colleges/Universities: University of Santa Barbara, CA
The information gleaned from these interviews was used to guide the development of the survey instrument. Once a draft survey instrument was developed, the NWS shared the draft with these eight interviewees for their feedback.
Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA, to design the survey and implement the data collections. ERG’s project manager for this work is Linda Girardi (703-841-0501; linda.girardi@erg.com).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Lou |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |