Supporting statement

A. Supplemental Questions for DOC/NOAA Customer Survey Clearance (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

In May 2015, the National Weather Service (NWS) awarded a contract to McKinsey & Company (“McKinsey”) with the overarching goal of reforming the NWS’s Weather-Ready Nation vision, including the implementation of Impact-Based Decision Support Services (IDSS). A key focus is to expand the NWS’s capacity to deliver IDSS, which is the provision of relevant information and interpretative services to enable core customers’ (“partners”) decisions when weather, water, or climate has a direct impact on the protection of lives and livelihoods. A major component of improving the provision of IDSS is understanding the NWS’s customers’ experiences with its services through a survey. McKinsey will be conducting the survey.

The office that will be conducting the survey is the Office of the Chief Operating Officer (OCOO) of the National Weather Service.

The NWS as a whole provides a variety of weather product services, including: annual trainings and exercises; watches, warnings, and advisories; specialized briefings, emails, and consultations; on-site deployments and after-action event reviews; tailored weather products; and general information on the NWS website and social media accounts to both the general public and partner organizations. (The survey will target only partner organizations, discussed in the next paragraph.) The office, OCOO, specifically manages the day-to-day mission execution units responsible for delivering NWS weather, water, climate, and space weather products, services, and information. Among its many responsibilities, the OCOO also manages the budgetary planning for 11 National Service Programs, manages the NWS Operations Center (NWSOC), and provides liaison services to other partnering agencies. The OCOO is responsible for coordinating and integrating all aspects of mission execution to ensure consistency of NWS products and services, in addition to monitoring operational service efficiency, effectiveness, and customer satisfaction (which is what this survey aims to do).

Generally, the National Weather Service’s customers are both the general public as well as organizations that work with the NWS to acquire information and resources that help customer decision-making related to weather and water events. This survey will specifically target the organizations that work with the NWS (its core partners and deep relationship partners), not the general public. With these resources and information, NWS’s customer organizations prepare, disseminate, and discuss information put out by the NWS about hazardous weather or other incidents impacted by weather. These customers’ work pertains to legal and/or national security interest. They serve populations or entities particularly vulnerable to weather, and act as force multipliers to help amplify the NWS’s messages. Examples of major categories of customer groups include emergency response centers, school districts and universities, public health institutions, utility and transportation companies, federal/state/local government agencies, the military, and the media.

The services listed above in the third paragraph of Section A, Question 1 are provided to the customers through online and virtual channels, social media, traditional media (e.g., television and radio), and individual consultations and visits (virtually and in-person).

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

At the beginning of the survey development process, an IDSS team (composed of NWS field managers e.g., Meteorologists in Charge of Weather Forecast Offices) was created to brainstorm the goals of the customer experience survey, oversee the development process, and help analyze the results and implement the takeaways. The McKinsey team worked in tandem with the IDSS team to develop the survey. The goals of the survey are: understanding NWS’s customers’ underlying needs and overall satisfaction with NWS products and services, identifying NWS services most critical to IDSS, and optimizing customer service going forward so the NWS can better bring value to its customers. After outlining the goals, the team designed the structure of the survey and questions were drafted.

The McKinsey team consulted with the IDSS team and a McKinsey survey expert through the survey design process. The IDSS team was the main consultant during the development of the content of the survey (e.g., the structure of the survey and the questions included in the survey). A McKinsey survey expert was the main consultant during the development of the survey mechanics and statistics (e.g., how many questions the survey should include, how many respondents are sufficient to make sure the results are representative enough to allow them to shape understanding of the NWS customer experience).

The suggestions that were received about improving the survey related to the structure and the content. Structurally, the pathway of the survey was revised and shortened in order to minimize the number of questions that the participant has to answer while still and gathering enough information. For example, while a certain participant may qualify to answer 8 sets of questions, the survey will be curtailed to randomly select only a few sets so that the participant does not have to answer too many questions. Content-wise, there were a variety of suggestions about the depth of questions to include (e.g., whether to focus on general NWS satisfaction or whether to identify specific drivers of satisfaction and ask about those instead). Content suggestions also included how to categorize NWS products and services and present them in a clear way that is mutually exclusive, collectively exhaustive, and easy for participants to understand.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The survey will be administered to participants online. The link to the survey will be sent to participants via email from an outside vendor, the firm that will collect the participants’ responses: IntelliSurvey, Inc (see the last paragraph of Section B, Question 5 for more details about the firm). A cover note explaining the purpose of the survey will also be sent from an NWS email address in order to communicate the survey’s legitimacy. That cover note was submitted among the materials necessary for PRA approval.

The sampling approach was multifaceted. First, a random sample of NWS offices was selected to identify their customers. The offices were selected as follows:

1) A list of all NWS locations was generated including data on office type, marine vs. land-based coverage, and population density.

2) Locations were randomized to be representative across the above criteria to ensure at least two visits per region in continental U.S. and one visit per region in Pacific/Alaska.

3) A national perspective was taken to consider NCEP locations and other factors (such as type of weather events and general service needs). Test office and roadmap pilot offices integral to implementing new policies and programs were also added into the mix, in order to ensure that customers working with the most innovative programs were included.

The offices subsequently identified partners to receive the survey. Additional partners across major NWS partner categories were randomly selected from a master list of partners (collected through a previous internal questionnaire to round out the sample and ensure a diversity of responses).

It is estimated that 30% of customers asked to take the survey will respond.

In order to increase the response rate, the NWS will send out a notice beforehand (in the form of a cover note via an NWS email) letting the participants know to expect the survey, communicating its importance, and encouraging them to fill it out as authentically as possible. It will highlight the anonymity of the survey responses, the fact that the time the survey will take to complete will not be burdensome, and the legitimacy of the link participants will receive from IntelliSurvey. Furthermore, the IDSS team will ask the customers’ partner offices to reach out to their customers to encourage them to take the survey, in order for the NWS to gain constructive information that will help it improve its customers’ experience. Finally, IntelliSurvey will send reminder emails to participants who have not completed the survey.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The results of the customer experience survey will be analyzed in Excel to identify characteristics of the customer experience. Statistical rigor will not be required to analyze the results. However, the survey will be designed to be as robust as possible without being a burden on participants, which may result in some statistical analysis incremental to the bulk of the work. These analyses are a feature of this effort, but not the foundation; the core objective is to build the customer experience fact base. Any results will be used to improve NWS products and services to create a better customer experience.

The results of the survey will not be generalized to the entire customer population, in large part due to potential nonresponse bias. When relevant, simple t-tests may be used on a question-by-question basis as a means of estimating if differences in answers between different subgroups are genuinely different, and if so the question and the responses will be closely examined to determine how the information will help improve the NWS customer experience. This will help the NWS to understand how representative a certain takeaway may be of the customer experience, which will assist the NWS in determining how to make the overall experience better, (but no claims about the general population will be made).

This survey is not intended to measure a GPRA performance measure.

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The estimate of the potential respondent universe is 2000 respondents total (see estimates in the table below). As described in Section A Question 3, after generating a list of potential customers by working with a selection of NWS field offices representative of all offices, the customers of these random offices will be selected, along with additional partners (randomly selected) across major NWS partner categories to ensure a representative group of participants and objectivity of responses. In short, a subset of customers will be randomly selected from a list of all known customers (identified by NWS offices).

Category of Respondent	No. of Respondents	Participation Time	Burden	Expected Burden based on 30% response rate
State, local, or tribal governments	1800	15 minutes	450 hours	135
Federal government	200	15 minutes	50 hours	15
Private sector	0	N/A	N/A	N/A
Totals	2000	15 minutes	436 hours	150

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

In terms of the statistical methodology for stratification and sample selection, this methodology is outlined in Section A Question 3. The sample pool is composed of two groups: one, a selection of partners identified by a random and representative selection of NWS offices, and two, a selection of additional partners randomly selected from a master list of NWS partners.

As is outlined in Section A Question 4, the results of the survey will not be generalized to the entire customer population, primarily due to potential nonresponse bias. When relevant, simple t-tests may be used on a question-by-question basis as a means of estimating if differences in answers between different subgroups are genuinely different, and if so the question and the responses will be closely examined to determine how the information will help improve the NWS customer experience. This will assist the NWS in determining how to make the overall experience better, helping the NWS to understand how representative a certain takeaway may be of the customer experience, but no claims about the general population will be made.

There are not any unusual problems requiring specialized sampling procedures, and there will not be any use of periodic data collection cycles to reduce burden.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield “reliable” data that can be generalized to the universe studied.

As mentioned in Section A Question 3, in order to maximize response rate a series of customers’ partner offices will be encouraged to reach out to their customers to encourage them to take the survey, in order for the NWS to gain constructive information and better serve its customers. Additionally, the customers will be informed through official NWS channels that the survey will be sent to them before the link is shared via a cover note sent from an NWS email, so that customers know the link is coming and the survey is legitimate and do not ignore the survey out of skepticism. Finally, IntelliSurvey will send reminder emails to participants who have not completed the survey.

Due to the random sampling technique and the large sample size, the information will be accurate and reliable enough for the intended use, to better understand the NWS customer experience so the NWS can understand what is important to its customers and improve its processes in order to better serve them, not make claims about the general population. The collection of data will be reliable and will allow for a deeper understanding of customers, specifically core and deep relationship partners as discussed in Section A Question 1.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The survey was tested internally within the IDSS and McKinsey team. Only a few customers were involved with the testing (fewer than ten respondents). The IDSS and McKinsey team was involved with testing content, structure, and survey design logistics, and the handful of customers were involved with testing terminology choices, inclusivity of customer types, and clarity of questions.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals consulted on the statistical aspects of the design: Kevin Nuffer, a McKinsey survey expert; (404) 335-3491; Kevin_Nuffer@mckinsey.com

Who will collect the information for the agency: IntelliSurvey, Inc (signed an NDA); John Hopkins; (503) 208-7483; jhopkins@intellisurvey.com

Who will analyze the information for the agency: McKinsey & Company (contractor); Sarah Ray; (901) 497-6360; Sarah_Ray@mckinsey.com