NWS IDSS updated survey

Supplemental Questions for DOC/NOAA Customer Survey Clearance (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This collection request is based on a preapproved request dated July 16, 2018 for the Measuring NWS' Impact Decision Support Services. This request is for adding additional items to the questionnaire. The new items constitute a validated Trust in Physician Scale which more adequately measures the construct of trust (see items #12-22 of the attached questionaire).

The proposed information collection will be led by Dr. Vankita Brown, Michael Scotten and Douglas Young of the National Weather Service (NWS), supported by a team of atmospheric scientists representing various regions in NOAA. The program offices responsible for the proposed information collection are the Operations Division and Performance and Evaluation Branch in the Office of the Chief Operating Officer (OCOO) and Digital and Graphical Information Support in the Analyze, Forecast, and Support Office (AFSO).

NWS provides Impact Decision Support Services (IDSS) to core partners when weather, water, or climate has a direct impact on the protection of lives and livelihoods. IDSS is defined as the provision of relevant information and interpretive services to enable core partners’ decisions.

IDSS may be characterized as either episode or recurring. Episodic IDSS is support provided to core partners for a particular event/incident (ex. Webinars, NWSchat, on-site deployment). Recurring IDSS is ongoing support provided to core partners throughout the year to improve partner mitigation, preparation, response, and recovery related to event/incidents, or to support routine-high value decisions. (ex. Joint training, scenario planning, table-top exercises, and daily coordination regarding routine high-value decisions).

Core partner is defined as government and non-government entities who are directly involved in the preparation, dissemination and discussions involving weather, water, or climate related NWS information that supports decision making for routine or episodic, high impact events. Such partners include emergency managers, media, public officials, and federal/state/local government.

Feedback from core partners on these specialized services is needed in order to evaluate NWS’ performance for recurring and significant events to improve services to the nation.

The OCOO’s Operations Division is responsible for providing Headquarters-level support to the entire field structure through the provision of objective and impact-based performance metrics, monitoring forecast and warning consistency, facilitating communications between senior leadership, core Federal partners and the field and focusing on Continuity of Operations (COOP). Specifically, the Division:

• provides oversight and insight to NWS service performance and ensures NWS is responsive by delivering products and information that are useful to customers and partners

• oversees NWS service assessments and their recommendations as many affect NWS partners in the delivery of services including the media, local officials, emergency managers, and weather and climate enterprise

• ensures optimal resources are available to cover the HQ-based NWS Operations Center as well as liaison support to partnering Federal agencies

• ensures that training requirements for both operations and service assessments feed into the Office of the Chief Learning Officer

The Performance and Evaluation Branch provides the foundation for all NWS service improvement activities by measuring, analyzing and reporting operational performance and assessing customer satisfaction with service delivery. The Branch provides leadership and establishes policy for the verification, service assessment, customer satisfaction and forensic services programs. Specifically, the Performance and Evaluation Branch:

• develops the requirements for a robust verification system and oversees the system’s development and maintenance

• uses teams established at each NWS service delivery point to assess needed improvements

• uses assessment findings to recommend national changes to improve service programs

• organizes and deploys teams to assess the level of service performance and make recommendations for increased effectiveness

• creates customer satisfaction indices and assists service program managers in translating results into service improvements

• establishes policy for archiving NWS data and products that are used by the public, weather sensitive economic sectors, and academia for both research and legal purposes

• maintains the NWS natural hazards database - the only official repository for natural hazard statistics - used by agencies such as FEMA, the EPA, USGS, and organizations such as the Institute for Business and Home Safety to suggest mitigation strategies

AFOS’s Digital and Graphical Information Support Branch provides national oversight of services-based requirements for NWS´s web presence, oversees product and information dissemination policy, and provides programmatic execution of the national service change process. Activities within these categories include:

• coordination with NWS´s National Internet Dissemination System (NIDS) unit to ensure critical web-based requirements are developed and deployed

• national coordination of new grids for inclusion in the NDFD and National Digital Guidance Database (NDGD)

• coordination with NWS´s Science and Technology Integration (STI) Portfolio to develop new products and services

• oversight of associated grid collaboration thresholds to assure national consistency of grids

• providing leadership of policy development with respect to dissemination of NWS warnings and forecasts via emerging communication modalities (including social media and personal electronic assistants)

• advancement of GIS capabilities and requirements to promote the delivery of geospatial information to the public and NWS partners.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The proposed research was planned in close consultation with a team of NWS meteorologists and social scientists with extensive expertise in the full array of weather event types. A high level of effort was expended to propose a list of questions that were informed by the cleared questions from the NOAA Generic Clearance, OMB Control No. 0648-0342. The NWS survey team identified areas in which the survey questions possessed certain limitations in question clarity, order, variable measurement, and ability for participants to meaningfully respond to question items. Thoughtful effort has been made to keep the proposed surveys as brief as possible, inquire only upon tangible actions, and to ensure relevance to the primary services offered.

The proposed survey methodology was pilot tested and evaluated with a small sample of the target audience for elements including format, comprehension, readability, ease of completion, and estimated response time. Additionally, pilot participants were also interviewed to further assess the instrument to refine clarity, and identify additional concepts to test. The pilot resulted in changes to format such as numbering each question, as well as the elimination and replacement of one item. The instrument was also tested for reliability using reliability statistics. Cronbach’s alpha showed the instrument reached acceptable reliability at ⍺ = .75. A score of over .7 indicates high internal consistency, and that the instrument is reliable. Cronbach’s alpha analysis of each question also showed that questions included in the instrument were worthy of retention, resulting in a significant decrease in the alpha if deleted.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Participants of this data collection represent a very discreet population - emergency managers, media, public officials, and federal/state/local government partners who work closely with Weather Forecast Offices (WFO) throughout the six NWS regions. Since this work is based on evaluating IDSS to these targeted partners, forecasters will disseminate the electronic questionnaire (informed consent will be requested before respondents can advance to survey questions; see survey introduction for informed consent statement) to participants within their respective County Warning Area (CWA) only after hazardous weather events (i.e., tornadoes, hurricanes, floods, etc). This same approach was used during the pilot testing of the instrument and based on initial responses, we anticipate a 60% response rate from participants.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of survey data will be undertaken through basic descriptive statistics only (e.g., percent, mean scores). This information collection seeks to assess the quality of IDSS, and to better serve the information needs of our core partners. The respondent universe will include a sampling of our core partners. A core partner is a “government or nongovernment entities directly involved in the preparation, dissemination and discussions involving weather, water, or climate related National Weather Service information, that supports decision making for routine or episodic, high impact events.” Core partners include emergency managers, media, public officials, and federal/state/local government. For example, EM for schools, hospitals, or utilities. This audience is clearly defined and the sample will be representative of members of the community/region. The research team will rely heavily on NWS WFOs within their regions to aid in identifying core partners who have received IDSS.

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Purposive sampling method will be used, as this will be a close ended survey to target only core partners who receive IDSS.  Thus, the people most likely to take the survey are recipients of NWS IDSS.

The potential respondent universe is 12,220 core partners. We estimate a 60% response rate or 7,320 total responses ranging from our core partners (emergency managers, public officials, and federal/local/state government partners). Of the 7320 responses we are estimating, we expect that 100% will be from NWS partners.

 

The team will utilize the same questions and will be testing IDSS services received after multiple hazards throughout the year (Winter Weather, Flooding, Extreme Heat, and Severe Thunderstorms).

The estimated time necessary for each respondent to complete the survey is 8 minutes, based on trials with a small pilot sample. Total estimated public burden associated with this information collection is 976 hours (7320 surveys @ 8 minutes per response).

Population	% of Participants (total 100%)
Core Partners (Anticipate an 60% response rate)	(12200 individuals solicited) 100% (target: 12200, estimated responses: 7320)

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The information collection will be administered via an electronic questionnaire. Questions will inquire (post event) upon the effectiveness of received IDSS.

A purposive sampling strategy, based on forecasters at NWS WFOs, will be instituted to access respondents. Staff at the forecast offices are the most connected to the targeted population, thus utilizing these existing connections to disseminate the survey is the most appropriate sampling approach. WFO staff will email the questionnaire to their respective partners after IDSS has been provided.

Based on the target population of 12,200 respondents, a 95% confidence interval level, and 5% margin of error, the calculated sample size is 373 responses to yield a representative sample of the target population. Our estimated response rate at 60% yields 7,320 responses, which significantly exceeds the 373 calculated sample size.

Survey data will be stored on a password-protected work space at NWS headquarters, accessible only by staff particular to this project. Respondents will be instructed not to provide identifying information in their interviews (names, social security numbers, dates of birth, etc.), and any identifying information provided will not be recorded by survey administrators.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The intent of this information collection is to assess partner feedback on NWS IDSS. The intended approach will yield an informed sample of the respondent universe – and the information gained will be extremely valuable in making improvements to NWS services. The survey sampling process for this collection request is appropriate for the target population, and the proposed plan provides the best opportunity for the target population to provide feedback related to NWS customer satisfaction. In order to improve response rates for this information collection, the survey process has been made as concise as possible. Moreover, the survey design and approach taken will yield reliable data that can be generalized to the universe studied.  

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Pilot surveys were administered and received comment to less than ten representative members of the target population. Reviewers were asked to offer feedback on the length, appropriateness and clarity of questions, content, or other aspects to improve the survey process. Comments from reviewers were quite helpful and resulted in design, and content changes to clarify questions and include additional concepts to test.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The primary point of contact for this information collection request is Dr. Vankita Brown at the NOAA National Weather Service, available by telephone at (301) 427.9338 or by email at vankita.brown@noaa.gov. Dr. Leticia Williams, NOAA National Center for Atmospheric Science and Meteorology, will also analyze data. Dr. Williams can be reached at (301) 427-9308.