Supplelmental and Part B Questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

NOAA’s National Weather Service (NWS) will be conducting the survey. Integrated Water Resources Science and Services (IWRSS) is supported by a consortium of federal agencies with complementary missions in water science, observation, management and prediction: The U.S. Geological Survey (USGS), U.S. Army Corps of Engineers (USACE), and NWS. The objective of IWRSS is to design, develop, and implement a national water modelling and information services framework to infuse new hydrologic science into current water resources management, develop hydrologic techniques and decision support applications for operational use, and provide advanced hydrologic services to address growing stakeholder needs. Toward this end, IWRSS applies a cross-cutting, multi-disciplinary systems approach to address complex water problems collaboratively.

Planned IWRSS services include: a) high spatial and temporal resolution “summit to sea” analyses and forecasts for full spectrum of water budget parameters; b) short- through extended range river forecasts which quantify uncertainty; c) static flood inundation map libraries and real-time flood forecast inundation mapping services which show the areal extent and depth of flood waters; d) linking river forecasts, and associated flood inundation maps, to potential socioeconomic impacts; and e) integrating the access to geospatial water resources information from multiple federal agencies through a single portal.

IWRSS stakeholders include government and private sector water resources decision makers whose interests include hydropower, emergency management, reservoir management, watershed management, agriculture, fish and wildlife, river commerce, municipal and industrial water supply, recreation, and water quality.

The IWRSS agencies are developing a plan to demonstrate new IWRSS capabilities in the Ohio River and Russian River basins. The customers of this demonstration plan include river basin commissions and their water resources management stakeholders. This plan is an extension of the stakeholder demonstration efforts in progress in the Potomac, Delaware, Susquehanna, and Hudson River basins. As a result of the successful implementation of the project in the mid-Atlantic river basins, NWS is seeking approval to conduct a survey of IWRSS stakeholders in the Ohio and Russian River basins. The survey effort for the Potomac, Delaware, Susquehanna, and Hudson River basins was approved under OMB Control No. 0648-0342 on 3/4/13. ¹

NWS is proposing to use the same survey instrument, but with revisions to reduce respondent burden by focusing on the questions that yielded the most useful information from the original version of the survey. The objectives of the current survey are: a) validate existing, and identify new, gaps in water resource services, b) identify priority issues for decision making, c) identify gaps in information needed for decision making, and d) obtain stakeholder perceptions of the importance of needed information.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

NWS consulted with Eastern Research Group, Inc. (ERG) on the development of the survey instrument. ERG developed and implemented the initial surveys of river basins in the mid-Atlantic region and has significant experience assessing technical assistance provided by Federal agencies through detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The survey development process was informed by reviewing the previous survey instrument, reviewing relevant background materials on the availability of water resources information, documentation of previous stakeholder engagements on this topic, and other previous surveys of user needs, preferences, and satisfaction.

The previous survey instrument provided detailed information on respondent priorities, information needs and gaps but proved to be too long and detailed for some respondents. NWS and ERG reviewed the survey instrument and identified the questions most subject to high non-response and data quality issues; these questions tended to be open-ended questions requesting the respondent to describe the nature and magnitude of the expected economic value of new information. While this is important information for the stakeholder engagement and demonstration plan, it is not easily obtained through a survey instrument. These questions were eliminated from the revised survey instrument and the information will be obtained through structured interviews with individuals with the experience and job responsibilities to respond effectively. These interviews are not included in this approval request because they will be limited to nine or fewer non-federal respondents.

The revisions to the survey instrument reduced response time from 45 minutes to 25 minutes.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

As a key initial step in the stakeholder engagement process, NWS, with ERG’s support, is compiling a list of stakeholders in each of the river basins (the Ohio River and Russian River basins). In the Russian River watershed, the Sonoma Water Agency will be the primary source of stakeholder lists; in the Ohio River Basin, the Ohio River Valley Water Sanitation Commission (ORSANCO) will be the primary source of stakeholder lists. These organizations are responsible for coordinating water resources management for the watershed across political boundaries. Each stakeholder in the Russian River watershed will be categorized into one of the 13 sectors identified as a potential IWRSS customer: (1) water quality, (2) fish and wildlife, (3) emergency management, (4) reservoir management, (5) watershed management, (6) agriculture,

(7) hydropower, (8) other energy extraction (e.g., hydraulic fracking), (9) river commerce, (10) municipal and industrial water supply, (11) recreation, (12) education, and (13) legislative. Each stakeholder in the Ohio River watershed will be categorized into one of the 13 sectors identified as a potential IWRSS customer: (1) water quality, (2) fish and wildlife, (3) emergency management, (4) reservoir management, (5) watershed management, (6) agriculture, (7) hydropower, (8) other energy extraction (e.g., hydraulic fracking), (9) river commerce, (10) municipal and industrial water supply, (11) recreation (12) insurance, and (13) flood protection.

No statistical methods are being used in the survey; all individuals in the stakeholder listing will receive a survey. NWS expects that 80% of the recipients will respond to the survey because these tend to be individuals who rely on water resources information to inform decision making as a part of their profession. These decisions include routine decisions that can have significant financial value and event-driven (e.g., flood-related) decisions which are critically dependant on accurate and timely water resources information. The previous survey effort in the mid-Atlantic region had response rates of 51 to 58 percent of respondents who clicked on the survey link and submitted a completed survey; another 23 to 27 percent of respondents provided incomplete responses. The revised survey removes several question items that had high item non-response, so NWS anticipates a higher response rate.

NWS will perform the survey using a Web-based survey instrument. NWS has chosen a Web-based instrument for two reasons. First, the water resources information we are asking about is primarily provided to the customers via the internet in the form of GIS, formatted data sets, graphs, and other visualization tools. For this reason, the survey will also be made available to interested members of the public via a Web link posted on the websites of the Sonoma Water Agency and ORSANCO. Second, the program has significant interaction with stakeholders over the internet (e.g., via email, on Web forums, previous surveys, etc.) and so a Web-based survey represents a logical mode for implementing the survey.

NWS expects that a high response rate is achievable; however, NWS and ERG will continue to follow good survey practices to ensure high participation, including the following:

• NOAA regional staff will send the potential respondents a pre-notification email to inform them of the upcoming survey.

• ERG will send the email with the survey link 3-4 days after the pre-notification email.

• ERG will send 2 reminders to non-responders one and two weeks following the email with the survey link.

Additionally, a Web link to the survey will be placed on the website of the Sonoma Water Agency and ORSANCO to encourage participation from other interested customers, such as private citizens, who do not appear on the list of stakeholders for the basin.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

NWS is not using any statistical methods to select from the population and will select all population members in the survey.

NWS will use the data collected under this effort to prioritize water resources information needs in the two river basins based on the importance of the information to decision making. Additionally, the prioritized information needs will be used to inform the design of a demonstration project plan for the river basins. NWS will have ERG summarize the results on information needs within and across river basins to look for trends and similarities between river basins and sectors.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe includes at least 200 individuals who are listed as stakeholders by Sonoma Water Agency and ORSANCO (i.e., approximately 100 individuals per river basin), plus a number of interested individuals who regularly access the websites of the Sonoma Water Agency and ORSANCO, see Table 1. Each organization will compile a list of stakeholders. The survey will not sample from the list of stakeholders; the entire list of will receive a survey and it will also be open to interested individuals visiting the organization’s Web sites.

Table 1. Summary of potential respondent universe, response rate, and expected number of respondents by river basin and sector.

Description	Potential Universe		Response Rate	Actual Respondents
	Percent [a]	Number [b]
Both River Basins	100%	250	80%	200
Russian River Basin		125	80%	100
Water quality	20%	25	80%	20
Fish and wildlife	10%	13	80%	10
Emergency management	15%	19	80%	15
Reservoir management	10%	13	80%	10
Watershed management	15%	19	80%	15
Agriculture	2%	3	80%	2
Hydropower	2%	3	80%	2
Other energy extraction (e.g., hydraulic fracking)	1%	1	80%	1
River commerce	5%	6	80%	5
Municipal and industrial water supply	10%	13	80%	10
Recreation	5%	6	80%	5
Education	3%	4	80%	3
Legislative	2%	3	80%	2
Ohio River Basin		125	80%	100
Water quality	20%	25	80%	20
Fish and wildlife	10%	13	80%	10
Emergency management	15%	19	80%	15
Reservoir management	8%	10	80%	8
Watershed management	15%	19	80%	15
Agriculture	5%	6	80%	5
Hydropower	2%	3	80%	2
Other energy extraction (e.g., hydraulic fracking)	2%	3	80%	2
River commerce	5%	6	80%	5
Municipal and industrial water supply	10%	13	80%	10
Recreation	5%	6	80%	5
Insurance	1%	1	80%	1
Flood protection	2%	3	80%	2
[a] The estimated distribution of respondents across the sectors is based on similar distributions from the previous survey of the Potomac, Susquehanna, Hudson, and Delaware river basins. [b] Some columns might not sum due to rounding.

NWS expects that 80 percent of the recipients will respond to the survey since these tend to be individuals that have a strong interest in information about their river basin and rely on NWS and the other IWRSS partner agencies for water resources information. The previous survey effort in the mid-Atlantic region had response rates of 51 to 58 percent of respondents who clicked on the survey link submitted a completed survey; another 23 to 27 percent of respondents provided incomplete responses. The revised survey removes several questions that had high item non-response, so NWS anticipates a higher response rate.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

An email invitation to participate in the survey with a Web link to the survey instrument will be sent to all individuals in the list stakeholders in the two river basins, and a Web link to the survey will be placed on the website of each organization (Sonoma Water Agency and ORSANCO) with a brief announcement explaining its purpose. No sampling will be performed in selecting the sample, so NWS has not developed a statistical methodology for stratification and sample selection. Additionally, no estimation procedure or degree of accuracy is needed, as no sampling procedures are being employed.

This will be a one-time data collection for the Ohio River and Russian River basins. There is no need for periodic collections.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

NWS expects that a high response rate is achievable; however, NWS and ERG will continue to follow good survey practices, including the following:

• NOAA regional staff will send the potential respondents a pre-notification email to inform them of the upcoming survey.

• ERG will send the email with the survey link 3-4 days after the pre-notification email.

• ERG will send 2 reminders to non-responders one and two weeks following the email with the survey link.

As discussed above, ERG, on NWS’s behalf, will administer the survey to the full population of stakeholders of the Sonoma Water Agency and ORSANCO; no statistical methods will be used.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

NWS contracted with ERG to develop the survey instrument. ERG has developed a number of surveys related to assessing customer satisfaction with assistance. Additionally, as part of ERG’s work, ERG is coordinating stakeholder meetings in each of the river basins.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA to design the survey instrument, implement the survey and analyze the results. NWS’s lead for this project is Mary Mullusky, who will be providing oversight of the contractor (301-713-0006 x169; Mary.Mullusky@noaa.gov). ERG’s project manager for this work is Arleen O’Donnell (781-674-7220; Arleen.odonnell@erg.com); ERG’s task lead for survey development is Dr. Lou Nadeau (781-674-7316; lou.nadeau@erg.com).

1 Stakeholder Engagement to Demonstrate IWRSS for RBC in the Mid-Atlantic. Application for DOC/NOAA Customer Survey Clearance under OMB Control Number 0648-0342.

Stakeholder Engagement to Demonstrate IWRSS for RBCs 12