Supplemental and Part B Questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The proposed information collection will be conducted by a team of Human Dimensions Program staff of the NOAA Coastal Services Center to evaluate the effectiveness of the National Weather Service (NWS)-issued Hurricane Local Statement (HLS) and the accompanying Tropical Cyclone VTEC (TCV) product, which contains all of the long-fused (greater than 6 hours) watches and warnings in effect relating to a given tropical cyclone.

The Coastal Services Center serves the needs of coastal and natural resource management programs and professionals (e.g., state natural resource management agencies and staff, conservation organization staff, as well as colleagues across NOAA line offices) through development and delivery of data and information products, decision-support tools, professional development training, evaluation, and technical assistance on a variety of topics. Data and information products and decision support tools are delivered per customer requests via online systems (e.g., clearing house, direct download). Professional development training is offered in three distinct areas: geospatial technology, coastal issues, and process skills. Delivery methods include face-to-face instructor-led training held at the Center and local host sites, self-guided web-based training, and instructor-led web-based training. Evaluation and technical assistance services are conducted electronically, or remotely (on-site) depending on the needs of the specific customer and the specific topic.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

The proposed research was planned in close consultation with the Charleston, SC NWS Weather Forecast Office (WFO), the NWS HLS working group and NWS headquarters-based social science staff. The proposed interview methodology was piloted (fewer than ten individuals) and evaluated for elements including question comprehension, relative ease of completion, and estimated response time. The interview questions for the proposed study were developed by reviewing the list of cleared questions for the Generic Clearance, OMB Control No. 0648-0342, and through consultation with Charleston WFO staff. They identified areas in which the interview questions possessed certain limitations in question clarity, order, variable measurement, and ability for respondents to meaningfully respond to question items. Thoughtful effort has been made to keep the proposed list of questions as brief as possible, inquire only upon tangible actions, and to ensure relevance to the primary evaluation user audience, which are the NWS WFOs.

Further, the Coastal Services Center research team possesses extensive expertise in the areas of customer interview, survey design and administration, focus group administration, and product evaluation and has closely collaborated in producing the proposed interview question list.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The information collection will be administered via customer interviews. Since this work is based upon evaluating a tool used for extreme weather events, interviews will take place only in a selection of geographies where this hazard type is prevalent. The research team will receive assistance from the NWS WFOs responsible for disseminating HLS information. Assistance will come in the form of providing the contact information for the target audience/potential interviewees.

The number of responses and content will be recorded. It will be possible to calculate an actual response rate for reporting purposes. The number of questions has been made as brief as possible in an effort to maximize response and participation. The intended sample size is 50. Based on previous interactions with comparable audiences, the expected response rate is expected to be relatively high (approximately 80%).

A stratified random sample will be used to select individuals from each of the three target audience sub-categories: broadcast meteorologists, regional and local emergency managers, and members of the public that are informed on NWS-related information and products. Because the target population possesses great diversity in professional roles and responsibilities, it is important to select those individuals that have a knowledge of, and familiarity with the HLS product. Only individuals that have such credentials will be targeted for interviewing. The purpose of the interviews is to generate discussion around the actual HLS message content in order to ascertain which components are understandable, actionable, meaningful, and useful. Such data shall provide rich insight into making improvements to the HLS and accompanying VTC products.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Analysis of interview data will be undertaken through basic content analysis using open coding. This information collection seeks to assess the utility of content and the format/structure of the HLS and VTC. The general public audience subcomponent of the respondent universe is large and somewhat undefined. The research team will rely heavily on NWS Weather Forecast Office contacts within their regions to aid in identifying individuals that are familiar with technical NWS products, such as the HLS. The sample will be representative of members of the NWS product-informed public, but not necessarily representative of all Americans. This will be clearly articulated in subsequent reporting of the results. However, the sample will be much more representative of broadcast meteorologists and local emergency managers since the population is significantly smaller and very well defined.

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The population for the resource being evaluated consists of members of the public that are informed on NWS-related information and products (designated below as general public), regional and local emergency managers (designated below as local government), and broadcast meteorologists (designated below as private industry). Following is the breakdown of the population of interest.

Population (Anticipate an 80% response rate)	% of Participants (total 100%) (50 individuals solicited)
General Public	40% (target: 20, estimated responses:16)
Local Government	20% (target: 10, estimated responses:8 )
Broadcast Media	40% (target: 20, estimated responses:16)

The interviews will be administered via telephone. Questions will inquire on comprehension and effectiveness of information shared through HLS and TCV communications. The estimated time necessary for each respondent to complete the interview process is 60 minutes, based on trials with a small (less than ten) pilot sample. Total estimated public burden associated with this information collection is 40 hours (40 interviews @ 1 hour per response).

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Information will be collected via telephone interviews. A stratified sampling process will be employed where all individuals will first be placed into a specific subgroup (NWS-informed public, broadcast meteorologists, regional/local emergency managers. Once subgroups have been created, relevant percentages of individuals (40%, 40%, and 20% respectively) will be randomly selected from NWS contact lists to the interview sample. The NWS forecast office staff will provide assistance in the participant sampling process, based on their intimate knowledge of local emergency management and broadcast media staff in their local areas. While the small N will likely not ensure an entirely representative sample, steps are being taken to ensure the sample is as representative as possible. Additionally, it is not the intent of such research to make generalizations across the entire population, but rather, to obtain targeted feedback to improve a targeted, highly technical product of the NWS. Interview data will be stored on a password-protected work space at the Coastal Services Center, accessible only by staff particular to this project. Respondents will be instructed not to provide identifying information in their interviews (names, social security numbers, dates of birth, etc.), and any identifying information provided will not be recorded by interviewers.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The intent of this information collection is to assess user feedback on the HLS and accompanying VTC products. The intended approach will yield an informed sample of the respondent universe – and the information gained will be extremely valuable in making severe weather communication improvements. This interviewee sampling process allows equal and independent opportunity for the target population to provide feedback on the HLS and VTC.

In order to improve response rates for this information collection, the interview process has been made as concise as possible. We do not really anticipate nonresponse bias in the local government and broadcast media, as these groups are relatively homogenous. We do not have a way to address any nonresponse bias in the general public subsample. However, again, we will value all feedback received.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Pilot interviews were administered and received comment to less than nine representative members of the target population. Reviewers were asked to offer feedback on the length, appropriateness and clarity of questions, content, or other aspects to improve the interview process. Comments from reviewers were helpful and resulted in design, and content changes to clarify questions and simplify instructions.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The implementation of the information collection and data analysis will be coordinated by Dr.

Chris Ellis at the NOAA Coastal Services Center, available by telephone at (843) 740-1195 or by email at Chris.Ellis@noaa.gov.