0648-WxGov Supporting Statement Part B

SUPPORTING STATEMENT

U.S. Department of Commerce

National Oceanic & Atmospheric Administration

Weather.gov Visitor Experience Survey

OMB Control No. 0648-XXXX

SUPPORTING STATEMENT PART B

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The survey will cover 2 types of target audience:

• General population of the US 18+ with access to the internet (access to internet is required since the study is about website improvements). We will call this group “Gen Pop” moving forward. This group includes the following sub-groups of interest that represent vulnerable communities dependent on timely, convenient access to weather information:

  • Native Americans,

  • Rural population, and

  • Spanish speakers with limited English proficiency* (we will call this group “Spanish speakers” moving forward).

• NWS partners who use NWS forecast and decision support information to make risk assessments and risk management decisions on behalf of society when hazardous weather threatens (we will call this group “NWS partners” moving forward).

*2 language versions will be offered to respondents to pick from: 1) English version; 2) Spanish version

Sampling/Respondent Selection Method:

• Gen Pop – A non-probabilistic quota sampling approach will be used to ensure representativeness of data obtained through online panels. Quotas on age and gender will be set and further weighting by region and ethnicity may be applied if needed. There will be light weighting on age groups in case we run into feasibility issues with older age groups to ensure this data are representative of the population of the US 18+ with access to internet. Data collection will start from the Gen pop approach (i.e., sending invitations to online panelists without any particular targeting, in accordance with ESOMAR standards), and will continue with a targeted approach to cover hard-to-reach demographic groups. This data collection will be supplemented by the following (also obtained using targeted approach):

  • An oversample of Native Americans (self-defined in the survey)

  • An oversample of Rural population (determined by zip code)

  • An oversample of Spanish speakers (self-defined in the survey)

• NWS partners – There is no sampling strategy per se because we are reaching out to all our partners. NWS partners include entities across the weather enterprise such as Broadcast meteorologists, Emergency Management community representatives, and local and regional officials, with whom we are in frequent contact to ensure there is ongoing communication to make critical decisions before, during and after significant weather events. The NWS Weather Forecast Offices each have specific partner mailing lists which will be used to reach the partners. Those partners requested to be on the mailing lists and voluntarily provided their contact information.

General Universe – approximate size:

• Gen Pop: 236,000,000

• Native Americans: 1,400,000

• Rural population: 42,300,000

• Spanish speakers (with limited English proficiency): 10,000,000

• NWS partners: 120,000 (The NWS Weather Forecast Offices each have specific partners mailing lists which will be used to reach the partners. Those partners requested to be on the mailing lists and voluntarily provided their contact information)

Sample size:

• Gen Pop: n=2,000 (the total n size is determined based on the expected data cuts needed for the analysis, where each sub-group would need to have ideally not less than 350 respondents to ensure statistical significance of comparisons at the 95% confidence level)

  • Oversample for Native Americans: n=250 (determined based on the maximum feasibility of online panels for this group (given the small Universe of Native Americans) and the minimum n required to ensure accuracy at the 90% confidence level (~n= 270, which will be achieved by combining the oversample with Native Americans in the Gen Pop group)

  • Oversample for Rural population: n=250 (determined based on a) the maximum feasibility of online panels for this group, b) the expected number respondents that are from the rural population in the general universe (n=~350 that, together with this oversample, will provide a total n of ~600, with some share of this group (on best effort basis) covering coastal areas and other areas of interest; the total n=~600 will allow for a more granular analysis with various data cuts by region)

  • Oversample for Spanish speakers: n=250 (determined based on the expected number respondents that are Spanish speakers (~85) obtained from the Gen Pop sample that, together with this oversample, will provide a total of ~335 Spanish speakers, which will ensure statistical significance of the results for this group)

The total number of completed interviews expected from online panelists is 2,750 including the oversample.

• NWS partners: No sampling will be used for NWS partners (all 120,000 partners will be contacted). The expected response rate based on prior studies is ~10%, which means that the expected number of responses is ~12,000.

Sources: US Census data, American Indian Policy Institute, Statista, 2020-2021

Quotas:

Gen Pop:

Nested gender-age quotas:

	Male	Female
18-24	6.6%	6.2%
25-29	5.0%	4.8%
30-34	4.6%	4.5%
35-39	4.5%	4.4%
40-44	4.1%	4.1%
45-49	4.3%	4.3%
50-54	4.2%	4.3%
55-59	4.3%	4.5%
60-64	4.0%	4.4%
65+	8.0%	9.0%
Total	49.5%	50.5%

Source: US Census data on age-gender distribution + Statista data on internet penetration by age groups, 2021

Target distribution (+/- 2%) (Source: Census 2021 data):

By regions:

Northeast	17%
Midwest	21%
South	38%
West	24%

By ethnicity:

Asian	6%
Black/African/Caribbean	14%
Caucasian	55%
Hispanic, Latino or Spanish origin	19%
American Indian or Alaska Native	1%
Middle Eastern or North African	4%
Native Hawaiian or Other Pacific Islander	0.30%
Not Listed/Prefer not to answer	0.7%

Oversamples of Native Americans, rural population, and Spanish speakers: No quotas (if any significant bias is found once data collection is over, weighting will be used to ensure oversample replicates the distribution by gender and age of the respective group within the Gen Pop sample)

NWS partners: No quotas will be set. We estimate a conservative response rate of 10%, which translates to approximately 12,000 responses across all partner groups.

Expected response rate:

• Gen Pop sample, as well as the oversamples of Native Americans, rural population, and Spanish speakers, will be collected through online panels. The response rate will be relatively high since respondents will be incentivized to participate. It is difficult to calculate the actual response rate due to the process used by online panels, where potential respondents are sent generic invitations to log into the portal and are then shown the full list of studies they may qualify for based on information in their profile. Reminders will be used to increase the response rate; however, they won’t be specific to this study (generic reminders inviting respondents to the portal will be used). Based on previous studies, we estimate the expected response rate to be approximately 40%%, which results in 2,750 expected number of respondents.

• NWS partners sample – Based on prior experience with a similar target audience, we expect a conservative response rate of ~10%, since respondents won’t be incentivized. Reminders will be used to maximize the response rate. Hence, the expected number of respondents is 12,000.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Gen Pop sample and the oversample for Native Americans, rural population, and Spanish speakers:

Statistical methodology for stratification and sample selection

Generic invitations to visit the panel provider’s portal with a list of available surveys (where respondents will choose which surveys to participate in) will be sent out to all panelists (regardless of their demographic group) in a few rounds/batches, until demographic quotas for this survey start filling up. Once certain quotas are filled, new survey entrants from those demographic groups will screen out. Once certain demographic quotas start falling behind, the panel provider will switch to “targeting” based on demographic information they already have about their panelists. We believe this approach can be called “quota sampling” since we will set demographic quotas to ensure representativeness of the data, and since we will use open recruitment online panels (i.e., this is not a probabilistic approach).

Estimation procedure

A non-probabilistic quota sampling approach will be used to ensure representativeness of data obtained through the online panels, with set quotas on age and gender will be set (and further weighting by region and ethnicity may be applied, if needed), as well as light weighting on age groups in case we run into feasibility issues with older age groups to ensure this sample data is representative of the population of the US 18+ with access to internet. Data collection will start from the Gen Pop approach (i.e., sending invitations to online panelists without any particular targeting, in accordance with ESOMAR standards), and will continue with a targeted approach to cover hard-to-reach demographic groups. This data collection willcollectionsample will be supplemented by the following (also obtained using targeted approach):

• An oversample of Native Americans (self-defined in the survey)

• An oversample of Rural population (determined by zip code)

• An oversample of Spanish speakers (self-defined in the survey)

Degree of accuracy needed for the purpose described in the justification
The total n takes into account survey logic when certain questions may be asked on a reduced base, as well as the need to analyze multiple data cuts. Data cuts with the minimum size of any single subgroup of n=~350 will allow us to run statistical tests to analyze if differences are statistically significant with the confidence level of 95%. Differences in subgroups with smaller n will be considered directional. The total n will reflect the general universe with a high degree of accuracy (greater than or equal to 95% confidence level).

Unusual problems requiring specialized sampling procedures
We do not anticipate any problems that would require specialized sampling procedures.

Any use of periodic (less frequent than annual) data collection cycles to reduce burden
The survey will be conducted on a one-time, ad-hoc basis.

NWS partners survey:

Statistical methodology for stratification and sample selection
There will be neither sampling nor stratification of the NWS partners. All partners will be invited to participate. We estimate a conservative response rate of 10%, which translates to approximately 12,000 responses across all partner groups. The survey will be distributed through the network of Warning Coordination Meteorologists at WFOs and the Service Coordination Hydrologists at River Forecast Centers.

Estimation procedure
Most data analyses will be conducted at the overall level, with data cuts based on survey options. As we described for the Gen Pop group, data cuts with the minimum sample size of any single subgroup of n=~350 will allow us to run statistical tests to analyze if differences are statistically significant with the confidence level of 95%. Differences in subgroups with smaller sample size will be considered directional.

Degree of accuracy needed for the purpose described in the justification
For the NWS partners survey, the expected number of responses is higher than the minimum number of responses required and will reflect the general universe with a high degree of accuracy (greater than or equal to 95% confidence level).

Unusual problems requiring specialized sampling procedures
We do not anticipate any problems that would require specialized sampling procedures.

Any use of periodic (less frequent than annual) data collection cycles to reduce burden
The survey will be conducted on a one-time, ad-hoc basis.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Gen Pop sample, including the oversample for Native Americans, rural population, and Spanish speakers:

Online panel respondents are sent generic invitations to log into a portal, where they are then shown the full list of studies they may qualify for based on information in their profile. Generic reminders (not specific to this study) will be used to invite respondents to the portal and motivate them to take this survey. We do not envision any significant sampling bias issues since: a) the population of online panels is similar to the US population with internet access (no significant coverage bias); b) online panels have their own set of measures/processes in place to ensure their population is as representative of the US population with internet access as possible (no significant selection bias); and c) online panels use generic descriptions for all studies on the portal that respondents can access to avoid non-response bias in favor of other studies. Demographic quotas will be set to ensure the data represents the general universe to the extent possible.

NWS partners:

In order to maximize response rates and reduce non-response, the survey will be sent to NWS core partners who are trusted liaisons between the NWS office and the communities they serve. Up to two survey reminders will be sent, one after 1 week, and the second after 3 weeks (if required).

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Before launching the survey in full, we will launch it for 10% of the expected number of respondents within each sub-group (i.e., Gen Pop, 3 oversample groups, and NWS partner group) to ensure respondents don’t experience any difficulties completing the survey, the logic in the programmed survey is working correctly, and all variables are captured in the database correctly. If issues are identified in the review of the soft launch data, adjustments will be made to the collection instrument and/or survey logic prior to proceeding with the full launch of the study.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Noel Keene, noel.keene@noaa.gov, 541-776-4303

Tyra Harris, tyra.harris@noaa.gov, 202-468-8972

Brian Miretzky, brian.miretzky@noaa.gov, 301-427-9315

Corey Pieper, corey.pieper@noaa.gov, 469-340-2857

Timothy Oram, timothy.oram@noaa.gov, 682-703-3705

Forrester Team

Aliona Madjar, amadjar@forrester.com

Sean Galvin, sgalvin@forrester.com

Adam Taufiq, ataufiq@forrester.com