Supplementary and Part B Questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

NCA3 Use and Applications Survey

1. Explain who will be conducting this survey. What program office will be
   conducting the survey?

A third-party evaluation team led by Dantzker Consulting, LLC, in collaboration with New Knowledge Organization Ltd., and others, will be conducting the survey. This team is conducting the survey under contract with North Carolina State’s North Carolina Institute of Climate Studies (NCICS) whose primary activity is the operation of the NOAA Cooperative Institute for Climate Studies North Carolina (CICS-NC). NCICS/CICS-NC is colocated with the NOAA National Centers for Environmental Information (NCEI) and has engaged Dantzker Consulting, LLC, to conduct an evaluation of the processes and products associated with the Third National Climate Assessment (NCA3). The survey will be conducted as a component of the NCA3 evaluation funded under NCEI/CICS-NC’s Climate Assessment activities and in collaboration with the US Global Change Research Program (USGCRP). NCEI is the National Environmental Satellite, Data, and Information Service (NESDIS) data center and CICS is a NESDIS cooperative institute. NCEI/CICS-NC’s Climate Assessment activities are funded by NOAA’s Climate Program Office (CPO) under the NOAA Office of Oceanic and Atmospheric Research (OAR).

What services does this program provide?

NESDIS acquires and manages the Nation's operational environmental satellites, operates the NOAA National Data Centers, provides data and information services including Earth system monitoring, performs official assessments of the environment, and conducts related research.

NCEI is responsible for hosting and providing public access to one of the most significant archives for environmental data.

CICS-NC advances NOAA's mission to understand and communicate the current and future state of the climate primarily through collaborative research into the use of in situ and remotely sensed observations in climate research and applications.

CPO manages competitive research programs in which NOAA funds high-priority climate science, assessments, decision support research, outreach, education, and capacity-building activities designed to advance our understanding of Earth’s climate system, and to foster the application of this knowledge in risk management and adaptation efforts.

Who are the customers? How are these services provided to the customer?

Customers are environmental data seekers / environmental data users.  Data services are provided via data products, reports, etc. via respective websites and other formats.

The audiences for the NCA3 (the subject of this specific survey) include federal, state, local, and tribal agencies and decision makers; regional federal entities; members of the United States Congress; the private sector; participants of NCAnet (a network to engage users and producers of the NCA and its related products), and science educators.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics?

CICS-NC contracted with Dantzker Consulting, LLC, to conduct an external, third-party evaluation of the NCA3 production and dissemination processes and associated communication products.

The survey was developed by Dantzker Consulting, LLC, and its subcontracted partners (together ‘the evaluation team’) with the aim to learn how potential users of the report who were part of the dissemination strategy (approx. 5450) perceive the value of the NCA3 report and related communications products disseminated.

This number is based on USGCRP’s dissemination strategy – through their newsletter (n=4773), attendees of a White House event about the NCA3 (n=180), NCANet members (n = 215), and those who were sent print copies of the report (n =274).

The evaluation team consulted with members of the NCICS and USGCRP to identify the products of interest that need to be the focus of the report’s use. The team also learned about the dissemination strategy used by NCICS and USGCRP to identify the approximate number of potential user groups for the report.

What suggestions did you get about improving the survey?

The Dantzker Consulting, LLC, evaluation team consists of experienced survey research professionals. The team conducted multiple rounds of review and feedback while developing the survey, which helped refine understanding of the potential user groups, their numbers, as well as questions about the specific products to include in the evaluation, including the main NCA3 report, specific chapters categorized by regions (e.g., Northeast) and sectors (e.g., Water resources).

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)?

Due to privacy restrictions, the evaluation team will not be privy to USGCRP’s newsletter list, and hence a contact at the organization will forward an email with the invitation for the survey to the newsletter subscribers on behalf of the evaluation team. In this role, USGCRP will support the evaluation team’s data collection activities, without having access to the data collected via the online survey.

Individuals in all the other lists as provided by the USGCRP will be contacted by the evaluation team via email to invite them to participate in a brief online survey.

What percentage of customers asked to take the survey will respond?

The response rate is anticipated to be approximately a third of the total numbers in the potential user group, or 1,816 responses.

What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The following strategies will be employed to increase the response rate of the online survey:

1. The invitation email language has been developed to motivate participation by emphasizing that as a potential user of the NCA3 report, their feedback is critical to help support and improve future NCAs and related products.

2. The survey is designed to take no more than 5-10 minutes of a respondent’s time.

3. Three survey reminders are planned during the anticipated four week period while the survey will be available. Reminders will be sent on Mondays to maximize the chance of responses, based on research on survey procedures.

4. Describe how the results of this survey will be analyzed and used.

The quantitative survey will be analysed to present descriptive statistics including frequencies, means and standard deviations. The results of the survey will be analysed to elucidate potential differences in use, perceived value, and implications for individuals’ work based on different groups of users (e.g., regional location based on those that were the focus of the NCA3 report, different levels of engagement with NCA3). The results of this evaluation will enable NOAA to improve future climate assessments.

If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population?

Contingent upon aggregate summary statistics being available for the newsletter subscribers, we will use this data in addition to similar group level data for the others who were part of the dissemination strategy as group characteristics to analyse the extent to which the survey responses are representative. In this case, if survey responses are not representative of the potential user group population – for example, if particular sectors are under or overrepresented compared to their percentage of the population – we will conduct weighting adjustments to statistically “correct” under- or overrepresentation using auxiliary variables such as institutional affiliation or region where they are based.

Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The survey is not intended to measure a GPRA performance measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential user group for the NCA3 report comprised a total of approximately 5,450 individuals, who were included in the dissemination strategy adopted by USGCRP/NCICS. This number is based on USGCRP’s dissemination strategy – through their newsletter (n=4773), attendees of a White House event about the NCA3 (n=180), NCANet members (n = 215), and those who were sent print copies of the report (n =274)

The survey invitation will be sent to all members of this group (the newsletter subscribers by USGCRP, and the rest by the evaluation team).

As far as the evaluation team knows, data has not been conducted on potential users group of the NCA3 report. However, based on survey research trends, we anticipate a response rate of 33% from this group.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A contact at USGCRP will forward an email on behalf of the evaluation team to USGCRP’s newsletter subscribers. The evaluation team will directly send an invitation email to all other individuals who were part of the dissemination strategy (attendees of a White House event about the NCA3, NCANet members, and those who were sent print copies of the report) as identified by the CICS-NC/USGCRP team.

The email will request their participation in a brief online survey about their opinions and perceived value of the NCA3 report and related products for their work. The link to the survey will be included in the recruitment email. Upon clicking the link, an intro page will provide information on the goals of the survey, the required time commitment (5-10 minutes) and reiterate the informed consent information provided in the recruitment email. Clicking the link to continue with the survey will be considered voluntary agreement on the part of respondent to participate with an understanding of the risks and benefits of their participation.

Since the entire population will be contacted, no sampling procedures will be employed.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The following strategies will be employed to increase the response rate of the online survey:

1. The invitation email language has been developed to motivate participation by emphasizing that as a potential user of the NCA3 report, their feedback is critical to help support and improve future NCAs and related products.

2. The survey is designed to take no more than 5-10 minutes of a respondent’s time.

3. Three survey reminders are planned during the anticipated four week period while the survey will be available. Reminders will be sent on Mondays to maximize the chance of responses, based on research on survey procedures.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The survey will be tested with three to five people prior to finalizing to help ensure a smooth user experience for the survey and to help clarify any language or questions that might unclear. Data from these tests will be treated as mock data and will not be included in final data set. Nonetheless, all protections under IRB will be accorded to the participants, including consent to participate and treatment of data as anonymized, confidential and de-identified for any (unofficial) reporting purposes.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The data will be collected and analysed by members of the Dantzker Consulting, LLC, evaluation team, including its subcontractors New Knowledge Organization Ltd. and Clarus Research who will conduct post-hoc representational analyses.

The primary contacts are: Heather Dantzker, PhD, (Dantzker Consulting, LLC) Project Director and PI, 607-592-6040; heather@dantzker.com, and Rupu Gupta, PhD (New Knowledge Organization, Ltd.) Co-PI; 347-766-3399; rgupta@newknowledge.org]