Supplemental and Part B questions

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

NCA3 Development Group Survey

1. Explain who will be conducting this survey. What program office will be
   conducting the survey?

A third-party evaluation team led by Dantzker Consulting, LLC, in collaboration with New Knowledge Organization Ltd., and others will be conducting the survey. This team is conducting the survey under contract with North Carolina State’s North Carolina Institute of Climate Studies (NCICS) whose primary activity is the operation of the NOAA Cooperative Institute for Climate Studies North Carolina (CICS-NC). NCICS/CICS-NC is collocated with the NOAA National Centers for Environmental Information (NCEI) and engaged Dantzker Consulting, LLC, to conduct an evaluation of the processes and products associated with the Third National Climate Assessment (NCA3). The surveys will be conducted as a component of the NCA3 evaluation funded under NCEI/CICS-NC’s Climate Assessment activities and in collaboration with the US Global Change Research Program (USGCRP). NCEI is the National Environmental Satellite, Data, and Information Service (NESDIS) data center and CICS is a NESDIS cooperative institute. NCEI/CICS-NC’s Climate Assessment activities are funded by NOAA’s Climate Program Office (CPO) under the NOAA Office of Oceanic and Atmospheric Research (OAR).

What services does this program provide?

NESDIS acquires and manages the Nation's operational environmental satellites, operates the NOAA National Data Centers, provides data and information services including Earth system monitoring, performs official assessments of the environment, and conducts related research.

NCEI is responsible for hosting and providing public access to one of the most significant archives for environmental data.

CICS-NC advances NOAA's mission to understand and communicate the current and future state of the climate primarily through collaborative research into the use of in situ and remotely sensed observations in climate research and applications.

CPO manages competitive research programs in which NOAA funds high-priority climate science, assessments, decision support research, outreach, education, and capacity-building activities designed to advance our understanding of Earth’s climate system, and to foster the application of this knowledge in risk management and adaptation efforts.

Who are the customers? How are these services provided to the customer?

Customers are environmental data seekers / environmental data users.  Data services are provided via data products, reports, etc. via respective websites and other formats.

The audiences for the NCA3 (the subject of this specific survey) include federal, state, local, and tribal agencies and decision makers; regional federal entities; members of the United States Congress; the private sector; participants of NCAnet (a network to engage users and producers of the NCA and its related products), and science educators.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? statistics?

CICS-NC contracted with Dantzker Consulting, LLC, to conduct an external, third-party evaluation of the NCA3 production and dissemination processes and associated communication products.

The survey was developed by Dantzker Consulting, LLC, and its subcontracted partners (together ‘the evaluation team’) with the aim to learn the experiences of the Development group comprising 395 individuals who were involved in producing the report.

To develop the survey, the evaluation team consulted with members of the CICS-NC and its USGCRP collaborators to learn the composition of the NCA3 developers, factors related to their participation in the NCA3 development process (e.g., their role, institutional affiliations), specific processes involved, products to focus on for the evaluation, and other relevant information.

What suggestions did you get about improving the survey?

The Dantzker Consulting, LLC, evaluation team consists of experienced survey research professionals. The team conducted multiple rounds of review and feedback while developing the survey, which helped refine questions about the development group members’ specific role in the process and the communication and interactions between them.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)?

All members of the NCA3 development group (N=395) as identified by the CICS-NC/USGCRP client team will be contacted by the evaluation team via email to invite them to participate in a brief online survey.

What percentage of customers asked to take the survey will respond?

The response rate is anticipated to be approximately a third of the group membership.

What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The following strategies will be employed to increase the response rate of the online survey:

1. The invitation email language has been developed to motivate participation by emphasizing that as a contributor to the NCA3 development, feedback is critical to help support and improve future NCAs and related products.

2. The survey is designed to take no more than 10-15 minutes of a respondent’s time.

3. Three survey reminders are planned during the anticipated four week period while the survey will be available. Reminders will be sent on Mondays to maximize the chance of responses, based on research on survey procedures.

4. Describe how the results of this survey will be analyzed and used.

The quantitative survey will be analysed to present descriptive statistics including frequencies, means and standard deviations. Where feasible, between-group differences will be compared using T-tests. The survey data will provide NOAA information on many aspects of the collaborative NCA3 development effort that will help inform improvements in the next National Climate Assessment and future NCAs.

If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population?

Based on learning the characteristics of the Development group population, in collaboration with the University/USGCRP, we will analyze the extent to which the survey responses we receive are representative of the population of the Development group as a whole and describe these group characteristics. If survey responses are not representative of the Development group population – for example, if particular subgroups or sectors are under or overrepresented compared to their percentage of the whole Development group sample – we will conduct weighting adjustments to statistically “correct” under- or overrepresentation using auxiliary variables such as group member experience. A brief description of the differences between the weighted and the unweighted samples will be provided to emphasize how the approach enables us to get a sample that reflects the survey population.

Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The survey is not intended to measure a GPRA performance measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The development group for the NCA3 report comprised a team of experts guided by a 60-member Federal Advisory Committee), and totalling approximately 395 individuals. This group included members from the National Climate Assessment and Development Advisory Committee (NCADAC), NCA3 authors, NCA and USGCRP staff, Technical Support Unit, Federal agencies and White House, Technical input report authors, Review editors, and NRC reviewers. The survey invitation will be sent to all members of this development group.

The evaluation team has been informed about a survey administered with the NCADAC subgroup of the larger development group, with a very high response rate, in that most members responded. Due to the inclusion of other sub-groups for which we have no information on survey response rates, we anticipate a lower response rate of about 33% as indicated in the response to B3.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The evaluation team will send an invitation email to all members of the Development group as identified by the CICS-NC/USGCRP team, requesting their participation in a brief online survey about their experience developing the NCA3. The link to the survey will be included in the recruitment email. Upon clicking the link, an intro page will provide information on the goals of the survey, the required time commitment (10-15 minutes) and reiterate the informed consent information provided in the recruitment email. Clicking the link to continue with the survey will be considered voluntary agreement on the part of respondent to participate with an understanding of the risks and benefits of their participation.

Since the entire population of NCA3 Development group members will be contacted, no sampling procedures will be employed.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

The following strategies will be employed to increase the response rate of the online survey:

1. The invitation email language has been developed to motivate participation by emphasizing that as a contributor to the NCA3 development, feedback is critical to help support and improve future NCAs and related products.

2. The survey is designed to take no more than 10-15 minutes of a respondent’s time.

3. Three survey reminders are planned during the anticipated four week period while the survey will be available. Reminders will be sent on Mondays to maximize the chance of responses, based on research on survey procedures.

4. Finally, post-hoc analysis of representation in relation to the Development group population overall will be conducted in terms of key factors such as gender, institutional affiliation, and role to understand if the data is skewed; based on under or overrepresentation of subgroups compared to their percentage in the whole Development group sample – we will conduct weighting adjustments to statistically “correct” under- or overrepresentation using auxiliary variables as mentioned.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The evaluation team intends to conduct pilot testing of the survey with a maximum of five members of the CICS-NC/USGCRP team to refine the instruments and ensure they are face valid, asking relevant questions.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The data will be collected and analysed by members of the Dantzker Consulting LLC evaluation team, including its subcontractors New Knowledge Organization Ltd. and Clarus Research who will conduct post-hoc representational analyses.

The primary contacts are: Heather Dantzker, Ph.D., (Dantzker Consulting, LLC) Project Director and PI, 607-592-6040; heather@dantzker.com, and Rupu Gupta, Ph.D. (New Knowledge Organization, Ltd.) Co-PI; 347-766-3399; rgupta@newknowledge.org