Supplemental and Part B questions

Supplemental Questions for DOC/NOAA Customer Survey Clearance

(OMB CONTROL NO. 0648--0342)

Reef Smart Participant Satisfaction Survey

Biogeography Branch

Center for Coastal Monitoring and Assessment

NOAA National Centers for Coastal Ocean Science

Supplemental Questions for DOC/NOAA Customer Survey Clearance

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

What program office will be conducting the survey?

This survey will be conducted by staff at the Biogeography Branch within the Center for Coastal Monitoring and Assessment (CCMA) of the National Centers for Coastal Ocean Science (NCCOS), National Ocean Service (NOS). The survey will be used to evaluate participant satisfaction with outreach events designed to inform policy-makers about the status and importance of scientific research being conducted by NOAA’s Coral Reef Conservation Program (CRCP) in the seven U.S. states and jurisdictions containing coral reefs. This outreach initiative is called “Reef Smart.”

What services does this program provide?

The mission of the CCMA Biogeography Branch is to develop information and analytical products through research, monitoring and assessment of the distribution and ecology of living marine resources and their habitats. The Biogeography Branch routinely collects and consolidates spatial and other physical and biological data that are used to produce a variety of biogeographic maps, tools and products. These services are provided to other program offices within NOAA, who are our primary customers.

NOAA CRCP is a cross-Line Office Program that brings together expertise from multiple scientific disciplines in order to increase understanding of coral reef ecosystems, and improve their management. In partnership with NOAA offices and local jurisdictions, the Program seeks to improve the health of coral reefs by mitigating threats and restoring coral ecosystems. The program accomplishes these goals by providing funds for research and management activities, as well as by disseminating information and providing forecasts. The newly launched Reef Smart initiative is a vehicle whereby information generated by CRCP-funded research can be disseminated to key stakeholders within the seven coral reef jurisdictions.

The idea for Reef Smart originated with similar outreach events that were hosted in 2011. One such outreach event was associated with a research cruise that was conducted by the Biogeography Branch and partners in the U.S. Virgin Islands. During this field research expedition, staff hosted an open house for key stakeholders, students and others aboard the NOAA research vessel in order to increase awareness within the local community about ongoing scientific research, as well as to encourage a sense of openness between researchers and the local community. This event culminated with a “Day at Sea” for local policy makers, where invited guests were taken out on the research vessel to hear presentations on scientific research and see demonstrations of research equipment.

The Reef Smart Initiative will be designed in a manner similar to the outreach event described above. The “Day at Sea” activity will be a core component of the Reef Smart program. It is during this activity that we propose use of the present survey in the future. In other words, we plan to administer this survey during each “Day at Sea” event hosted by the Reef Smart project managers in the future.

Who are the customers?

The Reef Smart Initiative aims to provide information about CRCP-funded coral reef ecosystem research to a variety of federal, territorial and local audiences. Primary target audiences include policymakers, natural resource managers and teachers/educators. The customers that will be surveyed with the proposed survey are policymakers who participate in the Reef Smart ‘Day at Sea’ event. Due to limitations of space onboard NOAA research vessels, the maximum number of invited participants for any Reef Smart ‘Day at Sea’ event will be 18.

How are these services provided to the customer?

Reef Smart’s central goal is to improve awareness of CRCP activities and other coral conservation initiatives. Reef Smart’s primary service is the hosting of science forums that combine presentations/demonstrations by scientists about research activities/findings and with informational lectures from local resource managers about management issues and activities. The daily schedule of Reef Smart events, generally, is:

Day 1: Fishing and Management Communities Workshop- Guests will hear overviews of the science being conducted related to coral reef ecosystems, get a briefing on recent research findings and management success stories, explore the tools used to collect data, and have the opportunity to talk with NOAA researchers about opportunities for collaborative research.

Day 2: Reef Smart ‘Day at Sea’—This will be a non-technical, interactive event geared toward policymakers. This event will highlight the impact of NOAA coral reef research, identify collaborative research success stories, and provide policymakers with a chance to ask questions about coral reefs, research, etc. This event will take place aboard a NOAA research vessel and will include a short time at sea where equipment demonstrations will occur.

Day 3: ‘Get Reef Smart!’—This will be an education program for K-12 students where select students and teachers will come aboard the docked vessel for a rotation of short, reef-centric classes. Each class will focus on a particular topic and will be age appropriate.

The survey associated with this submission has been developed to evaluate participant satisfaction with Day 2, the Reef Smart “Day at Sea’ event, of the Reef Smart program. Participants in the Reef Smart Day at Sea event could include policymakers at the federal, territorial, state, provincial, count, or municipal level. Participants will be asked to voluntarily complete the participant satisfaction survey. The questionnaire will be administered in-person to all invited attendees. Responses from the questionnaire will be analyzed and findings will be used to improve future outreach activities.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

Explain how this survey was developed.

Staff developed this survey instrument in accordance with the Generic Clearance for Customer Surveys provided by the NOAA Office of the Chief Information Officer, in consultation with individuals of the Biogeography Branch and CRCP who will be responsible for planning and executing the Reef Smart Initiative. Survey design and layout was informed by scholarship published in peer reviewed journals, as well as in non-peer reviewed, “gray” literature on outreach/education best practices and effective methods for evaluating public outreach events (Fedler 2001; Kelly 2004). Literature on the development of effective survey instruments (de Leeuw 2008; Dillman et al 2009) was also used. Additionally, staff consulted literature related to the assessment of respondent ‘awareness’ both generally and in relation to ocean issues (Ciochetto and Haley 1994; Steel et al 2005a; Steel et al 2005b).

With whom did you consult during the development of this survey on content? Statistics?

During development of this survey, we consulted with program staff from the NOAA Coral Reef Conservation Program and the NOAA Biogeography Branch. A draft version of the survey was provided to seven people, including two NOAA social scientists with experience in survey development. The two social scientists, Chris Ellis (NOS CSC) and Peter Edwards (NMFS Habitat) were asked to provide comments on the survey design and content. The remaining five people were asked to take the survey (i.e., read each item and select a response) and provide comments on a) how long it took them to complete the survey, b) any items that were confusing or unclear and why and c) any other observations or comments that they would like to share. Consultation with a statistician was not undertaken because only descriptive statistics will be used to analyze the data. The social science program manager for NOAA’s Coral Reef Conservation Program was also invited to provide comments on the survey design and content, although comments were not offered.

What suggestions did you get about improving the survey?

The two social scientists made suggestions as follows:

• Consider whether or not answers to the questions posed will measure “expected outcomes" of the workshop. Are metrics needed such as the number of attendees, diversity of attendee organizational affiliations, or increased understanding about coral reef ecosystems amongst participants? Response: The program managers for Reef Smart will know who attends the event because attendance is by invitation only. Moreover, the Reef Smart project managers will be able to conduct headcounts easily at each event because the maximum number of attendees at any one event is 18. To effectively measure increased understanding about coral reef ecosystems one would wish to implement an experimental study design. However, because of practical limitations, such as a heightened risk of participant burden and a lack of time in the event schedule for administration of a pre and post survey, we have opted not to use an experimental study design. We did include the items in Question 3 to allow participants to self-report about whether the Reef Smart event increased their awareness of coral reef ecosystems.

• Consider adding a question asking how likely a participant is to use the information presented during the Reef Smart event at some future date. Given the speculative nature of the suggested question, and our inability to verify actual use of the Reef Smart by participants in the future, we decided that such a question would be of dubious value and therefore opted not to include it.

• For Question 2, consider whether participants will understand the difference between the items “Duration of event activities” and “Timing and scheduling of the event.” Having considered the possibility for confusion and the usefulness of the information, the item “Duration of event activities” was changed to “Duration of the event.” The item “Timing and scheduling of the event” was dropped from the survey.

References:

Ciochetto, Susan and Barbara A. Haley. 1995. How Do You Measure "Awareness"? Experiences with the Lead-Based Paint Survey. Proceedings of the Section on Survey Research Methods, Alexandria, VA: American Statistical Association, pp. 1163-1168. Available online at: http://www.census.gov/srd/papers/pdf/sm9501.pdf.

de Leeuw, Edith. 2008. Self-Administered Questionnaires and Standardized Interviews. Chapter 18 in The Sage Handbook of Social Research Methods, P. Alasuutari, L. Bickman and J. Brannen, Eds. Los Angeles: Sage.

Dillman, Don A., Jolene D. Smyth and Leah Melani Christian. 2009. Internet, Mail and Mix Mode Surveys: the Tailor Design Method, Third Edition. Hoboken, N.J.: Wiley.

Fedler, Anthony J., Ed. 2001. Defining Best Practices in Boating, Fishing, and Stewardship Education. Prepared for the Recreational Boating and Fishing Foundation under Contract #RBFF-00-C-004 by Human Dimensions Consulting, Gainesville, FL.

Kelly, Leah F. Evaluation of Public Participation and Outreach Methods for the North Carolina Coastal Habitat Protection Plan. Thesis. Duke University, 2004. Available online at: http://dukespace.lib.duke.edu/dspace/handle/10161/244.

Steel, Brent, Court Smith, Laura Opsommer, Sara Curiel and Ryan Warner-Steel. 2005a. Public Ocean Literacy in the United States. Ocean & Coastal Management 48: 97-114.

Steel, Brent, Nicholas Lovrich, Denise Lach, and Valentina Fomenko. 2005b. Correlates and Consequences of Public Knowledge Concerning Ocean Fisheries and Management. Coastal Management 33: 37-51.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Explain how the survey will be conducted.

At the conclusion of the Reef Smart ‘Day at Sea’ event, all participants will be asked to complete the survey. The survey will be self-administered at the venue and returned by participants before they leave the event.

How will the customers be sampled (if fewer than all customers will be surveyed)?

No sampling will be undertaken. All invited participants to Reef Smart ‘Day at Sea’ events will be asked to complete the survey. This could include up to 18 persons per event.

What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate?

We anticipate a 100% response rate. The survey has been designed to make its completion as easy and straightforward as possible for participants. Additionally, we will implement several strategies to ensure a 100% response rate. First, we will allocate a dedicated period of 10 minutes just prior to the conclusion of the event for participants to complete their surveys. We anticipate that participants will be able to complete the survey in 4 to 5 minutes. Participants will be instructed by the host to complete their survey during this time. Second, each participant of a Reef Smart ‘Day at Sea’ event receives a complimentary ‘goodie bag’ containing small NOAA theme items (e.g., ink pen, magnet, etc.) and informational brochures for additional reading (e.g., brochures about coral reefs ecosystems, reports of research, etc.). These ‘goodie bags’ will be given to participants as they turn in their survey, although completion of the survey will not be a prerequisite for receiving a ‘goodie bag.’ Finally, the period allocated for survey completion will be timed prior to the vessel reaching port. By doing this, participants’ should feel at leisure to complete the survey before disembarking from the vessel and getting back to their busy schedules.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

Describe how the results of this survey will be analyzed and used.

The data from this survey will be analyzed using the software package SPSS. Data from this survey will be analyzed using only descriptive statistics (e.g., frequency, percent, means, etc.). The purpose of this survey is to provide information to those responsible for planning and executing the Reef Smart Initiative on the level of satisfaction that participants report after having attended a Reef Smart ‘Day at Sea’ event. Information from the survey will be used to assess the success of specific Reef Smart ‘Day at Sea’ events as well as the initiative as a whole in terms of participant satisfaction and self-reported improvement in awareness. Additionally, the information will be used to improve future events by informing adjustments in content, planning, and execution.

If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population?

We do not intend to employ a statistical sampling methodology for this collection. The universe of possible respondents will be surveyed. We do not plan to analyze the data collected using inferential statistics.

Is this survey intended to measure a GPRA performance measure?

The present survey is not intended to measure a GPRA performance measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.

‘Day at Sea’ participants will be policymakers, or their designated representatives, who are from or represent political jurisdictional units that include, or are in proximity to, focal coral reef ecosystems. Participants might also be policymakers whose constituencies have a social, cultural, or economic interest or stake in a jurisdiction that includes or is adjacent to a focal coral reef ecosystem. A ‘policymaker’ includes, but is not limited to, any elected or appointed government official, tribal leader, or the officially-designated staff of such officials. Policymakers could be from any level of government, including federal, territorial, tribal, state, provincial, county, parish, city, municipal, or township. Because of the broadness of this definition, we are not attempting to quantify the number of these policymakers.

Invitees to the ‘Day at Sea’ event will be purposively selected by the managers of the Reef Smart Initiative and their local collaborators, and in consultation with local coral reef management agencies and legislative liaisons with CRCP. The purpose of this event is to inform key policymakers, meaning those who have an interest or some jurisdictional authority over focal coral reef ecosystems, about research and management activities in the region. Thus, criteria for selection of invitees will not be scientific, but purposeful and intentional. Selection will be based on the professional judgment of program managers and recommendations from their associates. However, invitees will meet these basic criteria:

• Policymakers who are from political jurisdictional units that include focal coral reef ecosystems;

• Policymakers who are from political jurisdictional units that are in close proximity to focal coral reef ecosystems;

• Policymakers who are not from the jurisdiction, but whose constituencies have a social, cultural, or economic state in focal coral reef ecosystems;

Only individuals who attend Reef Smart ‘Day at Sea’ events will be asked to respond to the survey. Because the goal of the survey is to assess our participants’ experience related to engaging in a Reef Smart event, we define our universe of possible respondents as any person who attends a Reef Smart ‘Day at Sea’ event, as opposed to any policymaker related to a jurisdiction. The number of policymakers attending any one Reef Smart ‘Day at Sea’ event will vary, but will never exceed 18 individuals per event, as this is the maximum number of guests that can be accommodated shipboard. At this time, only one Reef Smart ‘Day at Sea’ event is scheduled for FY2012. However, Reef Smart ‘Day at Sea’ events could be planned for any of the 7 coral reef jurisdictions, as defined by the CRCP program, where NOAA research is underway.

Universe of Survey Respondents	Expected Response Rate
18 participants per event	100%

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

We do not intend to employ a statistical sampling methodology for this collection. The universe of respondents, defined as participants at each ‘Day at Sea event’, will be surveyed. We do not plan to analyze the data collected using inferential statistics.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

We anticipate a 100% response rate. The survey has been designed to make its completion as easy and straightforward as possible for participants. Additionally, we will implement several strategies to ensure a 100% response rate. First, we will allocate a dedicated period of 10 minutes just prior to the conclusion of the event for participants to complete their surveys. We anticipate that participants will be able to complete the survey in 4 to 5 minutes. Participants will be instructed by the host to complete their survey during this time. Second, each participant of a Reef Smart ‘Day at Sea’ event receives a complimentary ‘goodie bag’ containing small NOAA theme items (e.g., ink pen, magnet, etc.) and informational brochures for additional reading (e.g., brochures about coral reefs ecosystems, reports of research, etc.). These ‘goodie bags’ will be given to participants as they turn in their survey, although completion of the survey will not be a prerequisite for receiving a ‘goodie bag.’ Finally, the period allocated for survey completion will be timed prior to the vessel reaching port. By doing this, participants’ should feel at leisure to complete the survey before disembarking from the vessel and getting back to their busy schedules. The universe of possible respondents will be surveyed. We do not plan to analyze the data collected using inferential statistics.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

A draft version of the survey was provided to five people. These five people were asked to take the survey (i.e., read each item and select a response) and provide comments on a) how long it took them to complete the survey, b) any items that were confusing or unclear and why and c) any other observations or comments that they would like to share. The survey instrument was revised based on comments from these individuals. Suggestions from persons who tested the survey instrument included:

• Question 1: Consider whether the items “Quality of presentations and demonstrations” and “Ability of presenters to clearly explain concepts and information” are assessing the same thing, meaning the ability of speakers to present information clearly in multiple formats. Action: Upon discussion, we determined that what the project managers are really interested in is whether participants valued the research/equipment demonstrations provided during the event. Thus, we dropped the item “Quality of presentations and demonstrations” and added the item “Demonstrations of research equipment and technology.” By doing this, we remedy possible redundancy across the two items, and also focus on information more useful to the project managers.

• Question 3: Consider addition of an item asking participants if the Reef Smart event increased their awareness of management issues and activities. Action: We added an additional item to this effect in Question 3.

• Question 4: Consider whether or not information provided in Question 4 will be useful for future planning if this information is provided during the course of the event. Action: On closer consideration, we decided that use of the items in Question 5 of the previous draft (a set of awareness questions) would yield information that is too difficult to interpret for purposes of planning future Reef Smart events. Therefore, we added the new Question 4, an open-ended question, to provide participants with an opportunity to suggest topics of interested for future events. This information will be of more use to the project managers.

• Misc: Add a question for participants to provide general comments or remarks. Action: We added two general comment questions, Question 4 and Question 5.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

No statistical consultation was obtained for this survey. We do not intend to employ a statistical sampling methodology for this collection, nor do we plan to analyze the data collected using inferential statistics.

Data collection will be implemented by Ms. Alicia Clarke, with the Biogeography Branch, Center for Coastal Monitoring and Assessment, National Centers for Coastal Ocean Science, National Ocean Service. Ms. Clarke may be reached by email at Alicia.clarke@noaa.gov or telephone by 301-713-3028 x189.

Data management and analysis will be completed by Dr. Theresa L Goedeke, Social Scientist, with the Biogeography Branch, Center for Coastal Monitoring and Assessment, National Centers for Coastal Ocean Science, National Ocean Service. She may be reached by email at theresa.goedeke@noaa.gov or by telephone at 301-713-3028 x 237.

8