Supplemental and Part B Questions - changes marked

OMB Control No. 0648-0342

Product User Profile Survey

Biogeography Branch

Center for Coastal Monitoring and Assessment

NOAA National Centers for Coastal Ocean Science

Supplemental Questions for DOC/NOAA Customer Survey Clearance

1. Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

This survey will be conducted by staff at the Biogeography Branch within the Center for Coastal Monitoring and Assessment (CCMA) of the National Centers for Coastal Ocean Science (NCCOS), National Ocean Service (NOS).

The mission of the CCMA Biogeography Branch is to develop information and analytical products through research, monitoring and assessment of the distribution and ecology of living marine resources and their habitats. The Biogeography Branch routinely collects and consolidates spatial and other physical and biological data that are used to produce a variety of biogeographic maps, tools and products. These services are provided to other program offices within NOAA, who are our primary customers.

However, in order to also serve the wider marine resource management community and the interested public, hereafter referred to as our secondary customers, the Biogeography Branch makes freely available all data, maps, tools, and products produced by the branch. These products are served to secondary customers in a variety of ways. First, individuals or organizations can request hard copies or electronic versions (either CDs or DVDs) of reports, maps, datasets, tools, etc., which we will send to them via the post or a delivery service. Additionally, the Biogeography Branch regularly provides many of its products, including databases, maps, mapping tools, reports, etc., over the Internet on its own Web pages. Thus, secondary customers may use these products anonymously and at their own convenience. Finally, staff from the Biogeography Branch provides support to any secondary customer who contacts the branch with questions about products, regardless of how these products were obtained by the user. To date, the Biogeography Branch has not attempted to systematically identify or profile its secondary customers in any manner, meaning the branch has not tried to understand who uses its freely-served products, and for what purposes.

The central objective of this survey is to help the CCMA Biogeography Branch to identify and develop a qualitative profile of secondary product users, by targeting a product user profile survey to users of six specific products served on the Internet. The information collected will be used by the Biogeography Branch to improve strategies to deliver these products to secondary users. An additional objective of this profile survey is to further development of a sampling frame that, combined with other strategies for identifying the universe of Biogeography’s secondary customers, can later be used to facilitate a full product user satisfaction survey.

2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?

Staff developed this survey instrument in accordance with the Generic Clearance for Customer Surveys provided by the NOAA Office of the Chief Information Officer, in consultation with individuals of the Biogeography Branch who are responsible for creating or providing focal products. Additionally, staff reviewed a number of similar customer surveys previously used by other NOAA offices (e.g., Coastal Services Center, Washington Volcanic Ash Advisory Center, NOAA Office of Education, etc.), as well as consulted relevant literature on customer satisfaction survey development and design. Finally, draft versions of the survey were provided to a total of six persons within the Biogeography Branch and in other offices in the National Ocean Service (e.g., Coastal Services Center and the Center for Human Health Risk) for comment on content and structure of the survey. Reviewers suggested: focusing on specific products, as opposed to general categories of products; offering respondents a response choice “Not in the United States or U.S. Territories” in the context of the location question (Question 4); adding more detailed categories for the question regarding the respondents’ professional organizational affiliation, as well as adding a category specifically for “News Media” (Question 5); and adding a question that would allow respondents to sign up for notification of product updates or new releases (Question 7). The survey was revised per reviewers’ comments. Only descriptive statistics will be used to analyze response data. Therefore, consultation with a statistician was not undertaken.

 The red items are newly added, while the green are simply laid out a bit differently to make online service more efficient for the respondent.

 

Q2 & Q2A: Essentially, we reorganized this question, creating two new questions, Q2A and Q2B. Turns out, the Biomapper tool is really one product that a user can access multiple data products through. I didn’t really understand this, previously. So, on the survey, we separated the Biomapper tool from the data products that are served using it, adding a skip logic prompting inquiry about which dataset the user was looking for (Q2A), if the user selected the Biomapper tool in Q2.

 

Q2 & Q2B: We added a new response category “Report and/or other publication” to Q2. Similar to the above, we’ll incorporate a skip logic to inquire in Q2B with the user about which reports or publications they were looking for, if they check this item in Q2.

 

Q5: To improve the layout for the user (i.e., make the survey administration tool more efficient and user-friendly), we created an entry for the location question, Q5. Skip logic will be used to prompt the respondent to either Q5A (if they are in a U.S. state, territory or commonwealth) or Q5B (not in a U.S. state, territory or commonwealth) based on their response to Q5.

 

Q8A: We added a bit of text here to thank the respondent for agreeing to participate in a future survey and to assure them that we will not contact them for any other reason, aside from the survey, or release their contact information to others. We felt this was important to add.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

Our target population is persons who use one or more of our six pre-identified products that are served on the World Wide Web. More specifically, we seek to survey secondary customers who visit our Web sites and launch any of the following online products served by the branch:

• Coral Reef Ecosystem Assessment and Monitoring Database

• BIOMapper

• Benthic Habitat Viewer

• Habitat Digitizer

• ArcGIS9.2 Sampling Design Analysis Tool

• Aerial Photography Search

Based on Web site visitation statistics provided by the National Ocean Service, the Biogeography Branch had an average of 1358 hits, or single requests, for all Web sites per year from 2007 to 2009. Daily hits for individual product download Web sites ranged from 2 to 13 in 2009. However, given data collection limitations for Web site statistics, we cannot distinguish multiple hits by one user versus single hits by multiple users for any Web site. This also means that when implementing the survey, because we lack the ability to attach persistent cookies to a visitor’s system, we cannot track who has completed the survey previously or block persons from taking the survey multiple times.

Consequently, rather than sample, we intend to invite every secondary customer (identified by the launch of a target product) to voluntarily complete the survey, even if they are a repeat customer. Each time they launch a product, the secondary customer will be invited to take the survey via a pop-up or redirect page before continuing on to the product or, alternatively, as they exit the product page, whichever approach is more technically feasible at the time of survey development and administration. However, in order to ascertain how many respondents are repeat secondary customers, we have added a question to the survey, Question 1, asking respondents if they have completed the survey previously.

Because federal agencies may not make surveys compulsory, product users will be offered the option to opt out of the survey by choosing a button labeled “No thanks,” despite reports that offering this option in the context of pop-up survey can negatively influence response rate (Comley 2000). However, record will be kept of the number of opt outs, although we will not know anything about these users or why they have chosen not to take the survey. Regardless, keeping track of this information will communicate cursory information about our refusal rate as well as serve as an indication about how frequently these online products are accessed by our secondary customers.

Reported response rates for pop-up and other Internet-based surveys vary widely (Comley 2000; Tingling et al 2003). Comley (2000) reports a response rates in the range of 15% to 50% for pop- up surveys, specifically. We will implement this online survey in a manner that is likely to maximize response rate. For example, we have intentionally designed the survey to be concise. According to our estimates, respondents will be able to complete the survey in one sitting within 2 or 3 minutes, with limited scrolling. The online survey will be relatively easy for the respondent to complete, with simple question formats and user-friendly features (e.g., auto-population of the respondent’s state of location after entry of the first few letters of the response). We will also include a progress bar that lets the respondent know how far along they are in the survey.

References:

Comley, P. (2000). Pop-up Surveys: What Works, What Doesn’t Work and What Will Work in the Future. Retrieved May 10, 2011 from http://www.virtualsurveys.com/news/papers/

Tingling, P., M. Parent and M. Wade. 2003. Extending the capabilities of Internet based research: lessons from the field. Internet Research: Electronic Networking Applications and Policy 13(3): 223-235.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

The purpose of this survey is to begin developing a qualitative profile of secondary customers for the Biogeography Branch, a product user group that has heretofore been largely unknown. This survey, specifically, will help the Biogeography Branch learn more about the users of particular products that are served online. Data from this survey will be analyzed using only basic, descriptive statistics (e.g., frequency, percent, means). Findings from this survey will not be generalizable to the entire Biogeography Branch customer population, and generalizations will not be attempted. It is understood that profiles depicted from this data pertain only to secondary customers who have used the aforementioned products online, and who responded to the survey. It is hoped, however, that respondents to this survey will agree to participate in a full product user satisfaction survey at some point in the future, once we have identified an adequate and statistically defensible sampling frame of secondary customers by use of this profile survey along with the set of names and contact information for persons who request and receive products or services from the Biogeography Branch. The present survey is not intended to measure a GPRA performance measure.

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

It is presently not possible to know the universe of persons who use the Biogeography Web site to obtain products because federal entities are prohibited from collecting web visitation statistics in a manner that would allow for the individual identification of web visitors or their activities. However, based on aggregate Web site visitation statistics provided by the National Ocean Service, the Biogeography Branch had an average of 1358 hits, or single requests, for all Web sites per year from 2007 to 2009. Daily hits for individual product download Web sites ranged from 2 to 13 in 2009. Yet, given data collection limitations for Web site statistics, we cannot distinguish multiple hits by one user from single hits by multiple users for any Web site. Thus, the primary goal of this collection is to provide the Biogeography Branch with a basic profile of the frequency and nature of web-product access and usage.

Reported response rates for pop-up and other Internet-based surveys vary widely (Comley 2000; Tingling et al 2003). Comley (2000) reports a response rates in the range of 15% to 50% for pop- up surveys, specifically. Because federal agencies may not make surveys of this type compulsory, product users will be offered the option to opt out of the survey by choosing a button labeled “No thanks,” despite reports that offering this option in the context of pop-up survey can negatively influence response rate (Comley 2000). Record will be kept of opt-outs, although we will not know anything about these users or why they have chosen not to take the survey. Regardless, keeping track of this information will communicate information about our refusal rate as well as how frequently these online products are accessed by our secondary customers. Our maximum estimate of responses to this survey is 500.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Given data collection limitations for Web site statistics, we cannot distinguish multiple web hits by one user versus single web hits by multiple users for any Web site. This also means that when implementing the survey, because we lack the ability to attach persistent cookies to a visitor’s system, we cannot track who has completed the survey previously nor can we block persons from taking the survey multiple times. Additionally, because federal agencies may not make such surveys compulsory, product users will be offered the option to opt out of the survey by choosing a button labeled “No thanks,” despite reports that offering this option in the context of pop-up survey can negatively influence response rate (Comley 2000).

As a result of the aforementioned data collection limitations for online surveys within federal agencies, we intend to invite all secondary customers to voluntarily complete the survey each time they launch one of our target products, that is, they initiate use of a target product online. The secondary customer will be invited to take the survey via a pop-up or redirect page before continuing on to the product or, alternatively, as they exit the product page, whichever approach is more technically feasible at the time of survey development and administration. In order to ascertain how many respondents are repeat secondary customers, we have added a question to the survey, Question 1, asking respondents if they have completed the survey previously. Finally, record will be kept of the number of survey opt-outs, although we will not know anything about these users or why they have chosen not to take the survey. Regardless, keeping track of this information will communicate limited information about our refusal rate as well as how frequently these online products are accessed by our secondary customers.

We do not intend to employ a statistical sampling methodology for this collection, nor do we plan to analyze the data collected using inferential statistics.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Because federal agencies may not make surveys compulsory, product users will be offered the option to opt out of the survey by choosing a button labeled “No thanks,” despite reports that offering this option in the context of pop-up survey can negatively influence response rate (Comley 2000). However, record will be kept of the number of individuals who opt out of the survey, although we will not know anything about these users or why they have chosen not to take the survey. Regardless, keeping track of this information will communicate information about our refusal rate as well as how frequently these online products are accessed by our secondary customers.

Reported response rates for pop-up and other Internet-based surveys vary widely (Comley 2000; Tingling et al 2003). Comley (2000) reports a response rates in the range of 15% to 50% for pop- up surveys, specifically. We will implement this online survey in a manner that is likely to maximize response rate. For example, we have intentionally designed the survey to be concise. Completion is estimated to take 3 to 5 minutes, with limited scrolling. The online survey will be relatively easy for the respondent to complete, with simple question formats and user-friendly features (e.g., auto-population of the respondent’s state of location after entry of the first few letters of the response). We will also include a progress bar that lets the respondent know how far along they are in the survey.

We do not intend to use this information to generalize to the population of persons who use the Biogeography Web site to obtain products. Rather, we would like to gather this information to obtain basic information from people who obtain products from the Web site.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

Draft versions of the survey were provided to a total of six persons within the Biogeography Branch and in other offices in the National Ocean Service (e.g., Coastal Services Center and the Center for Human Health Risk) for comment on content. The survey has not been formally tested on targeted respondents.

5. Provide the name and telephone number of individuals consulted on the statistical

aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other

person(s) who will actually collect and/or analyze the information for the agency.

No statistical consultation was obtained for this survey. We do not intend to employ a statistical sampling methodology for this collection, nor do we plan to analyze the data collected using inferential statistics. The implementation of the information collection and data analysis will be completed by Dr. Theresa L Goedeke, Social Scientist, with the Biogeography Branch, Center for Coastal Monitoring and Assessment, National Centers for Coastal Ocean Science, National Ocean Service. She may be reached by email at theresa.goedeke@noaa.gov or by telephone at 301-713-3028 x 237.