2012
Nautical Charting Customer Satisfaction Survey
This document provides the details of the 2012 Nautical Charting Customer Satisfaction Survey, which has been updated from the 2009, 2010 and 2011 survey approved under OMB Control No. 0648-0342.
Explain who will be conducting this survey. What program office will be conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?
The survey will be conducted by the National Ocean Service’s Office of Coast Survey (OCS). OCS produces nautical charts in paper and electronic format, the U.S. Coast Pilot, a companion publication to nautical charts, and other products supporting marine navigation. Nautical charts provide navigational coverage of ports and harbors and United States territorial waters. These nautical products are used by commercial vessels and recreational boaters transiting United States waters and making calls on seaports. Nautical products are provided to customers through a government warehouse, a network of approximately 1,800 nautical sales agents, and through the Internet.
The reason for the survey is to act in accordance with Executive Order 12862, the National Performance Review and good management practices. Therefore, NOS is seeking to continue to determine whether its customers/users are satisfied with the services/products they receive and whether they have suggestions as to how the services/products may be improved in the future or made more useful. The information will be used to obtain guidance on how to improve NOS’s products and services.
The current NOAA strategy focuses on traditional and non-traditional users of NOAA’s products and services. The 2012 survey will be focused on:
The largest group of traditional users, who are commercial mariners, to get to know their needs better and obtain more feedback about commercial craft sailing needs.
Non-traditional users of NOAA’s products and services, who are recreational boaters, to get to know their needs better and obtain more feedback about the needs of small craft sailers.
2. Explain how this survey was developed. With whom did you consult during the development of this survey on content? Statistics? What suggestions did you get about improving the survey?
The 2012 commercial mariner’s and recreational boater’s surveys were developed from prior surveys to ensure that there is a trend analysis developed for both surveys. The questionnaires for both surveys are also based on questions used in previous surveys. Based upon user comments, the length of the survey has been reduced compared to past surveys, which should help to increase the response rate, but still provide actionable information.
The surveys will be administered online and there will not be any paper version of the survey printed. This online version of the questionnaire will be able to be filled out only by the recipient of the cover email sent to each mariner, and he/she will only be able to fill out the survey once.
Professional assistance in question development has been provided by the survey firm, Strategy, Research, and Action, who have worked in producing questionnaires which have been effective amongst chart users in the past. NOS has ensured that the 2012 survey is designed and administered using the knowledge gained from previous surveys, which will ensure that the questionnaire achieves the best balance between maximizing data quality and controlling measurement error, while minimizing respondent burden and cost.
3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)
The universe for the sample is defined as ‘users of charts and nautical products relating to navigating in US charted waters’. The selection of the sample frame for this survey is based on many years of reviewing chart users’ lists in relation to how the data is to be used, to give guidance on how to improve services/products for the different groups.
Completing a census of users is beyond the budget allocation that is available for the completion of this survey. Therefore NOS has worked with Strategy, Research & Action to develop a sample to give as representative and unbiased data about chart users as possible.
The aim of this survey is to select a representative sample using a stratified sampling technique, in which the chosen sample is forced to contain potential respondents from each of the key segments of the population. Such a sample provides statistical efficiency, as the sampling error is likely to be smaller than using a systematic random sample.
As has been done in previous surveys (2009 and 2010 for commercial mariners and 2010 for recreational mariners), the survey emails will be sent with a link to the research company’s server, so that the survey can be completed online. We are able to email the respondents because we have lists giving the named chart users’ emails.
The commercial mariners’ survey will be conducted by e-mailing a link to the survey questionnaire to 6,000 commercial mariners. We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:
Masters and Mates Union – a sample of 3,000 members from the mailing list of its members of its 6,800 members
Professional Mariner – a sample of 3,000 subscribers to the journal and website out from its list of over 8,000 subscribers.
The recreational boaters’ survey will be conducted by e-mailing over 8,800 recreational boaters who sail small craft. The recreational boaters email will mainly be to members of the U.S. Power Squadron, a national recreational boating organization, and to members of BoatUS. Both of these organizations have the most up to date list of chart users, and their lists have provided an excellent response in the past. Therefore we will be using the same sampling lists as in 2010, updated by the relevant organizations providing them. The contact lists below are selected to cover all parts of the population of chart users.
We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:
BoatUS – 6,000 subscribers from the mailing list of 250,000 recreational mariners
U.S. Power Squadrons – a sample of 2,800 named chart users, participants in Co-operative Charting Program from the total of 40,000 on their list of members.
A response rate of approximately up to 25% for these surveys is likely to be achieved, based on previous results and because of the following:
Accuracy of the Email List
NOS has spent
time ensuring the accuracy of the email list and the extent to which
it is
segmented into the relevant users of NOS charts. This
includes those who are regularly
interacting with NOS for all
of its services. Use of these mariners’ lists in the past
has also contributed to a consistent response of 25% for the
surveys.
Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data Protection and Codes of Conduct for research, asking the respondents if they would be willing to take part in future surveys. A relatively high response of 50% has been achieved for this question in the past, indicating that NOS had contacted the correct target audience and this helps to keep the response rate high in future surveys, although it is recognised that not all of them will take part in the future.
The relationship between NOS and its Chart Users
Based on past survey responses, in which 65% of respondents stated that they would be interested in taking part again, we can infer that the majority of respondents:
Are interested in the subject and the opportunity to comment on charts and NOS’s service, and
Believe that taking part in the survey will improve the relationship and service received from NOS in the future.
The Cover email
The questionnaires will be emailed with a cover email signed by a senior member of NOS staff, which will identify:
The purpose of the research
The reasons why NOS would like the recipients to respond
The ease of completion, as the survey mainly asks the respondents to check boxes for options listed
The short time needed to complete the questionnaire
A reminder that no confidentiality can be assured as a result of participating in the survey, but that respondents can withhold the details of their name and address
A NOS and research supplier contact name for more information or questions about the survey
The time scale for return of the questionnaire.
Administration of the NOS Email
In planning this survey, we have done our best to ensure that there is a high response by taking the following steps:
Issue an initial email to all the targeted audiences to inform them
that a voluntary
questionnaire will be emailed shortly and
to raise their awareness of its pending
arrival.
Email the questionnaire 7 days later. NOS will inform respondents
that the
responses to the collection of
information are voluntary. The information will
also state that the questionnaire will take no longer than 10
minutes to complete.
The OMB Control Number and a
Paperwork Reduction Act Statement will be
shown
at the end of the online questionnaire.
Email a reminder 7 days after the questionnaire has been emailed reminding the target respondent to return the questionnaire as soon as possible.
This will result in approximately 3,700 completed survey responses, and at 10 minutes per response, 617 burden hours.
Each survey questionnaire will be checked by the research contractor upon receipt. Any inaccurate completions will be discarded and only questionnaires which have been completed accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. This is done by separating the survey responses from classification information, which would identify who has provided the response.
Most common apparent reasons for non-response in these surveys are refusals or non-availability of respondents, likely resulting from the timing of the survey; NOS management is taking care to schedule the survey during periods not including national holidays. Mainly because we have not had obviously skewed response profiles in past surveys, we did not build into the survey, or budget for, any follow-up activities with non-responders.
4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)
Analysis and Report Planning
A multivariate analysis will be conducted on the responses and Chart user types and other chosen classification information, such as affiliations, type of commercial mariner and type of recreational boater, etc. Means and standard deviations and standard errors for user representation will also be calculated. The survey is not intended to measure a GPRA performance measure. These analyses will be comparable with previous surveys, so that similar statistical conclusions can be derived from past survey data as is acceptable statistical practice.
Review of Information Products
NOS will be responsible for the quality of information that they disseminate and will institute appropriate content/subject matter, statistical and methodological review procedures to comply with OMB and agency Information Quality Guidelines.
Releasing Information
NOS will produce a research summary which can be used to release information intended for the general public in line with NOS dissemination policies and procedures. An Appendix will describe the procedures that have been used to evaluate the quality of the data, to allow users to interpret results of analyses. This will assist the management of NOS to use the survey data to improve its products and services.
Data Protection and Disclosure Avoidance for Dissemination
NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. This is done by separating the survey responses from classification information, which would identify who has provided the response. NOS will ensure that the information that is published or released on request will comply with applicable Federal legislation and regulations.
Survey Documentation
NOS will produce survey documentation that includes those materials needed to analyze data from the survey, as well as the information necessary to replicate and evaluate the survey’s results compared with previous surveys.
Documentation and Release of Public-Use Microdata
If NOS releases microdata to the public it will ensure that it will include documentation clearly describing how the information is constructed and if required will also provide the metadata necessary for users to access and manipulate the data.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.
The universe for the sample is defined as ‘users of charts and nautical products relating to navigating in US charted waters’. The aim of this survey is to select a representative sample using a stratified sampling technique, in which the chosen sample is forced to contain potential respondents from each of the key segments of the population. Such a sample provides statistical efficiency, as the sampling error is likely to be smaller than using a systematic random sample.
The commercial mariners’ survey will be conducted by emailing 6,000 self-completion survey forms to commercial mariners who sail commercial craft. As in 2011 the emails will be sent to commercial mariners with a link to the research company’s server, so that the survey can be completed on-line. This is possible because we have a list which gives named chart users’ emails.
The recreational boaters’ survey will be conducted by mailing over 8,000 self-completion survey forms to recreational boaters, together with a link to the research company’s server, so that the survey can be completed on-line. The recreational boaters emailing will mainly be to members of the U.S. Power Squadron, a national recreational boating organization, and to members of BoatUS. Both of these organizations have the most up to date list of chart users, and their lists have provided an excellent response in the past. Therefore we will be using the same sampling lists updated by the relevant organizations providing them. The contact lists below are selected to cover all parts of the population of chart users.
The commercial mariners email will mainly be to subscribers of Professional Mariner, as they have the most up to date list of chart users. The contact lists below are selected to cover all parts of the population of chart users.
We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:
Masters and Mates Union – a sample of 3,000 members from the mailing list of its 6,800 members
Professional Mariner – a sample of 3,000 subscribers to the journal and website out from its list of over 8,000 subscribers.
The recreational boaters’ survey will be conducted by e-mailing over 8,800 recreational boaters who sail small craft. The recreational boaters email will mainly be to members of the U.S. Power Squadron, a national recreational boating organization, and to members of BoatUS. Both of these organizations have the most up to date list of chart users, and their lists have provided an excellent response in the past. Therefore we will be using the same sampling lists as in 2010, updated by the relevant organizations providing them. The contact lists below are selected to cover all parts of the population of chart users.
We show below if each list will be used in its entirety or sampled, and the expected number to be sampled:
BoatUS – 6,000 subscribers from the mailing list of 250,000 recreational mariners
U.S. Power Squadrons – a sample of 2,800 named chart users, participants in Co-operative Charting Program from the total of 40,000 on their list of members.
2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.
When the lists have been assembled by NOS management, the names of those who will be contacted will be selected by using a ‘skip interval’, allowing NOS management to go through the lists and select the names on the basis of the skip interval calculated, using the following formula:
k= Size of population = N (Total number in lists provided)
Size of sample required n
In the past use of these lists has provided a high strike rate for contacting chart users. When the sample is selected, a random choice sequence for label selection (by taking a random number and counting the intervals of labels) for the contacts will be adopted, counting through the lists until the sample is full. This has been the procedure used by NOS management in the past that has ensured a good response and meaningful response.
In surveys of this nature, and in the past NOAA surveys completed, usually, 95% confidence limits are used for the accurate interpretation of the data - we are confident of using the 95% level for this survey, as the repeated samplings of the same population of previous surveys identified the properties of the normal distribution. This provides the confidence limits for 95% of the samples selected including the parametric mean.
Therefore, analysis of the data will be subjected to a calculation that will be used to calculate the confidence limits: multiplying the standard error of the mean x the appropriate t-value. This means that there is a 5% probability or a 1 in 20 chance that the result or finding has occurred by chance. This is the lowest acceptable level in most market research business to business projects, such as this survey and will be sufficient for interpreting the data for the guidance NOS needs.
For example, if we received back 195 questionnaires in the survey for one particular chart user type (such as Container Ship operators), or 195 commercial mariners say that NOAA service has improved in the last 12 months, the following calculations could be made:
Number Of Observations = 195
Mean = 9.261460
Standard Deviation = 0.2278881e-01
Standard Deviation Of Mean = 0.1631940e-02
Referring to the following table provides the confidence levels that will be used for the survey:
Value (%) Value Limit Limit
50.000 0.676 0.110279E-02 9.26036 9.26256
75.000 1.154 0.188294E-02 9.25958 9.26334
90.000 1.653 0.269718E-02 9.25876 9.26416
95.000 1.972 0.321862E-02 9.25824 9.26468
99.000 2.601 0.424534E-02 9.25721 9.26571
99.900 3.341 0.545297E-02 9.25601 9.26691
99.990 3.973 0.648365E-02 9.25498 9.26794
99.999 4.536 0.740309E-02 9.25406 9.26886
The table shows the confidence interval for several different significance levels. The first column lists the confidence level (which is 1 - expressed as a percent), the second column lists the t-value, the third column lists the t-value times the standard error, the fourth column lists the lower confidence limit, and the fifth column lists the upper confidence limit.
Therefore, if we use a 95% confidence interval, the row identified by 95.000 in the first column then you can see an interval of (9.25824, 9.26468) from the last two columns.
3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.
Response rates of approximately up to 25% for the surveys are likely to be achieved, based on previous results and because of the following:
Accuracy of the Email List
NOS has spent time ensuring the accuracy of the email list and the extent to which it is segmented into the relevant users of NOS charts. This includes those who are regularly interacting with NOS for all of its services. Use of these lists in the past has also contributed to the high response of 25% for the surveys.
Re-contacting the Target Audience
Past surveys have taken into account the recent requirement made in all Data Protection and Codes of Conduct for research, asking the respondents if they would be willing to take part in future surveys.
A high response of 50% has been achieved to this question in the past, indicating that NOS had contacted the correct target audience and this helps to keep the response rate high in future surveys, although it is recognised that not all of them will take part in the future.
The relationship between NOS and its Chart Users
Based on past survey responses, in which 65% of respondents stated that they would be interested in taking part again, we can infer that the majority of respondents:
Are interested in the subject and the opportunity to comment on charts and NOS’s service, and
Believe that taking part in the survey will improve the relationship and service received from NOS in the future.
The Cover Email
The questionnaires will be emailed with a cover email signed by a senior member of NOS staff, which will identify:
The purpose of the research
The reasons why NOS would like the recipients to respond
The ease of completion (most questions require only checking the applicable box)
The short time needed to complete the questionnaire
A reminder that no confidentiality can be assured as a result of
participating in the
survey, but that respondents
can withhold the details of their name and address
A NOS and research supplier contact name for more information or
questions about the
survey
The time scale for return of the questionnaire.
Administration of the NOS Email
In planning this survey we have done our best to ensure that there is a high response to complete the emailing as follows:
Issue an initial email to all the targeted audiences to inform them
that a questionnaire
will be emailed shortly and
to raise their awareness of its pending arrival.
Email the questionnaire with the covering letter 7 days later.
Email reminding the target respondent to complete survey as soon as possible.
Each survey questionnaire will be checked by the research contractor upon receipt. Any inaccurate completions will be discarded and only questionnaires which have been completed accurately and with due consideration will be included in the final analysis.
NOS will implement safeguards throughout the survey production process by asking the research contractors to ensure that survey data are handled to avoid disclosure. This is done by separating the survey responses from classification information which would identify who has provided the response.
Most common apparent reasons for non-response in these surveys are refusals or non-availability of respondents, likely resulting from the timing of the survey; NOS management is taking care to schedule the survey during periods not including national holidays.
Mainly because we have not had obviously skewed response profiles in past surveys, we did not build into the survey, or budget for, any follow-up activities with non-responders.
4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.
No additional tests will be undertaken.
5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
NOS has consulted the appointed contractor for the statistical aspects of the design. The Contractor is Robin J Birn, President, Strategy, Research and Action, Inc, 29 Lyon, Newport Coast, CA 92657 Tel: 949-760-3980 Email: Robin.Birn@imparta.com.
2012 Nautical Charting Customer Satisfaction Survey Robin J Birn – Strategy, Research & Action Inc. – April 2012
- DG133C09SE4763-Mod 0002 - NCNJ50001201127
| File Type | application/msword |
| File Title | Survey Planning |
| Author | Robin Birn |
| Last Modified By | sarah.brabson |
| File Modified | 2012-04-25 |
| File Created | 2012-04-25 |