Survey Pre-test Report

Glen Canyon Survey

Pretest Report

Summary of Glen Canyon Survey Pretest Results

The National Park Service submitted an Information Collection Request (ICR) to the Office of Management and Budget (OMB) requesting approval to pretest the Glen Canyon Survey. The pretest was conducted during November and December 2014. The primary goal of the pretest was to assess whether the survey instrument and data collection process worked as expected. This report summarizes the results of the pretest.

Overall, the data from the pretest suggest that the survey instrument worked well. Several small typographical (layout or punctuation) changes were made based on the pretest responses, and on closer review of the survey materials. These edits, none of which impacted the core survey valuation questions, are noted below.

1. Response rates

   1. The response rate was marginally lower than expected, but well within similar results from the literature.

The pretest followed the data collection plan described in the ICR and supporting statements. The households in the sample were mailed a pre-notification letter informing them that their household had been selected to be part of the survey. Following the letter, households received a packet containing a cover letter on NPS/University of Montana letterhead introducing the survey, a copy of the survey instrument, and a postage-paid return envelope. A reminder postcard was sent one week later. Finally, a second packet was sent that included a letter asking the respondent to complete the survey and a second copy of the survey instrument. The following table shows the mailing schedule for the pretest.

Pretest Survey Mailing Schedule

Contact	Date Mailed
Pre-notification letter mailing	November 7, 2014
First mailing of survey Instrument	November 14, 2014
Reminder postcard including Web address	November 21, 2014
Second mailing of survey instrument	December 9, 2014

A total of 225 U.S. household addresses were selected for the pretest sample (the ICR package specified 200, but 225 were mailed to allow for bad address returns). Of the 225 surveys mailed, 23 were returned as undeliverable and 49 completed surveys were returned for an overall response rate of 24%. This response is somewhat lower than the anticipated 30% response for the U.S. household sample population (the surveys were mailed to a U.S. household sample only). Responses may have suffered marginally from timing. The mailing of the survey was held until after the November general election date in order to not be lost in the volume of mailed election materials. This timing however, pushed the administration of the survey into the holiday season. It is hoped that response rates for the final survey instrument will benefit from better timing.

2. Was the survey instrument understandable to the public?

   1. The results from the pretest suggest that most respondents could understand the questions, followed instructions and had adequate information to answer the stated-preference conjoint questions.

Following the presentation of background information on Glen Canyon Dam and Natural Resources in the Study Area, respondents were asked a series of True/False questions in order to gauge how well they understood the information presented. The following table shows the statements presented, the correct answer, and the percent of respondents who chose the correct answer. For all statements, 85% or more of respondents understood the information correctly, indicating a high degree of understanding of the material presented.

Statement	Correct Answer	Percent Correct
There are now many more beaches along the Colorado River than there were before Glen Canyon Dam was built.	F	95%
Native fish populations in the Colorado River have fluctuated dramatically since the dam was built.	F	88%
The decrease in river beaches is most severe along wide sections of the river.	F	85%
Trout are not native to the study area.	T	95%
All native fish species have disappeared from the Grand Canyon.	F	98%
Nearly all visitors to the Grand Canyon National Park use the beaches along the river.	F	88%
Water levels below Glen Canyon Dam are constant throughout a 24 hour period.	F	90%
The Study area consists only of the area in and along the Colorado River between Glen Canyon Dam and Lake Mead.	T	90%
Two of the native fish species are in danger of extinction.	T	88%
Reducing daily fluctuations in the amount of water released from the dam will reduce the total amount of hydroelectricity produced.	F	85%

PROPOSED SURVEY CHANGE

The second statement “Native fish populations in the Colorado River have fluctuated dramatically since the dam was built” is somewhat ambiguous given the information presented, and will be changed in the final survey to a more definitive,

“Native fish populations in the Colorado River have declined continuously since the dam was built.”

The correct answer to this new formulation will be “False,” in recognition that native fish populations have fluctuated somewhat since the dam was built.

As part of the survey, respondents were asked their level of agreement with a series of statements related to the choices they made in the conjoint questions. There were two statements that dealt directly with comprehension, presented below. Looking first at the statement “The descriptions of the plans were hard to understand”, only 18% of respondents agreed with the statement (only 2.6% strongly agreed). For the statement “The survey provided me with enough information to make a choice between the options shown”, only about 10% of respondents disagreed (again, only 2.6% strongly disagreed).

Responses to Comprehension Questions

Statement	Strongly Agree	Agree	Neither Agree or Disagree	Disagree	Strongly Disagree
The descriptions of the plans were hard to understand.	2.6%	15.8%	26.3%	36.8%	18..4%
The survey gave me enough information to make a choice between the options shown.	18.0%	51.2%	20.5%	7.7%	2.6%

We also looked at the written comments provided at the end of the survey for evidence that the survey was hard to understand or was biased. A total of 21 or 48 possible respondents wrote additional comments at the end of the survey As expected, there are comments expressing both “conservation” viewpoints as well as more “limited taxation/pro-resource development” viewpoints. None of the comments included any charge of bias in the survey. A number of respondents expressed appreciation for the opportunity to complete a survey on the topic.

3. Did the levels for the conjoint questions work?

1. Overall 39.7% of the sample voted in favor of the action plans presented, but as expected this percentage was lower when the cost of the plan (bid amount) was higher.

The tables below show the percent of respondents voting for the action plans and the no action plan. Overall, without accounting for differences in attribute levels across the plans, 39.7% of the respondents selected an Action Plan (with associated increased costs) and 69.3% selected no action (zero additional cost).

The table also breaks down the percent voting for an Action Plan by the cost of the plan for the sample, and for each of the two conjoint questions. For all question responses combined, the percent voting for an Action Plan (A or B) was highest at the $40 bid level (54%), declined at the $110 bid level (39%), and was lowest at the $280 bid level (15%). This pattern is consistent with expectations, and indicates the upped bid level is sufficiently high to capture not only the median of the distribution but much of the tail as well. Based on these pretest results, no changes in the range of bid levels are proposed for the final survey.

Reponses to Conjoint Questions by Question

	First Conjoint Question	Second Conjoint Question	All Responses
Voted for no action	49%	61.5%	60.3%
Voted for plan	41%	38.5%	39.7%

Vote by Cost of Plan

	$40	$110	$280
Voted for Action Plan, Total Sample	53.8%	38.5%	15.4%

Conclusions:

Overall, the pretest of the Glen Canyon Total Value survey provided valuable feedback and support for moving on to full survey implementation with only one minor question wording change (noted above).

While the pretest response rate was marginally lower than expected based on similar recent surveys, the PIs feel this can be attributed to survey timing of administering the pretest between a national election and the holiday season. We fully expect response rates for the final survey to match our a priori expectations.

A key focus of the pretest was on the understandability and effectiveness of the conjoint question survey section in conveying information, and eliciting consistent, meaningful responses. While the pretest sample size is too small to estimate any meaningful discrete choice model parameters, respondents reacted to key choice attributes (cost) as predicted by theory (downward sloping demand curve). Additionally, the range of high and low bids presented suggests that no change in overall bid range is needed in the final survey instrument in order to capture the essential bid response distribution.

4