SSA Question 8 - Responses

SSA Question 8:

Consultation with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

The survey instruments and sampling plan were reviewed by three anonymous reviewers with expertise in the relevant scientific fields related to the survey effort. Reviewer IDs are indicted by number. Each comment, where applicable, includes the study team’s response and specific steps taken to address the content of the comment.

Reviewer #1
Topic Area	Reviewer Comments	Response
General	At the outset, I want to acknowledge that the planned survey and analysis is obviously a very high-level, meticulously planned study. I am aware of several of the participants in this study and know them to be highly capable researchers in the techniques employed in this study. Given this, my comments and suggestions deal largely with asking for clarifications, suggesting minor changes, and posing a few larger issue questions. I applaud the researchers on their work so far on this survey/study and am excited to see the final results.	No Response
2. Purpose and Methodology	The purpose of the study and proposed methodology are generally very well presented. The methods are clearly appropriate and valid. There is no explicit discussion the reviewed material to analysis or characterization of uncertainty. It does not appear that these sections were intended for inclusion in this review.	The data analysis methods are discussed in the supporting statement, which wasn't shared with the peer reviewers. We do not feel that a discussion of uncertainty would be particularly useful at this stage. Without preliminary information on the variance of model parameters (e.g., from the pilot), a power analysis would not be meaningful.
3 Novelty	NPS entry fee price elasticity has been a question of interest for some time. Since the Taylor-Univ WY report in 2006, I am not aware of any large-scale attempt to explore this issue. The current survey/sampling plan presents an ambitious study proposal to identify price elasticity for a range of park sizes/entry fee structures. Additionally, given the robust planned sample size, it is likely that price elasticities for a number of the larger individual park units could also be estimated. The use of SP question formats in assessing WTP and price elasticity is (to my knowledge) a new application to the issue of park entry fees. The use of a carefully programmed computer-aided online survey instrument with complex branching and skips and automatic filling of key question attributes also provides a state-of-the-art application of survey methodology to the NPS fee issue.	No Response
4. Sampling Plan	A key strength of this study is the inclusion of a pretest, a pilot and a final survey. There are some novel aspects to the SP choice questions in this survey and these preliminary steps are essential to ensure that the key SP questions are "working." The inclusion of non-response follow-up surveys also strengthens the interpretation of the results. Finally, the very large anticipated final sample should allow ample sample to analyze subsamples, strata, and exclude potentially confounding observations.	No Response
5. Understandability	Because of the computer aided structure of the instrument, upon review it looks more complex than it will be in actual administration. Overall, it flows well, and is well-presented. I will include some additional comments on the SP section below. Particularly effective is the use of maps and NPS unit dropdown boxes to aid in identifying parks visited or intended to visit. In past surveys, I have found that even among parties who have purchased NPS or federal recreation passes, there is quite a bit of confusion as to who administers different federal recreation sites (NPS, USFS, USFWS, BOR, etc.) Constraining the set of NPS units that respondents can list to ACTUAL NPS units should really help in focusing on the usable response data.	No Response
6. Usefulness	If the SP models yield statistically significant results, then the data and information from the survey will be extremely useful to managers and policymakers. In addition to the information, the survey collects a myriad of other information which will also be of great use to NPS and Interior decisionmakers.	No Response
7. Strengths and Limitations	The sampling plan is very strong, incorporating well-accepted population base, non-response follow-up, weighting of responses, and a generous sample size target. It is well-conceived and follows the lead of many other successful household survey efforts. If I had one qualm about the plan, it would be the assumed response rate (25%). This rate is justified by the previous surveys listed in Table 1 of the Draft Sampling Plan. However, there are other recent large-scale NPS sponsored or affiliated surveys that got significantly lower response rates. The NPS total value survey (CSU-Harvard) had, I believe, around an 18% response rate. Also, the NPS Glen Canyon Dam Passive Use Survey (2016) got an even lower 13% response for the national household survey. Both of these survey efforts differed in certain ways from the proposed survey, but both were in many ways comparable to the proposed sampling plan as well. It is entirely plausible that the survey will achieve its 25% target response rate. However, I would caution the researchers to ensure their sample and analysis plans are still feasible if the number of responses is significantly lower. A final point on survey administration. I assume the survey is still winding its way through OMB approval. I am sure the researchers are keeping in mind the timing of the eventual administration. In the months leading up to the 2020 November election, households will be inundated with political mailings, and survey materials could easily get lost in the mess. Timing of mail surveys is always important, and with a potential conflict with a major election, that is doubly true.	We examined the two studies mentioned by the reviewer and determined that they differ in substantive ways from our survey. The CSU-Harvard total value study (which achieved a 17-19% response rate) was not a government-sponsored research effort. The lack of a "trusted" government sponsor tends to decrease response rates. The Glen Canyon Dam study (which achieved a 13% response rate) did not provide a financial incentive, which tends to decrease response rates. In addition, the survey focused on a topic (Glen Canyon Dam/Grand Canyon) that many potential respondents may have perceived as a regional issue less relevant to their households. Although it is certainly possible that a 25% response rate is overly optimistic, the sampling plan identifies several studies that have achieved response rates close to 25%. Response rates are somewhat difficult to predict, as they are affected by a number of factors related to survey design and implementation. We recommend that the sample size and assumed response rate be re-assessed after the pilot, as the pilot will provide the best estimate of our likely response rate.
Survey Question	Reviewer Comment	Response
Q1-Q4	Perhaps address up front (Q1-Q4) issue of potential confusion about agency management of recreation sites - just to put this to bed. Maybe simply a statement such as "The NPS is one of several agencies that manage federal recreation sites. Others include USFS, BOR, BLM, etc. In this survey, we will ask you about your use of parks administered by the NPS." Something along these lines-just a suggestion.	We agree with this suggestion. The wording has been added to the beginning of Q4 per the reviewer's comment.
Q4	Had you thought about limiting the number of NPS units visited in the past 2 years? Maybe 5-10? To do this you would need some selection criteria-most recent visits. Not a big deal, but some folks might have a long list of parks visited.	We agree with this suggestion. A small number of participants could have a very long list of recent park visits. We propose imposing a cap of 10 NPS units using the most recent visit criteria. This clarification has been added to Q4 and logic has been modified for Q5.
Q7	The wording is a bit unclear to me. Often, we have defined a "trip" as a single trip to the park from your home. You might want to make sure the nomenclature is clearly speed out as to what is a trip and what is a visit. Are they synonymous?	We do intend to differentiate "trip" from "visit," in that a trip is a departure from home to visit one or more national park units. We have clarified this text on Q7 and throughout the questionnaire.
Q11-16	Q11-16. Take care to ensure you know what the definition of Group and adult is. It appears that info from Q12 is used in the SP questions to fill out the number of adults in the group. How is "adult" defined? Is it only the number entered in "Other members of my household age 16 or older," or does it also include "friends or relatives not living in my household?" Q13 asks for number of vehicles in your group. It seems likely that there are many instances where a group is traveling together in 2 or more vehicles and someone different in each vehicle pays the entrance fees for that vehicle. This could really muddy the definition group, and "adults" in the SP question. I may be misinterpreting this series of questions, but at first blush it appears to allow for different interpretations by respondents. Please take this just as a cautionary note.	The comment on group size is well taken. We have adjusted Q12 to identify the number of all adults (16 or older) and all children (under the age of 16) in the visiting group, not just household members. In terms of the details related to the number of vehicles, entrance fee types and amounts, and group definitions, there are a very large number of permutations for these attributes. Respondents will have reference visits with full, partial, or no financial responsibility; that used a single entrance pass or multiple types of entrance passes, where they entered alone, with one other person or a family, or with large groups of multiple friends or family units. We believe that trying to explicitly understand the exact details of every possible permutation will require an undue level of burden for respondents. We think the amount of information collected in Q11-16 is sufficient to build the SP experiments and to understand the level of financial responsibility which can be interacted with price sensitivity.
Q14	Just a small point. For parallel construction, I might suggest wording the options as (per person one-day, 3-day, or weekly pass).	We agree with this suggestion and have updated the questionnaire to reflect 1-day, 3-day, or weekly pass throughout.
Q16	Wording suggestion. Perhaps change to a more direct construction such as "Who purchased the <pass type> used on this visit to <NPS unit>?	We agree with this suggestion and have updated the Q16 to reflect the suggested wording.
Q18	Asking for the total entrance fees paid on visit becomes problematic when applied to an Annual Pass. Is the first use of the pass the full cost of the pass and subsequent uses zero cost, or is the cost of the pass to be averaged across all uses in the year? You see the issue here-it is one of assessing what this response will be used for and how it should best be interpreted.	We intend to ask about the cost of the annual pass at the time of purchase. The wording has been adjusted to reflect this detail for respondents who used an annual pass to enter the reference visit park.
Q23	I wonder what the use of the expenditure data will be? The question is constructed to minimize the burden on the respondents, but as with all questions should only be asked if the data is informative to the study.	The expenditure data is intended to be tested as a covariate with entrance fee price sensitivity in the discrete choice models. The hypothesis is that trip expenditures will be negatively correlated with sensitivity to entrance fee amounts.
Q24	As you are aware, perhaps a large majority of visitors to large NPS units (like YNP, YOSE, etc.) enter in vehicles with a 7-day pass for the group. Again, a large share of these visitors visit less than the full 7 days. So, for a good share of respondents, this question will be redundant to their previous answers. Not sure what to do with this, but though I would mention it.	This question is intended to provide an upper bound on the total number of park visits assuming no financial limitations. This "trip budget" will be used to support the discrete choice model estimation.
Future Park Unit Visitation	Future Park Visitation: I understand the need to retain a parallel construction of this section with the Past Visitation questions, but, obviously, the answers to these questions are much more speculative, and the level of detail about the planned trip asked may seem difficult to answer. For instance, "group size, travel mode, type of entry fee used, trip expenditures, etc." may seem a bit deep in the weeds for someone just at the point of thinking "hey, I would like to get up and drive over Going to the Sun Road sometime this year." Again, I understand why the questions are structured this way, but this could be an issue.	We agree that these questions will be more difficult to answer for future visits and with greater uncertainty. We have included "not sure" options for all information that is not required to build the SP exercises.
Future Park Unit Visitation	There are a couple of carry-over errors in this section: Q34 references "most recent visit," Q40 asks "Did you plan" rather than "Do you plan."	These errors have been corrected.
SP	My biggest concern with the study/survey likely is rooted in my not fully understanding certain issues surrounding the SP question design. The methods that are presented are clear and appropriate, but I would have liked to see a bit more description of the construction of the price variables in the SP questions. I am assuming these are calculated from some limited set of price attribute levels tied to the actual cost incurred for park entry on the visit. My largest concern regarding the central SP question design is the interaction of the respondent choosing the number of days as well as the preferred option in the SP question. It appears that "total cost for entry" is the cost attribute being used. As noted above, there could be some confusion, or miscommunication as to how many group members the respondent actually paid for (or is including in their answer). I would want this to be pretty explicit in the question(s).	The price values used in the examples are placeholders and the final levels for the pretest and pilot will be set in coordination with NPS. We understand the concern related to setting the price attribute as part of the discrete choice. We have used this type of design in other studies to estimate the price elasticity of demand and feel that the application is appropriate here. Total cost for entry is the cost attribute reflecting all adults (16 or older) in the visiting party. This number of adults will be shown in each experiment along with the total cost.
SP	Since number of people is used in calculating the total cost for # adults, was consideration given to allowing the respondent to vary not just the number of days for the trip but also the number of adults?	Yes, consideration was given to allowing respondents to vary the number of adults in the group along with the length of visitation. We ultimately decided against this for a few reasons. First, including options for both the group size and number of days significantly increases the dimensions of the pricing problem and would likely lead to respondent confusion. Second, we believe that group size is effectively static for a large majority of respondents and the most reasonable behavioral responses will be to visit the park as planned, reduce the number of days visited, or not visit at all.
SP	I understand that the cost attributes in Q51 and 52 are examples, but the total cost for 3 adults is pretty eye-popping. Again, I would caution to be sure that it is clearly understood who paid for the groups' fees to the park, and who should be appropriately included in that group.	We understand the concern about the magnitude of prices shown. These values used in the examples are just placeholders and the final levels for the pretest and pilot will be set in coordination with NPS. Changing the entrance fees from per vehicle to per-person is a policy that NPS would like to be able to evaluate.
SP	In the case of the "Per Adult Annual Parks Pass" the example shows a VERY large increase in costs. I believe the current Annual Parks Pass costs $80 and allows all people in a vehicle or up to 4 adults at a per-person NPS site to enter on the pass. While varying the cost of the pass (for example to $150 as shown) is a big impact, what is much more impactful is the assumed requirement that each adult have a separate pass. In essence, in the Q51 example shown, the cost of the entrance goes from $80 for the group to $450. Just something to keep in mind.	We understand the concern about the magnitude of prices shown. These values used in the examples are just placeholders and the final levels for the pretest and pilot will be set in coordination with NPS. We have reduced the values shown in the example experiments, but per-person annual passes is a policy that NPS would like to be able to evaluate.
SP	One final comment. Have you considered adding a relatively robust WTP question following the SP questions? For example, "On your recent trip to <park unit> the entry fee for one vehicle and its occupants for 7 days was ($XX). If everything about your trip were the same except this entry fee was increased to ($YY), would you still have chosen to visit the park?" This old-style DCCVM construction could provide a robust fallback estimate of price elasticity. This is just a thought to avoid hanging everything on the somewhat novel construction of the SP questions. Likely, this would be unnecessary due to the careful attention to survey instrument development and testing. Best of luck with this ambitious project!	We’ve added a follow up question to the SP exercises (Q54).

Reviewer #2
Topic Area	Reviewer Comments	Response
Purpose and Methodology	The stratified sampling plan and mode of delivery are appropriate and valid. The writing is clear and easy to follow. The authors also do a thorough job of pretesting, executing a pilot study, and securing as high of a response rate as possible. Data collection methods are good and there’s a reasonable plan to secure data from respondents without access to the internet. There is very little discussion of data analysis or characterization of uncertainty. These things are possibly outside the scope of what I was asked to review.	The data analysis methods are discussed in the supporting statement, which wasn't shared with the peer reviewers. We do not feel that a discussion of uncertainty would be particularly useful at this stage. Without preliminary information on the variance of model parameters (e.g., from the pilot), a power analysis would not be meaningful.
Novelty	I don’t see too much novel about the study other than the application of a choice experiment to pricing access to national parks. Choice experiments have been used extensively and there was a study about 10 years ago that looked into pricing access to national parks.	No Response
Sampling Plan	The authors outline a good plan with the appropriate amount of pre-testing and attempts to minimize non-response bias. I’m not sure the plan is sufficient to accurately estimate the demand curve for access to national parks and the revenue impacts of a price change. See final comments for more details.	This appears to be a reference to later comments (e.g., #2 and #4 under "Other concerns or comments") which express concern about the fact that the study relies entirely on SP data and might therefore be vulnerable to hypothetical bias. These later comments suggest that we consider using "travel costs and other expenses to help estimate the demand curve." It seems unlikely that it would be feasible to fold in RP data in this manner, since travel cost calculations would be complicated by the need to address multiple-destination trips and to incorporate flying costs/probabilities.
Understandability	Questionnaire is clear, easy to follow, and logical. I especially like the ability to use search tools and a map to select parks that were visited. I also like the idea of referencing the SP questions to an actual park trip, either in the past or future. This will make it much easier for the respondents to select between the various options in the choice experiment because it will be familiar in their mind.	No Response
Usefulness	Yes, the data should be extremely useful for policymakers and park managers as they try to select an appropriate price. I’m not sure whether it will be publishable in top journals if the demand curve is estimated exclusively with SP data.	No Response
Strengths and Limitations	Again, the sampling plan and questionnaire are well organized and easy to follow. I also think the survey is about the right length. Any longer and you would risk losing the attention of respondents and data quality would be sacrificed.	No Response
Other	Overall, I think this is a well-organized and competent proposal. I like the idea of doing a choice experiment and referencing previous trips when answering questions in the choice experiment.	No Response
Survey Questions	Reviewer Comments	Response
Past/Future Visitation	Why not ask the respondents how important the cost of park access was in the decision to either visit parks in the past or for future visits? For example, if most respondents say “entrance cost is not a concern relative to other expenses, then that reveals something about the elasticity of demand for park access.	The discrete choice experiments are designed to quantify the relative importance of entrance fees in the decision to visit a park.
SP	My biggest concern is hypothetical and strategic bias. It’s easy for respondents to say they would choose the highest-priced option because they never had to actually pay. The previous study done by the NPS found significant hypothetical bias in respondents’ answers. It would be fairly straightforward to compare the number of trips claimed by the random sample of respondents to the number of trips in the general population. My guess is the random sample will show significantly more trips. This might be due in part to systematic non-response bias, but it may also be due to hypothetical bias.	While we agree that it would be useful to compare the number of trips based on the survey to the number of trips estimated by NPS through on-site counts, the comparison will be complicated by the presence of foreign visitors, children, potentially different definitions of visits/trips, etc. Per the comments below, the survey includes questions designed to help identify strategic bias in responses.
SP	Strategic bias could go the other direction. It should be acknowledged that respondents may, on the other hand, tend to choose the cheapest option if they think it will result in lower prices at the gate or for annual passes.	We agree that there could be strategic bias in both directions. The answers to Q19 [past_fee_opinion] and Q62 [revenue_opinion] will help identify these respondents to some degree.
SP	Can travel costs and other expenses be used to help estimate the demand curve? I didn’t see any details about how the demand curve and revenue impacts are going to be estimated.	If we are interpreting this comment correctly, we don't believe estimates of travel costs alone will be sufficient to estimate a demand curve. The demand curve will be estimated using the responses to the discrete choice experiments and discrete choice modeling outputs.
SP	Finally, why [not] do the CE exercise for those that have not visited or don’t plan to visit a park next year? You could choose an actual nearby park. I think this is probably the majority of U.S. households.	We believe that respondents who have not visited and don't plan to visit a park will have difficulty completing the SP exercises and may provide answers that bias the results. While we could collect the responses and exclude them from the analysis, if necessary, this would result in unnecessary respondent burden in our view.

Reviewer #3
Topic Area	Reviewer Comments	Response
Purpose and Methodology	No Response	No Response
Novelty	Nearly 20 years ago, user fee surveys were implemented by the USDA Forest Service to assist the design of the Fee Demonstration Program. However, the study design of the current NPS survey is more sophisticated, both in terms of sample design and the use of a choice experiment that is customized to the “reference visit”.	No Response
Sampling Plan	The sampling plan appears to represent the current state-of-the-art.	No Response
Understandability	Of course, studies of this type must be concerned with “recall bias”. Perhaps respondents could be alerted of this possibility and asked to carefully reflect on their Park experience during the past 2 years prior to answering questions.	We understand that recall bias is potential issue. While a visit to an NPS unit is a distinct type of activity not as prone to recall bias as other activities could be, it is possible that respondents will be prone to include visits that were made outside of the two-year window that is specified at the beginning of the survey. We have added text to top of Q4 specifying the boundary date for the past park visitation period to help clarify this.
Understandability	Beginning in question 18, reference is made to “your group”. I think there may be some ambiguity regarding group questions, and this should be clarified. Is this simply everyone I am travelling with? For example, if I am travelling with my wife and another couple, does group always refer to these 4 people? This could be confusing when asking questions about expenses made by the group (such as in questions 23 and 44). I won’t necessarily know how much is spent by the other couple. But I guess that the expense categories are wide enough that approximation may not be a problem.	The intent is for group to encompass anyone else visiting the park with the respondent. We believe this intent is made clear based on the response options provided in Q12.
Understandability	In the SP questions, I’m supposing that costs are given above and below reference trip costs?	Yes, we intend to vary costs both above and below the levels reported for the reference trip. The cost levels will be set in coordination with the NPS team.
Usefulness	Yes, these data should be extremely useful is setting user fees across various NPS settings.	No Response
Strengths and Limitations	Both the sampling plan and questionnaire represent the state-of-the-art.	No Response
Other	Overall, I am very impressed with the high quality and utility of the information that is likely to be gained from this survey. Thank you for the opportunity to provide a review.	No Response

3