Pretest Summary Slides 4.2.2014

Pretest Summary
•

EPA is conducting a stated preference survey to estimate benefits of improving water
quality in the Chesapeake Bay and lakes in the Chesapeake Bay Watershed

•

The stated preference survey will be mailed to randomly selected households using a
choice experiment approach with the following environmental attributes
- Water clarity in the Bay
- Striped bass population
- Blue crab population

•

- Oyster population
- Lakes with low algae levels

Our original experimental design included two baseline scenarios in three geographic
strata. In response to public comments we added a third baseline and another reference
year for attribute improvements to the pretest design
Baseline Conditions

Time Horizon

Geographic Strata

- Improving*
- Constant
- Declining

- 2025
- 2040*

- Bay States
- Watershed States
- Other Eastern States

* Added in response to public comments

Internal and deliberative

1

Pretest Administration Details
• The sample size for the pretest was based on the following:
– 6 survey versions and 3 geographic strata result in 18 sampling cells
– Target of 50 completes per sampling cell
– Anticipated effective response rate of 28%
–  50 .28 18  3, 214 households in the sample (rounding resulted in a sample
of 3240)

• EPA received approval for the pretest on Sept 17, 2013
• Began printing materials as soon as the government shutdown ended on
October 17th

• Pretest mailing schedule
–
–
–
–
–
–

Preview letter
First survey
Reminder postcard
Second survey
Final reminder
Non-response follow up
Internal and deliberative

November 13
November 19
November 26
December 9
December 16
January 16
2

Response Rate Calculation
• Accounts for: completed surveys, refusals, deceased,
physically or mentally incompetent, ineligible addresses,
non-respondents
• Am. Assoc. for Public Opinion Research (AAPOR)
– RR1: only removes ineligible addresses
– RR3: removes ineligibles PLUS a portion of non-respondents
based on an estimated eligibility rate (e)
Bay States

Watershed
States

Other Eastern
States

Overall

RR1

31.7%

26.4%

22.7%

26.9%

RR3

38.1%

33.3%

29.7%

33.7%

Internal and deliberative

3

Sample Summary Statistics
Typical of mail surveys, our sample includes smaller proportions of females and minorities
and tends to be older than the sample frame.
Sample

Population1

H0: Sample% = Population%

Male

55.2%

47.9%

4.09*

Age 20-29

2.82%

17.45%

-24.07***

Age 30-39

12.2%

16.5%

-3.24*

Age 40-49

16.7%

18.8%

-2.16*

Age 50-59

25.5%

17.9%

4.15*

Age 60-69

24.19%

15.76%

5.37***

Age 70 & over

22.7%

12.6%

6.77*

Black

11.2%

18.2%

-5.70*

Hispanic

4.2%

12.5%

-10.71*

College Degree
49.9%
48.1%
0.43
Average and median respondent’s household income falls between $50,000 to $74,999. The “median” population
income is $52,627.

Respondents’ familiarity with the Chesapeake Bay and Watershed lakes
Sample

Heard of the Chesapeake Bay before receiving the survey

90.6%

Has recreated at the Bay in the last 5 years

34.9%

Has recreated at a Watershed lake in the last 5 years
Internal and deliberative

32.4%
1. U.S. Fact Finder census data for entire sample frame

4

Item non-response
% Missing
30.00%
25.00%
20.00%
15.00%
% Missing

10.00%
5.00%
Q1
Q2_1
Q2_2
Q3_1
Q3_2
Q4
Q5
Q6
Q7
Q8
Q9
Q10_1
Q10_2
Q10_3
Q10_4
Q10_5
Q10_6
Q10_7
Q10_8
Q11_1
Q11_2
Q11_3
Q11_4
Q11_5
Q11_6
Q12
Q13a
Q13c
Q14
Q15
Q16
Q17
Q18
Q19
Q20
Q21
Q22
Q23

0.00%

•

Choice question nonresponse about 5.5%
– 19 (2%) respondents skipped
all three choice questions
– 716 (91%) respondents
answered all three choice
questions

Internal and deliberative

•

Open ended questions
asking where recreation
trips to the Chesapeake Bay
were taken

5

Debriefing Questions
•

•

•

Fourteen Likert scale questions following the choice experiments grouped into Q10 and Q11
–

Evaluate quality of responses to choice questions

–

Probe motivation for WTP

Respondents still answering questions thoughtfully late in the survey
–

Only 0.003% and 2.3% of participants responded the same for all Likert scale questions in Q10 and Q11,
respectively

–

2.8% and 4.3% said “don’t know”

–

4.7% and 5.0% skipped all of Q10 and Q11, respectively

Comparison with Banzhaf et al. (2006) Adirondacks SP study
Chesapeake
Pretest

Adirondacks

Did not vote as if household would face costs

7%

N/A

Did not believe improvements would be achieved

7%

37%

Costs should not be a factor

36%

25%

Against taxes/govt. spending

30%

21%

My household should not have to pay

36%

19%

32% - 46%

10% - 59%

Considering factors other than attributes in choice questions
Internal and deliberative

6

Probing for Scenario Rejection and Protest Responses
• Respondents generally accepted scenarios and responded to choice questions as if
the scenarios posed were real and consequential
• Some indication of protest responses
• Contradicted by answers to choice questions
• Could be improved by rewording some prompts
Question Prompt

Strongly
Disagree

Strongly
Agree
Don’t
Know

Missin
g

17% 55%

5%

6%

12% 21% 48%

8%

6%

I am against any more regulations and government
spending.1

21% 13% 25% 10% 21%

7%

6%

My household should not have to pay any amount to
improve Bay Waters and Watershed Lakes.2

17% 17% 21%

25%

5%

6%

It is important to improve waters in the Chesapeake Bay
Watershed, no matter how high the costs.3

15% 11% 27% 17% 19%

7%

5%

1

2

3

I voted as if my household would actually face the costs
shown in the questions.

5%

2%

9%

I voted as if the programs would actually achieve the
results shown by [YEAR]

4%

2%

4

9%

5

1. Since our payment vehicle did not include government spending we will change the phrasing for the full survey
2.Internal
114and
respondents
deliberative agreed w/ statement, but voted for policy option at least once
3. 67 of respondents agreed w/ statement but still chose status quo at least once
7

Motivation for WTP
• How much is WTP motivated by bequest and option value?
• To what extent did factors other than those presented in
the choice questions influence respondents’ votes?
Question Prompt:
Did the following affect your vote?

Strongly
Disagree

Strongly
Agree

1

2

3

4

5

Don’t
Know

Missing

Changes in the quality and price of
seafood

19%

12%

24%

17%

15%

8%

5%

Impacts on the economy and jobs

10%

9%

22%

24%

22%

7%

6%

Improving the environment for others

5%

4%

14%

26%

38%

7%

6%

Water quality improvements to lakes
outside the watershed

11%

9%

20%

21%

24%

9%

6%

Preserving the environment for future
generations

5%

3%

11%

22%

48%

5%

6%

Trips I may take to the Chesapeake Bay or
Watershed lakes in the future

19%

11%

19%

15%

18%

12%

6%

Internal and deliberative

8

Screening responses for analysis
•

Identify protest responses, hypothetical bias, etc. based on both responses to
choice scenarios and debriefing questions

•

Protest responses
– Against government regulation and spending
– Household should not have to pay for improvements
– Chose status quo in all choice scenarios

•

Hypothetical bias
– Did not vote as if household would actually face costs
– Chose highest cost option in every choice scenario

•

Warm Glow
– Improvements important no matter how high costs
– Chose highest cost option in every choice scenario

Full
sample

Protest
Responses

Hypothetical
Responses

Warm Glow
Responses

Protest, Warm Glow,
or Hypothetical Bias

784

97

4

29

127

100%

12.4%

0.5%

3.7%

16.2%

Internal and deliberative

9

Comparing Results Across Baseline Versions

Chi-sq Tests of Differences in Marginal WTP
clar

H0:(1)=(2)
1.19

H0:(3)=(2)
0.49

H0:(1)=(3)
3.39*

lake

2.2

0.01

0.26

Decreasing
(1)

Constant
(2)

Increasing
(3)

(1)=(2)

(2)=(3)

(1)=(3)

Protest

9.12%

13.18%

15.95%

t=1.51

t=-0.69

t=-2.20**

Hypothetical Bias

0.70%

0.39%

0.41%

t=-0.49

t=-0.05

t=0.43

Warm Glow

5.26%

3.10%

2.49%

t=-1.25

t=0.41

t=1.62

% respondents
exhibiting:

Internal and deliberative

10

Choice Questions and Econometric Model
Random utility model (RUM)
(v(Xi, D,Y-Fi) + εi) ≥ (v(Xj, D, Y-Fj) + εj)
v(.)=utility
εi = stochastic component of utility
Y = income
D = household characteristics

• Estimated several
conditional logit
regressions using pretest
data
Internal and deliberative

11

Validity Checks: Scope Tests
Pooled Model: all geographic strata, baselines,
and reference years
• Scope – signs and statistical significance of coefficient
estimates are often as expected
• We are able to improve the precision of our estimates
using the screening criteria

Internal and deliberative

12

Validity Checks: External Scope Tests
External scope
– Estimate each choice question separately
– Estimates based only on cross-sectional variation

– Coefficient signs are often as expected, suggesting that
respondents are not just being internally consistent

Internal and deliberative

13

Validity Checks: Theoretical Validity
• Positive coefficient on high income-cost interaction
indicates wealthier people are more likely to choose
higher cost options
• Monotonically decreasing coefficients on cost level
dummies reflects decreasing marginal utility of income

Internal and deliberative

14

Validity Checks: Theoretical Validity (2)
• Positive coefficient on user-attribute interactions shows that
users of the Bay are generally willing to pay more to improve
Bay attributes

Internal and deliberative

15

Validity Checks: Alternative Specific Constants
• Alternative specific constants
– No evidence of status quo bias (i.e., warm glow or cold feet)
– Tendency for respondents to choose Option B and the
cheapest alternative
– May be due to the lack of a full orthogonal design in the
pretest
 Main survey will have full orthogonal design
 If these confounding influences still exist, they can be
controlled for by inclusion of these dummy variables

Internal and deliberative

16

Comparing Reference Year Versions (2025 v. 2040)
Annual Household WTP for a 10% improvement in Bay Water
Clarity
 Comparison of preliminary WTP estimates across 2025 and 2040
show that respondents are discounting benefits that occur farther in
the future
 Estimates from 2040 version of the survey are less precise
Annual Household WTP for a 10% increase in Low Algae Lakes Watershed States Only

WTP for Low Algae Lakes is not statistically different from zero
for either reference year

Internal and deliberative

17

Evaluating the 2040 Time Horizon
•

Public comment: Some improvements from the TMDL will not be fully realized until
after 2025.

•

In focus groups and cognitive interviews long time horizons increased the incidence
of scenario rejection.

•

From our response to public comments: “We will include debriefing questions on all
surveys to test for scenario rejection of the type we encountered in focus groups. If
the pretest results show that a disproportionate number of respondents reacted
negatively to either reference year we will reconsider the split sample design for the
full survey.”

•

Debriefing Question: “I voted as if the improvements would actually be achieved by
2025/2040”
7.5% of respondents to the 2040 version of the survey disagreed, compared with
4.7% of respondents to the 2025 version (two-sample t-test p-value = 0.108)

Internal and deliberative

18

Evaluating the 2040 Time Horizon (2)
• Preliminary WTP estimates for improvements in Bay Water Clarity
– More precise estimates in the 2025 sample
– Results indicate people are discounting at reasonable rates;
implied discount rate is about 3%
• Standard practice to choose a shorter timeframe and discount
benefits accordingly (e.g. Alberini et al. 2004, Banzhaf et al. 2006,
Cameron and DeShazo 2013)
• Our preference is to collect data that will provide the “cleanest”
estimate of WTP and discount future benefits appropriately.
• Dropping the 2040 time horizon from our experimental design
would reduce the cost of conducting the main survey and the
burden placed on respondents by half.
Internal and deliberative

19

Non-response Survey Administration
• Sent to 900 households who did not respond to pre-test
– Randomly selected by strata
– Included $2 incentive payment and “Please Respond Within 2 Weeks”
stamp on envelope
– Brief questionnaire (4 pages total)

• 144 responses (16.2% response rate)
– 38 received from Bay States
– 49 received from Watershed States
– 57 received from Other States

Internal and deliberative

20

Non-response Survey Results
Survey

Non-response

Heard of the Chesapeake Bay

91%

85%

Seen the Chesapeake Bay/Lakes

58%/49%

45%/43%

Recreated at the Chesapeake Bay/Lakes

35%/32%

23%/19%

Against more regulations and spending

31%

36%

Should not have to pay to improve Bay or Lakes

34%

43%

Important to improve Bay no matter the cost

36%

39%

Income (“median” based on mid-point of range)

$62,500

$62,500

Male

56%

51%

Black

11%

6%

Hispanic

4%

8%

College Degree

49%

45%

Internal and deliberative

21

Proposed Changes for Full Sample
• Survey Edits
– Minor changes to phrasing of some debriefing
prompts
– Minor text changes to description of Lakes
attribute + changing question 5 to reinforce that
information

• Experimental Design
– Drop the 2040 time horizon
– Scale back improving baseline version of the
survey to Bay States stratum only
Internal and deliberative

22

Changes to survey text
Page 4
“Pollution reduction programs already in place to limit nutrients and sediment
flowing into the Chesapeake Bay also help limit algae growth keep algae levels low
in Watershed Lakes.”
Old Question 5. How do the predicted conditions for the Chesapeake Bay and the
Watershed Lakes in [year] compare with what you expected?
 I had expected conditions in [year] to be better than what is predicted
 I had expected conditions in [year] to be worse than what is predicted
 I had expected conditions in [year] to be about the same as what is predicted
 Don’t know
New Question 5. If you were taking a recreational trip to a lake, which would you
prefer?
 I would prefer a lake with low algae levels and clearer water
 I would prefer a lake with high algae levels and greener water
 I don’t have a preference, either type of lake would be fine
 I don’t know

Internal and deliberative

23

Full Survey Schedule
• Goal is to have the survey in the field before
summer vacation…
Dates
Printing and preparation

March/April 2014

Mail Preview Letter

April 28, 2014

Mail Final Reminder

May 26, 2014

Conduct Non-response bias study

June 9, 2014

Begin data analysis

July 2014

Internal and deliberative

24