Attachment E3 EDAP Reviewer 3

Review of the Experimental Design and Analysis Plan (EDAP) for the ERS experiment “Conservation Auction Behavior: Effects of Default Offers and Score Updating”

October 25, 2020

Project Summary

The proposed research seeks to test the effectiveness of two mechanisms, derived from findings in the Behavioral Economics literature, to increase the quality of bids received in the USDA Conservation Reserve Program (CRP) auction. The two mechanisms of interest are (1) the use of opt-out defaults, and (2) changes in the timing of information that decision-makers receive. To test these mechanisms in the context of the CRP auction, the authors design an experiment that closely replicates the features of the CRP auction, and they propose the use of two relevant treatments: (1) a high-quality default starting offer, and (2) live updates on the offer score at the point of offer selection. They hypothesize that these treatments will result in auction bids with (i) higher quality conservation practices, and (ii) lower rental prices.

I believe these are meaningful and important hypotheses and the proposed methods for testing them are appropriate. Based on previous literature, there is good reason to believe that these hypotheses are true. Furthermore, improvements in the quality of auction bids within the CRP could potentially result in significant reductions in program costs.

The treatments proposed closely match the hypotheses. The entire design of the experiment closely simulates the setting of the actual CRP auction.

I believe this study has strong external validity. In order to improve the experiment’s external validity, it is designed to closely simulate the actual CRP auction. In addition, half of the experiment will be performed using a subject pool sampled from actual farmers that have previously participated in a CRP auction.

There are, however, some features of the experimental design that I find to be potentially problematic. These concerns are listed below.

Comments Regarding the Experimental Design

1. The largest concern I have is with respect to the payoff mechanism used. The experiment consists of four rounds, with the first round being a practice round. For the remaining three rounds, it is unclear whether participants are paid for each round, or if one of the three rounds is randomly selected for payment. If participants are paid for each of the three rounds, this introduces incentives for participants to alter their behavior across the three rounds. Even if participants are not informed of the outcome of each round until the end of the experiment, they will still have an incentive to vary their strategies across rounds as a method of diversifying risk (e.g., a participant may want to submit a conservative bid in one round in order to provide themselves with a greater chance of winning at least one auction round, while submitting more risky bids in the remaining rounds to have a chance at “winning big”). This greatly complicates the Nash equilibrium, and it creates dependence between rounds, reducing the power of the study. It also greatly increases the expected cost of the study. For these reasons, I strongly suggest randomly selecting one of the three rounds for payment.

2. The experiment will use two separate subject pools: (1) undergraduate students at the University of Delaware, and (2) farmers who have previously participated in a CRP auction. The undergraduate subject pool receives lower incentives and is therefore less expensive, and the farmer subject pool is used to increase the external validity of the study. If there is a difference in behavior between the two samples, the sessions performed using undergraduate students will not provide any additional power to the study. Given that the study will be performed in multiple rounds, the results of the first round can be used to test whether there is any difference in behavior between the two subject pools. If a significant difference is observed, it seems like it might be in the best interest of the study to focus on the farmer subject pool, using the remaining undergraduate student funds to increase the sample size of farmers.

3. It is unclear to me how much information participants are provided with. Are participants aware of the distribution of field types (shown in Table 4) at the beginning of the experiment? Or are they only informed of the expected EBI cutoff score? If subjects are not informed of the distribution of field types at the beginning of the experiment, then there is the possibility of learning across rounds. As participants complete each round (including the practice round), they acquire more knowledge of the possible field types, and this acquired knowledge may change their strategies.

4. If subjects are informed of the set of possible field types, are they also informed of the distribution of field types across subjects? That is, are they aware of the proportion of subjects holding each type of field? If so, I believe the study may have greater external validity if this distribution was calibrated to match the actual distribution of farms that participate in CRP auctions, instead of using a uniform distribution across the nine field types shown in Table 4. The authors state that there is a correlation between land EBI values and the productivity of land (with land with greater environmental impact, and thus higher EBI values, tending to be less productive, thereby having a lower bid cap). The frequencies of the field types shown in Table 4 should reflect this correlation.
   
   Of course, a uniform distribution of field types provides an equal number of observations for estimating treatment effects for each field type. However, this difference in the distribution of field types also changes the strategic problem faced by participants in the auction, which could potentially impact the study’s external validity. Furthermore, if there are certain field types that are relatively underrepresented in actual CRP auctions, we may be less interested in estimating precise treatment effects for them. In fact, to the extent that treatment effects differ across the distribution of field types, the treatment effects of greatest interest are those for the field types that are most frequently accepted into the CRP. That is, a large treatment effect (i.e., a large improvement in the quality of bid) for a field type that isn’t accepted into the program has no effect on the efficiency of the program.

5. Treatment 1: The first treatment implements opt-out defaults into the auction design. The default selections used are always set to the most extreme options available to subjects. That is, the bid down level is set to where subjects will earn nothing, and the best cover practice is selected. It would be interesting to consider some variation in these settings (e.g., using the middle crop practice as the default choice). In the charitable giving literature, providing donors with a suggestion for how much to donate has been found to decrease donations when the suggestions become too large. It is possible that setting the default selections to more modest values could result in higher quality bids. Similarly, using unique default selections for each field type, based on the particular characteristics of each field, may also be an interesting avenue to consider.

6. Treatment 2: The second treatment varies the timing of information received by decision-makers by providing live updates on the offer score at the point of offer selection. An interesting modification of this treatment would be to also provide information about the expected probability of the offer being accepted. One way of doing this would be to use a color for the updated offer score (e.g., the colors red, yellow, and green could be used to signal offers with low, moderate, and high probabilities of being accepted, respectively).

Comments Regarding the Analysis Plan

1. I think it may be worthwhile to interact D1 with either the bid cap or the EBI endowment to see if the effect of using opt-out defaults varies at all by field type. In the charitable giving literature, the use of suggested donation amounts has been found to decrease donation amounts when the suggested value is considered to be too large. Given that the default selections are always set to the highest quality cover crop and highest bid-down level, it seems plausible that subjects who are endowed with more competitive field types may find the suggested selections to be excessively demanding, resulting in a reduction in the quality of their offer.

Other Notes

Table A1 in the Appendix is mislabeled. It states that it shows the expected payoffs for a $50/acre bid cap field at Nash equilibrium. However, the table does not show the expected payoffs; it shows the payoffs for each bidding strategy conditional on the bid being accepted. The values in the table have not been multiplied by the expected probabilities of acceptance for each bidding strategy at the Nash equilibrium.

1