Mini-supporting-statement-UMD-II-Part-A-Final

Mini-Supporting Statement for

Alternative tools for improving CRP cost-effectiveness, Part II

Section A

(Generic Clearance: OMB Control No. 0536-0070)

Please find attached all materials associated with this planned experiment under generic clearance (OMB Control Number 0536-0070). If approved, the experiment will be conducted at The University of Maryland and/or the University of California, Santa Cruz under a cooperative agreement with Professor Peter Cramton, Economics Department, University of Maryland, College Park.

Attachment list:

Attachment A - Email for recruitment

Attachment B - Experimental Design Protocol

Attachment C - Instructions

Attachment D - Questionnaire

Attachment E - Consent form (and Attachment E part 2 – UCSC Exception)

Attachment F - Disclaimer

Overview

The proposed experiment would be a follow-up study, expanding on the study conducted under clearance titled “Alternative tools for improving CRP cost-effectiveness” (hereafter “CRP Study 1”), a completed collection meant to generate preliminary feedback on alternative mechanisms, as we explain below.

The purpose of the project is to compare alternative mechanisms for competitive purchase (auctions). These experiments address an important question in conservation program design. When the goal is to purchase many identical items from a group of individuals, economic theory suggests that procurement costs will be lower if competitive mechanisms, such as auctions or bidding, are used. In an effort to harness competitive forces, the USDA has structured some of its signature conservation programs, including the largest, the Conservation Reserve Program (CRP), as auctions.

In the CRP, farmers participate in a competitive auction by offering to enroll land for a payment. These offers are ranked according to an index of environmental benefit and a cost metric. Each offer is constrained by a parcel-specific bid cap. Both economic theory and practical experience from other types of government auctions (e.g.: timber sales, toxic asset purchase, and communication spectrum sales) suggest that modifying the current auction structure could make CRP more cost-effective. Research (Kirwan et al., 2005)¹ estimates that $380 million or 20% of current annual payments exceed producer’s costs. In the proposed research, we explore options for controlling costs by adjusting the bid cap and/or using alternative auction mechanisms such as reference prices or groupings.

Subjects will be recruited from the student population at The University of Maryland (UMD), College Park, and/or the University of California, Santa Cruz using a standard email (see Attachment A: Email for recruitment). Sessions will be conducted in a classroom laboratory.

For more details on the specifics of the experimental design, please see Attachment B: Experimental Design Protocol.

We also attach examples of the experimental materials to be used in the experiment: the instructions for each of the five treatments (see Attachments C-Instructions) and the post-experiment questionnaire distributed to subjects (see Attachment D: Questionnaire).

Justification

1. Circumstances making the collection of information necessary

Substantial USDA funds for conservation are distributed using competitive mechanisms. Empirical evidence of the optimality of mechanisms is lacking. The standard economic model used to analyze behavior in auctions is the game-theoretic model. Analytical solutions for most real-world auction mechanisms, however, are impossible to derive. Furthermore, the behavior of bidders in auctions often deviates from predictions of the game-theoretic model. It has become standard practice in the analysis of auction design to compare mechanisms using experimental testing. The first step of the testing protocol is to test theoretically appealing mechanisms (loose bid caps, reference price, endogenous reference price, and selection by grouping) against the baseline mechanism (tight bid cap) in a laboratory setting.

2. Purpose and use of the information collection

This experiment would address an important question in conservation program design by comparing alternative mechanisms for competitive purchase (auctions). When the goal is to purchase many similar but not identical items from a group of individuals, economic theory suggests that procurement costs will be lower if competitive mechanisms, such as auctions or bidding, are used. In an effort to harness competitive forces, the USDA has structured some of its signature conservation programs, including the largest, the Conservation Reserve Program (CRP), as auctions.

Both economic theory and practical experience from other types of government auctions (e.g.: timber sales, toxic asset purchase, and communication spectrum sales) suggest that modifying the current auction structure could make CRP more cost-effective. In the proposed research, we explore options for controlling costs by adjusting the bid cap and/or using alternative auction mechanisms such as reference prices or groupings. The research question to be addressed by the study is:

Are there any significant differences in terms of procurement costs between the baseline mechanism (aka tight bid cap) and any of the four alternative CRP enrollment mechanisms (loose bid caps, reference price, endogenous reference price, and selection by grouping)? Initial testing under CRP Study 1 suggested that alternative mechanisms may be successful at price-discriminating, but more testing is needed under more realistic auction scenarios (a principle known in experimental economics as increased “parallelism”).

The primary quantitative outputs of the experiment will be comparisons of expected cost under the five auction regimes (t-tests and non-parametric tests of expected cost), and predictive bidding functions derived from experimental data.

The information from the proposed experiment investigating alternative CRP enrollment mechanisms will be shared with the Farm Service Agency (FSA) of the USDA. The FSA operates the CRP and is interested in exploring alternative enrollment mechanisms (ERS and FSA meet regularly and FSA has expressed interest both in the general notion of alternative mechanisms, and specifically in the mechanisms explored here). The experiment will be used as a preliminary test of three general types of alternative mechanisms. Note that an infinite number of variants are possible, but each variant that is proposed to be tested in this document fall under three labels:

1. Relaxed bid caps (relaxing the maximum price that is imposed by FSA on bidders in the CRP).

2. Grouping-based auctions (auctions which harness competition by asking ex ante similar parcels to compete among themselves before being considered in a national ranking).

3. Reference-price auctions (auctions which harness competition by ranking bids relative to an ex ante estimate of value).

The findings of the experiment will be shared with FSA in regular meetings that ERS and FSA hold. The primary outcomes of interest to both ERS and FSA are the cost-effectiveness of the auction mechanisms, i.e. the cost to procure a fixed number of parcels.

Research findings will be shared with researchers inside and outside the agency through seminars or training sessions, and may also be prepared for presentations at professional meetings or publications in professional journals.

3. Use of improved information technology and burden reduction

As referenced in the parent supporting statement of the generic clearance (pg 4).

ERS will employ information technology as appropriate to reduce the burden of respondents who agree to participate in its research.

ERS plans to use a single general method of information collection, in-person group experiment activities will be held at university computer labs. Computer assisted participation will be used when possible; else, paper and pencil will be used.

The proposed experiment will be conducted in a university experimental economics laboratory at UMD, College Park and/or UC Santa Cruz.

4. Efforts to identify duplication; use of similar information

A literature review was completed. Much of the relevant literature has been produced by the Principal Investigators (PIs) of this project, especially Nathaniel Higgins and Daniel Hellerstein. In addition, we have partnered with Professor Peter Cramton, Economics Department, UMD, one of the foremost market design economists in the world. A white paper co-authored by the ERS PIs and Cramton, Economics Department, UMD reviews the relevant literature, market design concepts, and applicable practical knowledge (Attachment C).

The literature review revealed that there are no relevant studies of similar mechanisms in the context of a multi-unit auction with many bidders. There is a long tradition of experimental auctions more generally (see for example Lusk and Shogren, 2007²) and of the study of the CRP (Latacz-Lohmann and Van der Hamsvoort, 1997³), but there are no studies – experimental or otherwise – directly comparing the proposed mechanisms, which ERS has selected in consultation with FSA. The proposed mechanisms compare to a baseline (an auction mimicking the current CRP) the three mechanisms listed above: “relaxed” bid cap (using an individual-specific maximum bid that is higher in expectation than the baseline), grouping-based auction (also explained in more detail in Attachment B - Experimental Design Protocol), and two variations on a reference price auction (explained in more detail in Attachment B - Experimental Design Protocol). These mechanisms are the ones that FSA is interested in considering, making the testing appropriate at this time.

We view this test as a first step in producing experimentally-valid knowledge (knowledge where the internal validity of the causal mechanism is not in question). Further testing, including field testing, is an obvious next step.

Initial testing of the mechanisms in a slightly different environment – an environment in which all potential bidders were able to submit bids in each round, unconstrained by the possibility that the bid cap could be less than a bidder’s opportunity cost – showed that alternative mechanisms could perform well. The proposed sessions will relax this important constraint.

4. Impact on small businesses or other small entities

No respondents will be small businesses. All respondents for this study will be students recruited to participate in experiments on the campus of UMD, College Park and/or UC Santa Cruz.

4. Consequences of not conducting data collection, or of collecting information less frequently

As referenced in the parent supporting statement of the generic clearance (pg. 5).

The proposed generic clearance mechanism will allow the development of more robust and efficient measures regarding agricultural behavioral economics, with minimal burden, that will benefit subsequent ERS and USDA information collections.

The quality of research that ERS can provide to its stakeholders will be increased if ERS is able to utilize state-of-the art experimental research mechanisms. The quality of quantitative research and its contribution to prospective policy will especially benefit under the proposed generic clearance. Experimental studies are often the only empirical tool that can be used to evaluate economic mechanisms that do not exist in the real world.

4. Special circumstances that would cause an information collection to be conducted so as to require respondents to report information to the agency more often than quarterly

There are no special circumstances associated with this information collection. All responses will be one time responses.

4. Comments in response to the Federal Register Notice and efforts to consult outside the agency

This mini-clearance has not been posted in the Federal Register and so not comments have been received from the public.

We have consulted with – and will work throughout the process with – Peter Cramton, a Professor of Economics at UMD College Park and a leading expert in auction design. Cramton helped directly in the design of the mechanisms to be tested and in the experimental protocol.

4. Explanation of any payment or gift to respondents

As referenced in the parent supporting statement of the generic clearance.

The experiment will be conducted using money payments to participants in the experiment. Consistent with the underlying scientific foundations of the experimental economics, a fundamental requirement of the research methodology is that participants value their time and treat the task of bidding seriously (as referenced in the parent supporting statement of the generic clearance). Each student will compete in multiple rounds (approximately 15, based on performance in the similar experiment CRP Study 1) of 3 different auctions (approximately 45 total rounds) within a 90 minute time period (see Table 1.). They will receive a cash payment based on the experimental market outcome which results from each student’s behavior.⁴ The cash payment will be of uncertain value before the experiments take place, but we do not expect any payments in excess of $50.^5,6 The average payments under the similar experiment CRP Study 1 were about $15. While a maximum cap would be desirable, given that the market equilibrates within the experiment and we are specifically testing a treatment without price caps, we cannot guarantee that someone will not earn more than the $50 if we calibrate the ECUs for a $25 USD average payment. The payments listed here are for the entire 90 minute session, i.e. all auctions participated in by a given individual. Although individuals participate in many rounds within a session, individuals are paid at the end of the 90 minute session based on 2 randomly-drawn rounds for each auction type (for example, in a session for one treatment that includes 12 rounds, experimentalists will draw two rounds at random to be the auctions on which payment is based). This practice prevents any wealth effects from distorting the findings of the experiment.⁷ This practice is standard in the literature.⁸ Therefore, their payments will be based on the sum of 6 randomly drawn rounds: 2 per each auction type. The minimum payment will be 7 USD for “showing up”.

Experience from the experiment conducted under CRP Study 1 suggest average payments of $15.

Table 1. Experimental Design
Session	Treatment*	Average # of rounds per treatment	Max # of rounds per treatment	Time (in minutes)	# of participants
1	1,2,3	12	15	90	16
2	1,2,4	12	15	90	16
3	1,2,5	12	15	90	16
4	1,3,4	12	15	90	16
5	1,3,5	12	15	90	16
6	1,4,5	12	15	90	16
7	1,2,3	12	15	90	16
8	1,2,4	12	15	90	16
9	1,2,5	12	15	90	16
10	1,3,4	12	15	90	16
11	1,3,5	12	15	90	16
12	1,4,5	12	15	90	16
* Random order of treatment within session.

As cited in CRP Study 1, the planned payment amounts are in line with the current payment structure utilized in the experimental laboratory at the University of Maryland, College Park. We developed the payment structure in consultation with Professor Cramton, UMD, and his graduate students who run the lab (and who implement the experiments at the lab). In the three years prior to 2015 (when the aforementioned ICR was proposed), experiments conducted at the Experimental Economics Laboratory at UMD have, on average, provided an hourly payment of 16 to 18 USD for undergraduate students. This amount implies that for 90 minutes of participation in an experiment the approximate payment will be between 24 and 27 USD. That is 25 USD for 90 minutes is in the range of earnings typical for the practices of this laboratory. See the table in the following open link for detailed information for four of the latest research work conducted at the Laboratory:

Recent papers whose experiments were conducted at UMD Exp Econ Lab
Paper	Authors	Average Payment USD	Approximate Duration (mins)	USD Payment / hour	Payment for 1.5 hours
"On the Demand for Expressing Emotions"	Brit Grosskopf; Kristian Lopez Vargas	13	45	17.3	26.0
"Risk Attitudes and Fairness: Theory and Experiment"	Kristian Lopez Vargas	17	60	17.0	25.5
"Multi-Object Auctions with Resale: Theory and Experimen"	Emel Filiz-Ozbay, Kristian Lopez-Vargas and Erkut Y. Ozbay	19	70	16.3	24.4
"Do Lottery Payments Induce Savings Behavior? Evidence from the Lab"	Emel Filiz-Ozbay; Jonathan Guryan; Kyle Hyndman; Melissa Kearney; and Erkut Y. Ozbay	18	60	18.0	27.0

Note carefully that these are average payments. We intend for our average payments to fall within the range here, but the maximum payment will likely be larger than 25 USD and the minimum payment will be less than 25 USD due to the fact that in auctions (real and laboratory) bidders will have different values and as such will each behave differently which causes earnings to vary.

The discussion above is focused on academic literature and common practice in economics, rather than common practice in government-sponsored research. In an effort to use the scientific best practices above in a government-sponsored research study, we propose a plan of reporting that involves special oversight by OMB. Researchers will report regularly to OMB the distribution of payments, including the minimum, maximum, and average payments of each session. Researchers will report these results after the first session, and monthly thereafter. In addition, if any payment to a single participant in excess of $50 occurs, researchers will notify OMB immediately. OMB will reserve the right to pause the collection if large payment outliers become a significant concern during the experiment. In such a case, we will consult with OMB for identifying appropriate methods to address this issue. The experiment may be resumed only after obtaining further OMB approval.

Because it is common to get a number of no-shows and last-minute cancellations for a given experiment session, and because it is important to have a particular number of participants in each session, we plan to “overbook” our sessions by 2 students to ensure we have the right number of individuals (16) to run the experiment. We will do our best to avoid it, but experience suggests that in some cases more people will show up than can be accommodated for a given session. In this case, each extra who show up before the scheduled start time will receive a $7 payment for time and travel and can reschedule for another session. This is common practice among experimentalists. For instance: “Given the high cost of cancellations due to insufficient attendance, most researchers err on the higher side and pay a decent sum to the extra subjects who present themselves on time to avoid alienating them as future recruits….”⁹ Friedman and Sunder do not discuss what is meant by “a decent sum.” The regular practice at each lab around the world varies. We set our payment amount, which is only for those individuals who do not participate in the experiment (and thus the figures listed above for average payments for participants are entirely separate), at the level used by researchers at the University of Maryland, College Park. This is the payment always used at this institution with the population we intend to target with the proposed experiment.

10. Assurance of confidentiality provided to respondent

Respondent data will be protected by the Privacy Act of 1974 (5 USC 552a).

Subjects will sign a consent form at the start of the experiment. They can withdraw from the study at any time. Subjects will receive an ID number that we will use to keep track of their bids and to match bids with background questionnaires used for control in regression analysis. Students will have to sign their names to a receipt but this sheet will be kept separate from the bids.

ERS researchers will not have access to participant names at all, and participant names will not be stored on government computers.

ERS has decided not to invoke the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA). The complexity and cost necessary to invoke CIPSEA is not justified given the nature of the collection; the collection will include a very limited amount of personally-identifiable information (PII), and is hosted in university computer labs, where CIPSEA compliance cannot be assured.

11. Justification for sensitive questions

No sensitive questions will be asked.

12. Estimates of hour-burden including hourly costs

Based on the extensive experience of the principal investigators in running experiments, as well as experiments conducted under CRP Study 1, this laboratory study will take approximately 90 minutes (and possibly less) for each participant to complete. We plan to conduct 12 sessions with 16 participants in each session. Therefore, we expect to use a total of 12*16*1.5 = 288 burden hours to conduct the experiments for this study.

In order to recruit subjects for this study an email will be sent to students, announcing the opportunity to participate. The experimental economics labs at UMD, College Park and UC, Santa Cruz maintain databases of students who have expressed interest in participating in economics experiments and who have shared their email addresses with the economics department. Signup rates to solicitations on this list are 5%.¹⁰ Per previous discussion in A.9, we plan to “overbook” each session by 2 potential participants as backups to ensure we have the right number of individuals to run the experiment. Therefore, if the response rate to our solicitation is exactly 5%, this would imply we would need to email 4,320 students in order to obtain 216 (12*18=216) participants. In order to be sure we obtain a sufficient number of responses, we plan to email participants in waves of 300 (approximately the number necessary to obtain one full session of 16 subjects). We estimate that participants will require five minutes to read the recruitment email and respond that they would like to attend an experiment. This will result in the use of 18 burden hours (5 minutes X 216 affirmative responses). We estimate that it will take individuals two minutes to read the entire email and decide not to respond. This results in a total of 136.8 burden hours (2 minutes X 4,104 nonresponses).

The total number of burden hours used for this study will be the sum of recruitment burden hours and experimental burden hours: 288+ 18 + 136.8 =442.8.

Instrument/experiment	Sample Size	Freq	Responses				Non-response				Total Burden Hours
			Resp. Count	Freq X Count	Min./Resp.	Burden Hours	Nonresp. Count	Freq X Count	Min./Resp.	Burden Hours
Recruitment email	4,320	1	216	216	5	18	4,104	1	2	136.80	154.80
Experiment	192	1	192	192	90	288	0	0	0	0	288
Total	4,320		216			306				136.80	442.80

13. Estimate of other total annual cost burden to respondent or recordkeepers

There will be no capital, operating, or maintenance costs to the respondent as the result of participation in an information collection under this generic clearance.

13. Estimate of costs to the Federal Government

The Federal Government has funded this research through a cooperative agreement with The University of Maryland. The total reimbursable cost of this cooperative agreement will be $66,000. ERS staff time for this agreement will be $20,000. Total cost to the government will be $86,000.

13. Changes in burden hour

This is a new data collection.

13. Plans for tabulation, publication, and project time schedule

If approved, we anticipate this research will be completed within six months of approval. Data will be analyzed and a report will be written in 2016. The results of the experiment will be shared in a memo and in meetings with FSA. The primary hypotheses to be tested will be univariate comparisons of expected cost under the five auction regimes (t-tests and non-parametric tests of expected cost), and estimates of bidding functions.

13. Reasons display of OMB expiration date is inappropriate

No exemption is requested.

13. Exceptions to certification for paperwork reduction act submissions

No exceptions to certification are requested.

1 Kirwan, Barrett, Ruben N. Lubowski and Michael Roberts, (2005), How Cost-Effective Are Land Retirement Auctions? Estimating the Difference between Payments and Willingness to Accept in the Conservation Reserve Program, American Journal of Agricultural Economics, 87, (5), 1239-1247

2 Lusk, Jayson L. and Jason F. Shogren (2007). Experimental Auctions. Cambridge University Press, Cambridge, UK.

3 Latacz-Lohmann, Uwe and Carel Van der Hamsvoort (1997). “Auctioning Conservation Contracts: A Theoretical Analysis and an Application,” American Journal of Agricultural Economics, 79: 407-418.

4 The number of auctions participated in by each individual within a session will be identical, but may vary across sessions. For more details, please see Attachment B - Experimental Design Protocol.

5 The maximum payment under CRP Study 1 was approximately $30.

6 Because auctions are competitive, it is not possible to directly limit the earnings that can be generated by participation without an explicit limit – a price cap. Because this experiment includes auctions without price caps as a very explicit treatment, it is not possible to guarantee that payments greater than $50 will not be made. Competition, however, is an excellent check on high payments. All auctions will be competitive and payments above $50 will be exceedingly rare. Furthermore, the payment design can be changed after the completion of a session, further reducing payment risk. That is, if in live testing – which by definition cannot be conducted at scale with 16 bidders until PRA clearance is received – individuals earn amounts in excess of the planned maximum, the rate of exchange between “experimental dollars” (the currency used in the experiment and displayed onscreen to the experiment participants) and $U.S. can be modified to ensure that payment stay within the proposed range in future sessions.

7 Wealth effects are the theoretical changes in behavior that occur after a given individuals’ wealth increases. Since the CRP is a “one-shot” auction – there is only one CRP auction conducted at a moment in time, not a series of CRP auctions – it is necessary to eliminate wealth effects.

8 See “Incentives in Experiments: A Theoretical Analysis” by Azrieli, Chambers, and Healy. http://www.econ.ucsb.edu/about_us/events/seminar_papers/Healy.pdf.

9 D. Friedman and S. Sunder, Experimental Methods: A Primer for Economists, Cambridge University Press 1994, p. 54.

10 Personal communication with UMD cooperator.

7