Attachment G - NASS Statistical Review Comments

Attachment G

National Agricultural Statistics Service Review of Supporting Statement Part B

Methodology Review of the Economics Research Service’s (ERS) Generic Clearance to Conduct Experimental Economic Research from FY2015 to FY2016.

By Peter Quan and Beverly Gaucher

The Economic Research Service (ERS) is renewing an existing generic clearance that allows them to conduct experimental research into the cost effectiveness of alternative policy mechanisms for nutrition and conservation. The research is only intended to be formative and results will be labeled as exploratory in nature.

The target populations are farmers and participants in USDA nutritional programs. However, due to the nature and goal of this project, ERS plans to conduct convenience and purposive sampling. Respondents will consist of university students and certain groups of farmers willing to participate in a laboratory and field setting, respectively. Note, that since these are non probability samples, the precision of the sample results is not measureable and the results may be biased.

It is important to note that the sample be representative of the population of interest in order to safely draw conclusions about the population from the experiments. Using sampling frames such as the university list, serves to randomly contact subjects via email or letters is a good way to get a representation of the university population. When using different sampling frames, always consider the population the sampling frame represents in order to control for confounding variables. For instance, as the experiment is interested in agricultural policy, targeting farmers and agricultural students for the subject pool is wise.

The document states that the testing will not incorporate any sampling methodology. This statement is unclear, but using strata, cluster analysis or multistage sampling can be worthwhile when attempting to generate a representative sample. The document also states that convenience sampling will be used. Convenience sampling is not always representative of the target population. Perhaps cluster analysis would target a more representative sample. In any case, when using convenience sampling, always be aware of potential confounding variables. The use of control variables to account for the potential influence of known confounds is an excellent practice.

Partnering with university professors or statisticians from the very beginning of the design of the experiment is an excellent idea. A sound design of an experiment provides the basis for all subsequent analyses and conclusions.