0243 - 2020 Census Content Test - Supporting Statement B - 6-18-2020

Supporting Statement

2020-2021 CENSUS OF AGRICULTURE CONTENT TEST

OMB No. 0535-0243

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

Up to 100 cognitive interviews and 100 usability interviews will be conducted under the Generic Testing Docket (OMB Number 0535-0248) prior to the work described here. Those interviews will be conducted with operations selected from NASS’s list frame based on characteristics about their operation including geography, the types and amounts of commodities grown, size of the operation, and other characteristics.

The Census of Agriculture Content Test described in this docket will consist of three phases. Phase One will consist of a mail out test with telephone follow-up, using a stratified random sample of 36,000 farm and ranch operations from a universe of approximately 2.2 million agricultural operations. The mailing will also provide instructions for how sampled operations may respond on the Internet. Telephone follow-up contacts will be monitored for quality assurance and to assess any problems with the telephone data collection instrument. Stratification will be used to ensure sufficient coverage of various sizes, locations, and types of agricultural operations. Samples will be drawn to include operations with specific types of commodities or for whom new or revised questions apply. The sample will be divided into control and treatment groups to test alternative versions of the questionnaires and/or data collection strategies. A minimum 50 percent response is expected, which will be adequate for the test design.

Phase Two will consist of up to 200 strategically selected agricultural operations from a universe of approximately 2.2 million agricultural operations. Sample operations will be asked to participate in cognitive interviews or usability interviews. The sample will consist of some agricultural operations that completed the questionnaire in Phase One, as well as some additional operations. The cognitive interviews and usability interviews conducted will be used to improve the overall 2022 Census of Agriculture questionnaire (and Internet data collection instrument) by allowing NASS to follow-up with respondents to explain unusual responses and to ascertain their comprehension of the questions. Additional cognitive and usability interviews will also be conducted to evaluate any additional changes to the Internet instrument or for additional reporting issues identified or unresolved in earlier phases. A minimum 80 percent response is expected, which is adequate for cognitive interviews.

Phase Three will consist of a stratified random sample of 15,000 farm and ranch operations from a universe of approximately 1.5 million agricultural operations believed to have Internet access. This phase will be focused on evaluation of additional or proposed enhancements to the Internet instrument. Stratification will be used to ensure sufficient coverage of various sizes and types of agricultural operations. The sample will be divided into control and treatment groups to test alternative versions of the Internet data collection instrument and methods to increase on-line response. A minimum 25 percent response is expected, which will adequately support the test design.

2. Describe the procedures for the collection of information including:

• statistical methodology for stratification and sample selection,

• estimation procedure,

• degree of accuracy needed for the purpose described in the justification,

• unusual problems requiring specialized sampling procedures

Including a control group, Phase One and Phase Three Test 1 has five groups, Phase One Tests 2 and 3 have two groups each and Phase One Test 4 has three groups.  The number of paired tests for each Phase-Test combination is one less the number of groups.  Phase One and Phase Three group sample sizes for each of four tests and one test, respectively, were determined to be 3,000 each.  Hence the overall Phase One and Three sample sizes are 36,000 and 15,000, respectively.  The resulting margin of error for each test was derived using the following formula:

A one sided multivariate hypothesis test (H₀: p_i > p_c vs. H₁: p_i ≤p_c, for sample size n_i =n_c = n_group) was used to compute the margin of error with fixed sample size, power and confidence level.  Bonferroni-corrected P-values are obtained by multiplying each P-value by the number of tests (m) conducted. To conduct m tests and attain an experiment-wise significance level alpha, then – using the conservative Bonferroni adjustment – the margin of error is:

Where:

p_i, p_c is the response rate for treatment group i and control group c, respectively.

   ∆ is the margin of error (P_i-Pc).

   n_group is the sample size for each treatment or control group.

σ_i², σ_c² are the variances of the response rate for treatment group i and control group c, respectively.

   M is the number of paired tests.

   α: is the Type I error.

   Β is the Type II error.

   Power is 1-β

i is 1,2,3 and 4 for Phase One, Test One and Phase Three, Test One, 1 for Phase One, Tests Two and Three and 1 and 2 for Phase One Test Four.

A binomial approximation of the normal variance (based on an estimated minimum response rate ( of 50 percent) was used to calculate a variance and with  

The resulting margin of error using a fixed sample size of 3,000 per group, 95 percent confidence level and 80 percent power presented are all within five percent (Table 1).

Table 1 Content Test Margin of Error Results by Phase-Test

The sample size for the cognitive interviews and usability interviews includes cognitive interviews for several purposes from fall 2020 through approximately fall 2022. Samples will not be random and results from this phase of the testing will be qualitative in nature, therefore there is no statistical criteria for the sample in this phase. First, there will be a small number (20 – 30 interviews) used between now and October to make final changes to the questionnaire used for Phase 1 (although most of the cognitive interviews for this purpose will be conducted under the Generic Testing Docket (OMB Number 0535-0248)). Second, there will be several targeted cognitive interviews conducted as follow-up interviews to Phase 1 (approximately 100 interviews). These interviews will target respondents who had problems with the paper or Internet version of the questionnaire (determined by inconsistent or unexpected responses) or reported differently than expected (determined by comparing reported data with list frame control data). Third, usability interviews will be conducted prior to and after the Internet test (approximately 100 interviews). The Internet instrument has been revised from previous versions of the Internet questionnaire to include additional functionality not available in the 2017 Census of Agriculture. Proposed modifications to the Internet instrument will be the focus of usability testing.

Due to the length of the questionnaire and the extensive nature of some of the probes used for some sections and some operations, the cognitive interviews target only specific sections of the questionnaire. Individual cognitive interview respondents will be asked to complete only some sections of the full form. This requires more cognitive interviews overall, but allows us to limit the interviews to 1.5 – 2 hours each.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Because results are not intended for publication, the anticipated response rate of 50 percent for Phase One and 25 percent for Phase Two are sufficient to provide sound basis for determining final 2022 Census of Agriculture instrument design.

4. Describe any tests of procedures or methods to be undertaken.

The methodology for this information collection is based on previous content tests and follows accepted practices for surveys of this nature. The treatment groups proposed will differ in the content, timing and methods of contacting selected operations in data collection. This may include changes to the timing and frequency of contacts, alternative messaging in the contacts, changes to the format or content of the contacts (for example, to include paper copies of the questionnaire or access instructions for the Internet instrument) or other modifications to the data collection procedures used in previous censuses of agriculture. These may be designed for use in general census of agriculture populations or targeted subpopulations. Alternative treatments will be evaluated for differences in response rates, data quality, costs or other factors.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Census of Agriculture Content Test methodology and procedures are overseen by the Research and Development Division; Chief Cognitive Research Methodologist is Jaki S. McCarthy, 202-690-2389, and Methodology Division, Standards and Survey Development Methodology Branch Chief, Dan Beckler, (202) 720-8858.

The sample size is determined by the Sampling, Editing, and Imputation Methodology Branch, Methodology Division; Branch Chief is Mark Apodaca, (202) 720-5805.

Data collection is carried out by NASS National Operations Division (NOD); the NOD Director is Joseph Prusacki, (314) 595-9501.

The overall Census of Agriculture program is overseen by the Census Planning Branch, Census and Survey Division; Branch Chief is Donald Buysse, (202) 690-8747.

.

June 2020

1