2021 Census of Agriculture – Cognitive Testing Protocol
Please read this entire document before conducting any interviews to become familiar with the data collection procedures, survey questions, and probes.
Background
Census content tests are conducted prior to the censuses of agriculture to evaluate and test a number of factors affecting the census program. They are conducted to evaluate factors affecting response and data quality including format and design of the instrument, new content items, changes to question wording, respondent burden, attitudes affecting response, selected procedural changes, and changes in respondent reporting which may lead to misclassification of a farm.
Research Goal
The primary objective of the cognitive interviewing is to evaluate respondents’ comprehension of the questions and their ability to provide valid responses. This test will be testing the 2020 Content Test questionnaire.
Target Operations:
AUM (Both those renting to and renting from; large, small, grazing associations)
Irrigated Cropland
Irrigated Pasture
Hay
Short Rotation Woody Crops
Sod
Vegetable Seeds
Local Foods
CRP
Sections Being Tested:
Section 1 Acreage in 2020
Section 2 Land Use in 2020
Section 4 Land Use Practices (irrigation and precision ag)
Section 5 Crop Insurance, Conservation Reserve Program, and other Government Programs
Section 6 Type of Organization
Section 8 Hay and Forage Crops (respondent fills out but no probing)
Section 9 Cultivated Christmas Trees, Short Rotation Woody Crops –Question 3
Section 10 Field Crops
Section 11 Nursery (vegetable seed, crops formatting)
Section 12 Vegetables, Potatoes, and Melons (vegetable seed)
Section 25 Practices (internet question)
Section 26 Food Marketing Practices
Section 30 Farm labor – unpaid workers
Questionnaire formatting: we have several different formats on the form for crop lists. See field crops, nursery, vegetables, etc. Our main concern is if respondents can locate their crops names and codes.
More detailed information on each section that is being testing can be found at the end of this protocol.
Recruitment
You will be responsible for recruiting the respondents and scheduling the cognitive interviews. Respondents will be selected based on specific characteristics. They will only have to complete selected sections of the report form. Please use attached spreadsheet for the number of interviews and target criteria for recruitment. If you need assistance identifying operations with the following criteria, contact Heather Ridolfo. When calling to set up interviews, you can use the recruiting script below, and insert the appropriate criteria into the script.
Respondents should be informed of the intent of cognitive interviews during the recruitment process and that their participation is voluntary. Respondents should also be told the interviews will be conducted via Zoom/WebEx and are expected to last approximately one hour.
Cognitive Interview Procedures
1. Interviewers should have the following available during the interview
Interview guide
Electronic copy of the questionnaire
2. Record the time at the start and end time of the interview.
3. Introduce yourself to the respondent and briefly explain the task to the respondent. Please use the script at the top of the interview guide.
4. After explaining the task, share the Census questionnaire with the respondent via Zoom/WebEx.
5. Go to the first section of the questionnaire that your respondent will need to complete (see assignments). Note that in some sections, the respondent does not need to answer all questions. See details below. This will save you time during the interview and allow you to focus on the questions we are testing.
The COA questionnaire we are testing is dated 2020. Please tell respondents that this survey would be mailed in January 2021 and they would report for the previous calendar year. If there are items they are not able to answer due to the date, they can provide estimates for this testing. But probe on how they would normally determine their answer.
6. Ask the respondent to tell you how they would complete the form; fill out the answers as they provide them. Ask them to tell you what boxes they would place answers in, not just what the answers would be. Try not to lead the respondent. We want to identify any formatting issues to the extent possible. Save a copy of the filled out questionnaire and send back with your interview notes.
7. DO NOT help the respondent when they are completing the questionnaire. If they ask questions or for clarification, tell the respondent to complete it as they would if you were not there. Also keep in mind that we are NOT interested in the particular answers they provide, but how they provide them and whether or not they have any questions or problems when they are answering. At the end of the interview, after all probes have been asked, you may explain how particular questions should be interpreted.
7. After they have completed each section, begin asking the probe questions for that section. You only need to probe on the sections you are assigned. When there are multiple probes listed together, ask each probe question and get a response before asking the next question. Even if a respondent screens out of a section, there are still probes you should ask. Some probes will be asked only of those respondents who screened into sections or answered particular questions a certain way (e.g., Land reported in Item 1). There are instructions in parentheses before the probe questions in these cases. Remember to ask probes even if it appears that the respondent understood the question. For any items that the respondent should have reported but left blank, probe to find out why they did not complete that item.
Emergent probes may be asked as needed. Emergent probes might include questions such as: You seemed to have difficulty when answering this question, tell me what you were thinking. Was there something confusing about that question? You seemed to have difficulty selecting a response, tell me why this was; why did you select the answer you did? When using emergent probes, please include the probe questions used in your notes. You may not be able to probe on all survey questions in the time allotted. Please be mindful of time and be sure to probe on questions that respondents had difficulty answering.
8. End of Interview. Thank the respondent.
9. Notes. Interviewers are expected to spend up to one hour per interview writing and summarizing their notes from that interview. Please type your notes in the fillable interview guide. Do not include any personally identifiable information (PII) in your notes (e.g., name of respondent, name or operation, address, phone number). Detailed notes for each interview and the scanned questionnaire that the respondent filled out should be emailed to Heather Ridolfo by July 30, 2020.
10. Record your time under project code 643.
Any questions or concerns should be directed to Jaki McCarthy at 202-690-2389 or jaki.mccarthy@usda.gov or Heather Ridolfo at 202-692-0293 or heather.ridolfo@usda.gov.
Recruitment Script
Hi, my name is _________________ and I’m calling from the USDA’s National Agricultural Statistics Service. We are contacting operators to ask for help in testing the 2020 Census of Agriculture Content Test report form. The Census of Agriculture is the leading source of statistics about the Nation’s agricultural production and the only source of consistent, comparable data at the county, State, and national levels. We can meet remotely. I will send you a link to website where I can share my screen with you. That way we can both look at the questionnaire at the same time. Is this something that you’d be willing to speak to us about?
In order to improve the quality of the data and the ease of reporting, we would like to get your feedback on selected sections of the report form. This would take about an hour of your time and I could schedule a time that is convenient for you. During this time, I will ask you to complete specific sections of the report form and gather your feedback on how we can improve the questionnaire.
The criteria for this testing is that you have [insert criteria] and access to a computer when we meet.
Do you have [insert criteria]?
Do you have access to a computer?
<if yes to both>, This means you are eligible to help us out. Could I schedule a time to meet with you and get your input on this report form?
I have the following days and timeframes open.
<dates, times>
Which of this days and times works best for you?
What email address can I reach you at?
I will call you to confirm the appointment a day or two before. If you need to change the appointment time, please call me back at <phone number>.
Respondents should complete all questions in this section.
We are testing this section for any major issues that stand out. Also, we want to ensure that respondents are reporting AUM and ACEP correctly. This section will be tested by all interviewers.
AUM (Animal Unit Month)
Operations that pay AUM to use land should not report this land in Question 1-4. It should only be accounted for in Question 6. Operations that own land that is rented to others on an AUM basis should report this land in Question 1 only.
For testing, some interviewers will target operations that rent land on an AUM basis and operations that own land that is rented out to others on an AUM basis
ACEP (Agricultural Conservation Easement Program)
This is a new federal program. Under this program, owners sell a portion of their rights to the land; they do not rent this land to others. Land in this program should be reported under land owned (Question 1), not under land rented to others (Question 3). We are probing on this program to see if respondents are familiar with it and where they think it should be reported.
Respondents should complete all questions in this section.
We are testing this section for any major issues that stand out and to verify that the totals in Sections 1 and 2 are equal. Home gardens was also added to the list of includes for All Other Land. This section will be tested in all interviews.
For this section respondents only need to answer Question 3 (precision agriculture) and the Question 4 (irrigation questions).
Precision Agriculture
A new question on precision agriculture was added to this section (Question 3). We want to understand how respondents interpret the term precision agriculture, and make sure they are providing valid responses.
Irrigation
The irrigation questions (Question 4) were revised. In the past COA respondents often misreported acres as irrigated when they were not. We want to examine if the new questions better capture irrigation. We are interested in both respondents’ ability and intentions to irrigate, not just if they irrigated.
We want to understand how people are getting water to their fields and make sure they are providing valid responses to these questions. Potential issues to look for include people who irrigate but do not consider themselves as having irrigation equipment or a system, other misunderstanding of irrigation (e.g., if they are directing rain water off of their fields they are not irrigating them but if redirected rain water to their fields they are irrigating them), irrigation methods that are not listed in these questions.
Respondents should complete all questions in this section.
The name of this section was changed. There was concern that respondents were not reporting CRP here because CRP was not mentioned in old section title. Some interviews will target those with CRP to see how they react to this section. Respondents should complete this entire section.
Respondents should complete all questions in this section.
Question 3 in this section has been modified. We are testing this section to determine if respondents can accurately report the legal status of their farm operation using the new categories, and if they can follow the formatting. Respondents should complete this entire section but will only be probed on question 3.
Respondents should fill out this section; however, we are not testing this section. One thing to be aware of is that hay grown for seed should be reported in field crops, not in this section.
Respondents should complete all questions in this section.
Short rotation woody crops
A specify box was added to Question 3. This was added to help with editing. Many respondents were misreporting things such as timber, boxwoods, Christmas tree, nursery, etc. We want to make sure respondents are specifying their crops in this box and reporting correctly.
Respondents should complete all questions in this section.
Changes to this section:
Preprints added back to the table
Crop listings reformatted
Hemp added
Reporting of Hops
Exclusion of home gardens
Sweet corn for seed was moved to the vegetable section
Reporting of Field Crops
We have issues with respondents reporting crops that were not field crops, underreports of field crops, and misreporting of crop codes. To address this preprints were added to the table and the crop listing were reformatted. Probing should focus on whether respondents are reporting the appropriate type of crops, and respondents’ ability to locate and report all field crops harvested and the appropriate codes. Respondents are also instructed to exclude home gardens. For growers of hops, acres should be reported to tenth acres.
To address misreporting of hay in field crops “All hay including alfalfa hay and other – report in SECTION 8” was added at the top of the crop listings.
Sweet Corn for Seed
Sweet corn for seed was removed from crop listings. Vegetable seeds should be reported in the vegetable section.
Respondents should complete all questions in this section.
Reporting of Nursery Crops
The format of the crop listing was changed. We want to make sure respondents can locate their crops and the appropriate codes.
Sod
Sod harvested was changed to “Sod Harvested or Intended for Sale in Future Years” (e.g. unharvested sod). It takes more than a year to produce sod. Farms may have sod in production that is not ready for harvesting. We want them to report all sod harvested and sod in production that is intended to be harvested in future years.
Vegetables grown for seed
Crops grown in the open for seed should be reported in the vegetable section. Only those grown for seed, under protection, should be reported here.
Respondents should complete all questions in this section.
Reporting of Vegetables
Respondents are now instructed to exclude home gardens and personal use.
Vegetable Seeds
Respondents should report vegetables grown for seed, in the open, under the crop name (e.g., sweet corn harvested for seed should be reported under sweet corn.)
The format of the crop listing was changed. We want to make sure respondents can locate their crops and the appropriate codes.
Respondents only need to answer Question 3 (internet question).
A new internet question has been added. We are testing this question to make sure respondents can provide valid responses.
Respondents should complete all questions in this section.
This section has been modified. We want to make sure respondents are understanding the terms used in this section and can report correctly.
Respondents should complete all questions in this section.
In 2017, there were quite a few reports where the respondent reported the number of unpaid workers in both the ‘150 plus day’ category and in the ‘unpaid’ category. The number of 150 plus workers is over-reported because respondents report unpaid workers in both of the categories because some unpaid workers work more than 150 days on the operation. They are possibly being misreported in the ‘less than 150 days’ category as well.
To address this, we moved the unpaid farm worker question to before the 150 day questions. Probing should focus on whether this move improved accurate reporting.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Cognitive Interview Protocol 3-2-20 |
| Author | Ridolfo, Heather - NASS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |