SUPPORTING STATEMENT
U. S. Department of Agriculture
Economic Research Service
Generic Clearance for Survey Research Studies
OMB No. 0536 - 0073
Part A - Justification
The Economic Research Service (ERS) of the United States Department of Agriculture (USDA) requests renewal from the Office of Management and Budget (OMB) for generic clearance that will allow ERS to conduct research to improve the quality of data collection by developing, testing, and evaluating its survey instruments, methodologies, technology, interview processes, and respondent recruitment protocols. The primary objective of ERS is providing timely research and analysis to public and private decision makers on topics related to agriculture, food, the environment, and rural America. This request is part of an on-going initiative to improve the quality of ERS’ data products in general and surveys in specific as recommended by both its own guidelines and those of OMB1.
The purpose of this generic clearance is to have an ongoing OMB clearance structure in place for the ERS’ continuing improvement of overall quality of its statistical surveys, to lessen the burden it places on respondents, and to shorten the time period between changes that affect surveys and ERS’ ability to formulate and update its surveys to address those changes. This clearance will also be used to aid in the development of new surveys. This information will be collected under the authority of 7 U.S.C. 2204(a) (Attachment A).
Prior to each survey development or improvement study, ERS will provide OMB with a copy of the questionnaire (if one is used), and all other materials describing the study.
The research techniques and methods to be used in these studies will include analyses of questionnaire construction, interview process, and respondent recruitment protocol, as well as survey technology. ERS envisions using a number of survey improvement and development techniques, as appropriate to the individual project under investigation. These include focus groups, cognitive and usability laboratory and field techniques, exploratory interviews, behavior coding, and respondent debriefing. The specific methods proposed for coverage by this clearance are described below.
Focus Groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for exploring and bringing to the surface issues with either respondents or stakeholders. Focus groups are a good choice during the development of a survey or survey topic, when a pre-existing questionnaire or survey questions on the topic do not yet exist. Focus groups may also be used to explore respondent’s general opinions about data collection technologies or survey material other than questionnaires. Focus groups tend to include individuals who are homogenous across some socio-cultural characteristic of interest (i.e. education, race, sexual orientation). The group needs to be large enough to generate rich discussion but not so large that some participants are left out. In general, focus groups include 6 – 12 people per group. Group sessions typically last approximately 1 - 2 hours. The participants of the discussion sessions will dictate the length of the sessions based on the amount of information they have and their willingness to participate. Although there is a consensus in survey research literate that it takes more than one focus group on any one topic to produce valid results, there is no iron-clad rule regarding how many focus groups are enough (Brancato et al, 2006; Eliot & Associates, 2005; Masadeh, 2012;).
Cognitive and Usability Laboratory and Field Techniques. These techniques generally involve intensive, one-on-one interviews with respondents. Cognitive techniques are generally used to clarify the question-response process. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents may have answering questions. This is frequently the first stage of revising a questionnaire. Various techniques described in Survey Research and Survey Methodology literature will be employed as appropriate. These include think-aloud methods, follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. Usability techniques involve getting respondent input to aid in the development of automated questionnaires and websites and associated materials. They are generally used to understand the physical features of an automated survey questionnaire, for instance, its display and navigational features. A number of different techniques may be involved, such as one-on-one usability interviews with think aloud, probing, and paraphrasing tasks, card-sorting techniques, and disability accommodation testing. The objective is to identify problems that keep respondents from completing automated questionnaires accurately and efficiently, with minimal burden or that prevent respondents from successfully navigating websites and finding the information they seek.
Despite of more than two decades of cognitive interviewing and usability testing, practitioners have little theoretical or empirical guidance to determine sample sizes for these tests. Decisions about sample size historically have been guided by past practices and tend to result more often in smaller rather than larger samples. A survey of cognitive interviewing practices at academic and government research organizations conducted in earlier 90’s found that sample sizes were typically small-most often under 30 in academic and rarely exceeding 60 in federal agencies (Blair and Presser, 1993). However, more recent theoretical and empirical research on sample size for cognitive interviewing and usability testing in survey literature have shown that the relationship of sample size to problem identification is a function of the nature of the problem, its occurring prevalence, and the efficiency of detection mechanisms. Those research studies indicated a strong positive relationship between sample size and problem identification. The probability of observing a given problem in a set of cognitive interviews or usability tests clearly increases as the sample size grows (Blair and Conrad, 2011; Faulkner, 2003). Certain socio-cultural characteristics of population subgroups (i.e. education) and the complexity of the questionnaire also affect sample size for these testings. For example, those with higher education may be better able to articulate their thoughts or do other cognitive interview tasks, thus lowering the necessary sample size.
Exploratory Interviews. This method involves conducting group sessions to understand a topical area when there are few or no earlier studies to which references can be made for information. For the most part, this will be used in the very early stages of developing a new survey. In exploratory interview/research the focus is on gaining insights and familiarity with the subject area for more rigorous investigation later. Exploratory interviews may cover discussions related to administrative records, subject matter, definitions, etc. It may also be used in exploring whether there are sufficient issues related to an existing data collection instrument to consider a redesign. Group size for exploratory interviews ranges from 10 to 20 (P. Phipps2, personal communication, December 15, 2015).
Behavior Coding is a quantitative technique in which a standard set of codes is systematically applied to respondent/interviewer interactions in interviewer-administered surveys or respondent/questionnaire interactions in self-administered surveys in order to evaluate the quality of the questionnaire. The advantage of this technique is that it is a more systematic, objective, quantitative, and representative means of evaluating survey questions relative to other methods such as cognitive interview and expert review. Criticism on behavior coding is that it does not necessarily illuminate the underlying causes of the problems. Behavior coding is a rather costly and time consuming method (Brancato et al, 2006; Zukerberg et al, 1995). The necessary number of interviews for a given study is affected by a variety of factors such as the purposes, time, and monetary resources of the study; the number of coders; the complexity of the questionnaire being evaluated; the usages of data collected by the questionnaire; etc. Discussions in survey methodology research literature on the number of behavior coding interviews which can be of use for evaluation vary between 30 and100.
Respondent debriefing. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the questionnaire being tested or at later on by re-contacting the respondents. It contains questions that probe to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. The purposes of this structured approach to debriefing are to determine whether the original survey questions are understood as intended, to learn about respondents’ form filling behavior and record keeping systems, and to elicit respondents’ satisfaction with the survey. This information can then be used to aid in improving the survey. Respondent debriefing provides a useful complement to other quantitative tests, such as behavior coding or item nonresponse. Design and sample size for debriefing vary according to the question designers’ needs (Brancato et al, 2006; N. Bates3, personal communication, December 15, 2015; A. Maitland4, personal communication, December 19, 2015).
1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The primary function of the ERS is to provide timely research and analysis to public and private decision makers on topics related to agriculture, food, the environment, and rural America.
ERS is requesting the approval of a generic clearance in order to respond quickly to emerging issues and data collection needs. The time frames ERS must meet rarely allows for adequate time to seek individual clearances for evaluation of survey instruments and methodologies. A second reason for requesting an approval for generic clearance is that the agricultural economy continues to change and ERS needs to continuously evaluate its surveys in light of these changes. Respondents continue to change (e.g., response rates decrease over time), technology continues to change (e.g., the web and smart phone quickly became new data collection options), and data needs continue to change. In addition, our understanding of how to improve surveys continues to evolve (e.g., the application of cognitive psychology to survey methodology has increased our understanding of surveys). Thus, ERS needs to have an ongoing OMB clearance structure in place to continue to improve the overall quality of its statistical surveys, to lessen the burden it places on respondents, and to shorten the time period between changes that affect surveys and ERS’ ability to formulate and update its surveys to address those changes.
Since the types of surveys included under the umbrella of the clearance are so varied, it is impossible to specify at this point what kinds of activities would be involved in any particular research study. But at a minimum, one of the types of methods described above or some other form of cognitive pretesting could be employed by each survey.
ERS will consult with OMB prior to submission on the appropriateness of submissions under this clearance that may raise policy or substantive issues. ERS will provide OMB with a copy of questionnaires, protocols, and debriefing materials in advance of any testing activity. ERS will report to OMB on an annual basis a summary of the projects conducted under this clearance.
Data collection for this project is authorized by the 7 U.S.C. 2204(a).
Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
The information collected in this program will be used by staff from the ERS and sponsoring agencies to evaluate and improve the quality of the data in the surveys and censuses that are ultimately conducted. Specifically, the information will be used to reduce respondent burden while simultaneously improving the quality of the data collected in these surveys. These objectives are met when respondents are presented with plain, coherent and unambiguous questionnaires that ask for data, which are compatible with respondents’ memory and/or current reporting and record keeping practices. The purpose of the survey improvement projects will be to ensure that ERS surveys continuously attempt to meet these standards of excellence.
Improved ERS surveys will help policy makers’ decisions on agriculture, as well as contributing to increased agency efficiency and reduced survey costs. In addition, methodological findings have broader implications for survey research.
Because the questionnaires and procedures being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics of ERS or other Federal agencies. Testing results will be included in research reports prepared for sponsors inside and outside of the ERS. The results may also be prepared for presentations related to survey methodology at professional meetings or publications in professional journals.
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.
When the census or survey being pretested employs automated methods for its data collection, the questionnaire or procedural research conducted under this submission will also utilize automated data collection techniques.
ERS intends to employ information technology, as appropriate, to reduce the burden of respondents who agree to participate in its survey improvement projects. ERS may also explore the use of state-of-the-art technology (e.g., satellite TV, the Web, or Smart phones) to conduct focus groups and other appropriate uses of technology, as yet unknown, to reduce burden on respondents.
4. Describe efforts to identify duplication.
This research does not duplicate any other survey research study being done by ERS or other Federal agencies. This research will provide critical, groundbreaking, and important supplemental information beyond what is currently available in the field of survey methodology as it applies to ERS surveys.
This research also does not duplicate any outside-of-government research effort, as its purpose is not to replicate survey research studies. This research will involve collaboration with staff from other agencies who are sponsoring the surveys conducted by ERS. The research may also involve joint efforts with staff from other Federal laboratory facilities as well as research entities in private sectors and academia. All efforts would be collaborative in nature, and no duplication in this area is anticipated.
This research will be designed as relatively small-scale data collection efforts. This will minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests.
6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
This clearance involves one-time questionnaire and/or procedural development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the surveys would suffer.
7. Explain any special circumstances that would cause an information collection to be conducted in a manner requiring respondents to report information to the agency more often than quarterly.
There are no special circumstances associated with this information collection.
Provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8 (d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.
Federal Register: A “Notice of Intent to Request New Information Collection” related to this ICR was published in the Federal Register on September 27, 2018 (vol. 83 FR, pgs. 48793 - 48794), inviting public comment on our plans to submit this request. A copy of the notice is included as Attachment B. No comment was received.
Outside Consultation: Consultation with staff from other Federal agencies that sponsor surveys conducted by the ERS will occur in conjunction with the testing program for the individual survey. Consultation with staff from other Federal laboratory facilities and/or research entities in private sectors and academia may also occur as part of joint research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance.
9. Explain any decision to provide any payment or gift to respondents.
Respondents for activities conducted in the laboratory under this clearance will receive a small stipend. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by the other Federal statistical agencies. The incentive for participation in a cognitive interview is $40, and for participation in a focus group is $50-$75. ERS may provide smaller incentives than these amounts at its discretion; however, any requests for larger amounts must be justified in writing to OMB.
Respondents for methods that are generally administered as part of field test activities (e.g. behavior coding of interviewer/respondent interaction, and respondent debriefing) or other research projects where ERS staff travel to and test in respondent’s residences will not receive payment unless there are extenuating circumstances that warrant it. Such circumstances and proposed incentives must be justified in writing to OMB.
Periodically, non-monetary tokens of appreciation such as baseball caps, pens, and other trinkets will be given to pre-testing participants.
10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
ERS intends to protect respondent information under the Privacy Act of 1974, Section 1770 of the Food Security Act of 1985, and 7 U.S.C. 2276. ERS has decided not to invoke the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA). The complexity and cost necessary to invoke CIPSEA is not justified given the nature of the collections; the collections would generally be conducted by ERS’ contractors and designed to be hosted in non-government owned computer systems, where CIPSEA compliance could not be assured.
Specific details regarding information handling will be specified in individual submissions under this generic clearance.
11. Provide additional justification for any questions of a sensitive nature.
Most of the questions that are included on ERS questionnaires are not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered.
12. Provide estimates of the hour burden of the collection of information. The statement should indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.
The total estimated respondent burden is 1,815 hours and total number of respondents is 3,630 for the requested three-year period. This estimate is based on our information collection activities in the past three years, and our anticipated use of the clearance over the next three years. Each study will differ substantively from the others. The projects are expected to be complex, involving at times hard-to-count populations, new collection instruments, and/or new modes of collection, which in general require larger samples, multiple testing methods, and an iterative process (multiple rounds of testing) to test the hypotheses of the given research questions (Brancato et al, 2006; N. Bates, personal communication, December 15, 2015; A. Maitland, personal communication, December 19, 2015; OMB, 2016).
Public reporting burden for these collections of information is estimated to average from .5 to 1.5 hours per respondent, dependent upon the survey and the technique used to test for that particular survey. No respondent would be contacted more than once in a calendar year for testing any of the survey improvement techniques. This estimate covers the time that each respondent will spend answering questions, including the debriefing procedure concerning the cognitive testing procedure used. The time required to travel to the laboratory, if needed, is not covered, since distances and modes of transportation are unknown. No retrieval of information by respondents is anticipated, although it is possible that validation of data at some point may require respondents to keep and check records. In this case, experiments will be designed to include retrieval time.
Total reporting cost to the public for studies conducted under this clearance is estimated to be $42,072. Cost of respondents’ time is estimated based on the average hourly wage for production and non-supervisory private-sector workers ($23.18 per hour in March 2019, as estimated by the U.S. Bureau of Labor Statistics; https://data.bls.gov/timeseries/CES0500000008) multiplied by the total response time (1,815 hours) for all respondents in the requested three-year period.
A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time.
13. Provide an estimate of the total annual cost burden to respondents or recordkeepers resulting from the collection of information.
There is no capital/startup or ongoing operation/maintenance costs associated with this information collection period.
14. Provide estimates of annualized cost to the Federal government; provide a description of the method used to estimate cost which should include quantification of hours, operational expenses (equipment, overhead, printing, and staff), and any other expense that would not have been incurred without this collection of information.
There is no way to anticipate the actual number of participants, length of interview, and/or mode of data collection for the surveys to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by divisions conducting the research from their data collection budgets.
15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I (reasons for changes in burden).
There are no changes in burden or costs expected for this docket.
16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.
This research program is for questionnaire and procedure development purposes. Data will be collected to develop new surveys or improve the methodology of current surveys. The information collected in this effort will not be the subject of any printed ERS reports; however, it might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may, however, be prepared for presentation at professional meetings or publication in professional journals.
Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
The ERS is not seeking approval to avoid displaying the expiration date for OMB approval of the information collection
18. Explain each exception to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions” of OMB Form 83-I.
No exceptions to the Certification Statement should be required. If so, OMB approval will be requested in advance of conducting the survey.
References
Blair, J., and Presser S. (1993). “Survey Procedures for Cognitive Interviews to Pretest Questionnaires: A Review of Theory and Practice.” Proceedings of the Survey Research Methods Section of the American Statistical Association, 370-75.
Blair, J., and Conrad, F. G. (2011). “Sample Size for Cognitive Interview Pretesting.” Public Opinion Quarterly, 75(4): 636-58.
Brancato, G. et al. (2006). Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System, European Commission Grant Agreement 200410300002. Available from http://www.istat.it/en/files/2013/12/Handbook_questionnaire_development_2006.pdf
Eliot & Associates. (2005). “Guidelines for Conducting a Focus Group.” Available from https://assessment.trinity.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf
Faukner, L. (2003). “Beyond the Five-User Assumption: Benefits of Increasing Sample Sizes in Usability Testing.” Behavior Research Methods, Instruments, and Computers, 35:379-83.
Masadeh, M. A. (2012). “Focus Group: Reviews and Practices.” International Journal of Applied Science and Technology, 2(10): 63-68.
Statistical and Science Policy Office, Office of Information and Regulatory Affairs, Office of Management and Budget (OMB). (2016). Statistical Policy Working Paper 47 Evaluating Survey Questions: An Inventory of Methods. Available from http://fcsm.sites.usa.gov/files/2014/04/spwp47.pdf
Zukerberg, A.L.; Von Thurn, D.R.; and Moore, J.C. (1995). “Practical Considerations in Sample Selection for Behavior Coding Pretests.” Proceedings of the Section on Survey Research Methods of the American Statistical Association, 1116-1121. Available from
http://www.census.gov/srd/papers/pdf/az9501.pdf
1 ERS Data Product Quality Standards are available on https://www.ers.usda.gov/about-ers/policies-and-standards/data-product-quality/ers-data-product-quality-standards/. OMB Information Quality Guidelines are available on https://obamawhitehouse.archives.gov/omb/fedreg_final_information_quality_guidelines/.
2 Polly Phipps is a Senior Survey Methodologist at the Bureau of Labor Statistics.
3 Nancy Bates is a Senior Researcher for Survey Methodology at the U.S. Census Bureau.
4 Aaron Maitland is a Senior Survey Methodologist at Westat, Inc.
| File Type | application/msword |
| File Title | REQUEST FOR GENERIC CLEARANCE OF |
| Author | mvburke |
| Last Modified By | SYSTEM |
| File Modified | 2019-03-11 |
| File Created | 2019-03-11 |