GenericClearanceforInternetPanelPretestingandQualitativeSurveyMethodsTesting_SupportingStatementA_080520_FINAL

SUPPORTING STATEMENT A

U.S. Department of Commerce

U.S. Census Bureau

Generic Clearance for Internet Panel Pretesting and Qualitative Survey Methods Testing

OMB Control No. 0607-0978

Abstract

This research program will be used by the Census Bureau and survey sponsors to test alternative contact methods, including emails and text messages (via an opt-in strategy), improve online questionnaires and procedures, reduce respondent burden, and ultimately increase the quality of data collected in the Census Bureau censuses and surveys. We will use the clearance to conduct pretesting of decennial and demographic census and survey questionnaires prior to fielding them as well as communications and/or marketing strategies and data dissemination tools for the Census Bureau. The primary method of identifying measurement problems with the questionnaire or survey procedure is split panel tests. This will encompass both methodological and subject matter research questions that can be tested on a medium-scale nonprobability panel.

This research program will also be used by the Census Bureau for remote usability testing of electronic interfaces and to perform other qualitative analyses such as respondent debriefings. An advantage of using remote, medium-scale testing is that participants can test products at their convenience using their own equipment, as opposed to using Census Bureau-supplied computers. A diverse participant pool (geographically, demographically, or economically) is another advantage. Remote usability testing would use click through rates and other paradata, accuracy and satisfaction scores, and written qualitative comments to determine optimal interface designs and to obtain feedback from respondents.

Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

At this time, the U.S. Census Bureau is seeking a three-year reinstatement of the generic clearance for pretesting for a variety of medium-scale iterative Internet research pretesting activities. We will dedicate a block of hours to these activities for each of the next three years. OMB will be informed in writing of the purpose and scope of each of these activities, as well as the time frame and number of burden hours used. The number of hours used will not exceed the number set aside for this purpose.

The Census Bureau is committed to conducting research in a cost efficient manner. There are several stages of testing in research projects at the Census Bureau. As a first stage of research, the Census Bureau pretests questions on surveys or censuses and evaluates the usability and ease of use of websites using a small number of subjects during focus groups, usability and cognitive testing. These projects are in-person and labor-intensive, but typically only target samples of 20 to 30 respondents. This small-scale work is done through another existing OMB generic clearance. Often the second stage is a larger-scale field test with a split-panel design of a survey or a release of a Census Bureau data dissemination product with a feedback mechanism. The field tests often involve a lot of preparatory work and often are limited in the number of panels tested due to the cost considerations. They are often targeted at very large sample sizes and are typically done using stand-alone OMB clearances.

Cost efficiencies can occur by testing some research questions in a medium-scale test, using a smaller number of participants than what we typically use in a field test, yet a larger and more diverse set of participants than who we recruit for cognitive and usability tests. Using a medium-scale test, we can answer some research questions more thoroughly than in the small-scale testing, but less expensively than in the large-scale field test. This clearance established a medium-scale cost-efficient method of testing questions and contact strategies.

Methods

Quantitative surveys. These methods may include probability- or non-probability samples to interview a number of participants with primarily close-ended questions. The goal of these surveys would be to measure public opinion for internal use only and not to provide official estimates of the population.

Split sample experiments. This involves testing alternative versions of questionnaires, invitations to questionnaires (e.g., emails or text messages), or websites, at least some of which have been designed to address problems identified in draft versions or versions from previous waves. The use of multiple questionnaires, invitations, or websites, randomly assigned to permit statistical comparisons, is the critical component here; data collection will be via the Internet. Comparison of revised questionnaires (or invitations) against a control version, preferably, or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire (or invitation or website).

The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty respondents.

Split sample tests that incorporate methodological questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than other pretest methods. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of Census Bureau data collection instruments.

Usability Interviews: This method involves getting respondent input to aid in the development of automated questionnaires and websites and associated materials. The objective is to identify problems that keep respondents from completing automated questionnaires accurately and efficiently with minimal burden, or that prevent respondents from successfully navigating websites and finding the information they seek. Remote usability testing may be conducted under this clearance, whereby a user would receive an invitation to use a website or survey, and then answer targeted questions about that experience.

Qualitative Interviews: This method involves one-on-one group interviews in which the respondent is typically asked questions about survey content areas, survey questions or the survey process. A number of different techniques may be involved, including cognitive interviews, online discussion forums, and focus groups. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents may have answering survey questions in order to improve the information ultimately collected in large-scale surveys and censuses.

Procedures for Clearance

Since the types of surveys included under the umbrella of the clearance are so varied, it is difficult to specify at this point what kinds of activities would be involved in any particular test, but a key component will be the comparison of one invitation, questionnaire or website to another.

We will provide OMB with a copy of questionnaires and invitations in advance of any testing activity. Depending on the stage of development, this may be the printed material from the last round of a survey or a revised draft based on analysis of other evaluation data. For a test of alternative procedures, the description and rationale for the procedures would be submitted. We will also provide a description of the sample design and the planned administration. OMB has previously agreed to endeavor to provide comments on substantive issues within 10 working days of receipt.

The Census Bureau will consult with OMB prior to submission on the appropriateness of submissions under this clearance that may raise policy or substantive issues.

Data collection for this project is authorized under the authorizing legislation for the questionnaire being tested. This may be Title 13, Sections 131, 141, 161, 181, 182, 193, and 301 for Census Bureau-sponsored surveys or other corresponding collection authorities for surveys sponsored by other Federal agencies. We do not now know what other titles will be referenced, since we do not know what survey questionnaires will be pretested during the course of the clearance.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The information collected in this program of developing and testing questionnaires, in the form of paper and electronic questionnaires based on self-administered or interviewer-administered surveys, will be used by staff from the Census Bureau, contractors and sponsoring agencies to evaluate and improve the quality of the data in the surveys and censuses that are ultimately conducted. Data from the research will be included in research reports with clear statements about the limitations and that the data were produced for strategic and tactical decision-making and exploratory research and not for official estimates. Research results may be prepared for presentation at professional meetings or in publications in professional journals to promote discussion among the larger survey and statistical community, encourage further research and refinement. Again, all presentations or publications will provide clear descriptions of the methodology and its limitations.

Information quality is an integral part of the pre-dissemination review of the information disseminated by the Census Bureau (fully described in the Census Bureau's Information Quality Guidelines). Information quality is also integral to the information collections conducted by the Census Bureau and is incorporated into the clearance process require d by the Paperwork Reduction Act.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.

This generic clearance will use at least 75 percent electronic data collection. It will use primarily electronic data collection instruments with some paper questionnaires. We will also test Internet data dissemination websites with email invitations or by posting a link on a Census Bureau webpage to test out the website.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Question 2.

This research does not duplicate any other questionnaire design work being done by the Census Bureau or other Federal agencies. The purpose of this clearance is to stimulate additional research, which would not be done under other circumstances due to time constraints. This research will involve collaboration with staff from other agencies that are sponsoring the surveys conducted by the Census Bureau. The research may also involve joint efforts with staff from other Federal laboratory facilities. All efforts would be collaborative in nature, and no duplication in this area is anticipated.

To the maximum extent possible, we will make use of previous information, reviewing results of previous evaluations of survey data before we attempt to revise invitations or questionnaires. However, this information is not sufficient to refine our census and survey invitations or questionnaires without conducting additional research.

This generic clearance request is an explicit addition to the current generic clearance for pretesting (Number 0607-0725) that allows small-scale cognitive and usability testing research and the generic clearance for field testing (Number 0607-0971) that allows large-scale regional and national field testing. This generic clearance is different from the previous clearances in two ways. First, the primary method of data collection is electronic – using online panels for frames and electronic invitation procedures. Though this does not preclude related, in person data collection, that is not the primary means of testing for this clearance package. Second, this package may not elicit a mandatory authority and is intended for research that would otherwise not be conducted in the production environment.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

This research will be designed as small-to-medium scale data collection efforts. This will minimize the amount of burden required to improve questionnaires and procedures, test new ideas, and refine or improve upon positive or unclear results from other tests.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

This clearance involves one-time questionnaire development activities for each survey that is connected with the clearance. If this project were not carried out, the quality of the data collected in the surveys would suffer. In addition, testing websites with a larger sample-base will allow us to have more confidence in our findings and the resulting revisions to the websites.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

• requiring respondents to report information to the agency more often than quarterly;

Respondents will not be required to report information to the agency at all, given the use of a voluntary authority. If respondents are invited to participate in a survey more than quarterly, they may decline any particular invitation.

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

Respondents will not be required to report information to the agency at all, given the use of a voluntary authority. Respondents will not be asked to prepare written responses to information collections.

• requiring respondents to submit more than an original and two copies of any document;

Respondents will not be required to report information to the agency at all, given the use of a voluntary authority. Respondents will not be asked to submit more than an original and two copies of any document.

• requiring respondents to retain records, other than health, medical, government contract, grant-in- aid, or tax records for more than three years;

Respondents will not be required to report information to the agency at all, given the use of a voluntary authority. Respondents will not be asked to retain records for more than three years.

• in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

Our surveys will be designed to produce valid and reliable results that can be generalized to the intended universe OR it will be clearly noted otherwise.

• requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

Our surveys will not require the use of a statistical data classification that has not been reviewed and approved by OMB.

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

Our pledges of confidentiality will be approved by Census Bureau’s Policy Coordination Office and will comply with all regulations.

• requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

Respondents will not be required to report information to the agency at all, given the use of a voluntary authority. All data collected will have appropriate legally required confidentiality protections.

8. If applicable, provide a copy and identify the date and page number of publications in the Federal Register of the agency's notice, required by 5 CFR 1320.8 (d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years - even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

Reg Baker, Consultant at Market Strategies International, and an expert in nonprobability internet panel testing was a consultant on the initial package. More recently, we have consulted with the BLS, the USDA ERS, SSA, NCHS, NSF and NCES on the use of this type of research.

Consultation with staff from other Federal agencies that sponsor surveys conducted by the Census Bureau will occur in conjunction with the testing program for the individual survey. Consultation with staff from other Federal laboratory facilities may also occur as part of joint research efforts. These consultations will include discussions concerning potential response problems, clarity of questions and instructions, and other aspects of respondent burden. Additional efforts to consult with potential respondents to obtain their views on the availability of data, clarity of instructions, etc., may be undertaken as part of the testing that is conducted under this clearance.

A notice was published in the Federal Register on June 3, 2020 (85 FR 34175), inviting public comment on our plans to submit this request. We have received five comments, none of which are substantively related to this collection.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

In general, respondents will not be paid or provided any cash incentive for their participation in online panels. In certain situations, a token incentive may be proposed. Details of this incentive will be outlined with each individual request submitted by the Census Bureau to OMB.

Respondents for activities conducted in the laboratory (that is, cognitive interviews, usability interviews and focus groups) under this clearance will receive a small stipend. This practice has proven necessary and effective in recruiting subjects to participate in this type of research, and is also employed by the other Federal cognitive laboratories. The incentive for participation in a cognitive or usability interview is $40 for an interview under 90 minutes and $60 for an interview lasting more than 90 minutes, and for participation in a focus group it is $75 unless approval for a higher amount is granted by OMB on a case-by-case basis.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy. If the collection requires a systems of records notice (SORN) or privacy impact assessment (PIA), those should be cited and described here.

All respondents who participate in research under this clearance will be informed that the information they provide will not be made available in any way that would personally identify him or her and that their participation is voluntary. This disclosure will be made prior to any data collection.

Depending on the collection, the confidentiality of information may be assured by Title 13, United States Code, or another applicable title. Per the Federal Cybersecurity Enhancement Act of 2015, data are protected from cybersecurity risks through screening of the systems that transmit data.

Internally sponsored studies will include the following disclosures:

The U.S. Census Bureau is conducting this voluntary study under the authority of 13 U.S.C. Sections 6, 8(b), 131, 141, 161, 182, and 193. The purpose of collecting this information is to improve and inform future surveys.

Your privacy is protected by the Privacy Act of 1974 (5 U.S.C. Section 552a). The Privacy Act permits sharing of the information provided to us and may be shared with other Census Bureau staff for work-related purposes as identified in the Privacy Act System of Records Notices (SORNs) COMMERCE/Census -3, Demographic Survey Collection (Census Bureau Sampling Frame); COMMERCE/CENSUS-5, Decennial Census Programs; COMMERCE/CENSUS-4, Economic Survey Collections; and COMMERCE/CENSUS-7, Demographic Survey Collection (non-Census Bureau Sampling Frame).

Externally sponsored studies will have their relevant disclosures attached to the ICR.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

Most of the questions that are included on Census Bureau questionnaires are not of a sensitive nature and should not pose a problem to respondents. However, it is possible that some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before the actual survey is administered. For information collections involving questions of race/ethnicity, the agency will ensure that the OMB Standards for the Classification of Federal Data on Race and Ethnicity are followed, unless we are specifically testing this question. In that situation, OMB will be made aware of the proposed changes and the related research agenda.

12. Provide estimates of the hour burden of the collection of information.

• Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

• If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens.

• Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included under ‘Annual Cost to Federal Government’ (Item #14).

We will use a variety of forms to conduct the research under this clearance, but the exact number of different forms, length of each form, and number of subjects/respondents per form are unknown at this time.

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden already reflected on the burden worksheet).

• The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation, maintenance, and purchase of services component. The estimates should consider costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling and testing equipment; and record storage facilities.

• If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10), utilize the 60-day pre-OMB submission public comment process and use existing economic or regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate.

• Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

There is no cost to respondents for participating in the research we are conducting under this clearance, except for their time to complete the questionnaire and (if applicable) travel to the location to participate in the research.

14. Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

Agencies may also aggregate cost estimates from Question 12, 13, and 14 in a single table.

There is no way to anticipate the actual number of participants, length of interview, and/or complexity of the data collection instruments for the surveys to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. The Census Bureau will cover the costs.

15. Explain the reasons for any program changes or adjustments reported in ROCIS.

There are no changes to the information collection since the last OMB approval.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

This research program is for questionnaire and procedure development purposes. We will use data tabulations to evaluate the results of questionnaire testing. The information collected in this effort will not be the subject of any printed Census Bureau reports; however, it might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may, however, be prepared for external presentation at professional meetings or publication in external professional journals.

Due to the nature of this clearance, there is no definite or tentative time schedule at this point. We expect work to continue more or less continuously throughout the duration of the clearance.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The agency plans to display the expiration date for OMB approval of the information collection on all instruments.

18. Explain each exception to the certification statement identified in “Certification for Paperwork Reduction Act Submissions."

The agency certifies compliance with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).

Page | 7