REQUEST FOR GENERIC CLEARANCE OF SURVEY IMPROVEMENT PROJECTS
FROM THE NATIONAL CENTER FOR SCIENCE AND ENGINEERING STATISTICS (NCSES)
OMB Clearance 3145-0174
The National Center for Science and Engineering Statistics (NCSES) of the National Science Foundation (NSF) requests a three-year extension of the Office of Management and Budget’s (OMB’s) generic clearance that will allow NCSES to continue to rigorously develop, test, and evaluate its survey instruments, methodologies, and dissemination methods and tools. NSF has a mandate to “provide a central clearinghouse for the collection, interpretation, analysis, and dissemination of data on scientific and engineering resources and to provide a source of information for policy formulation by other agencies of the Federal Government.” This request is part of an ongoing initiative to improve NCSES surveys and other data collections as recommended by both its own guidelines and those of OMB.0
In the last decade, state-of-the art techniques have been increasingly instituted by NSF and other federal agencies, and are now routinely used to improve the quality and timeliness of survey data and analyses, while simultaneously reducing respondents’ cognitive workload and burden. The purpose of this generic clearance is to allow NCSES to continue to adopt and use these techniques to improve its current data collections on science, engineering, and technology inputs and outputs. They will be used to improve the content of existing surveys, to aid in the development of new data collections to capture changes in the U.S. science and engineering (S&E) enterprise, and to fill gaps in coverage of the S&E enterprise in the existing NCSES portfolio. The generic clearance will also allow NCSES to explore the possibility of alternative methods of data collection, as well as data dissemination tools and mechanisms.
Following standard OMB requirements, NCSES will submit to OMB an individual request for each project it undertakes under this generic clearance. NCSES will request OMB approval in advance and provide OMB with a copy of the materials that describe the project, including the questionnaire (if one is used).
NCSES envisions using a variety of data collection techniques for the improvements, as appropriate to the individual projects, such as focus groups, cognitive and usability laboratory and field techniques, exploratory interviews, behavior coding, respondent debriefing, pilot studies, pretests, split-panel tests, and customer satisfaction surveys. NCSES has used such techniques in previous activities conducted under generic clearance. However, in contrast to prior generic clearance renewals, NCSES expects to take advantage of new online tools available for designing, evaluating, and testing efforts, which may allow the agency to recruit larger numbers of participants to its studies.
Focus Groups. A qualitative methodology that brings together a small number of relatively homogenous subjects to discuss pre-identified topics. A protocol containing questions or topics focused on a particular issue or issues is used to guide these sessions, and is administered by a trained facilitator. Focus groups are useful for exploring and identifying issues with either respondents or stakeholders. Focus groups are a good choice during the development of a survey or survey topic, when a pre-existing questionnaire or survey questions on the topic do not yet exist; they can also useful in obtaining data user requirements for new or improved data delivery tools. In the past, NCSES has used focus groups to assist with redesigning surveys when it became evident that the content of a survey was outdated and did not reflect current issues or the context that respondents were facing.
Cognitive and Usability Laboratory and Field Techniques. A qualitative methodology that refers to a set of tools employed to study and identify errors that are introduced during the survey process. These techniques are generally conducted by a researcher with an individual respondent, though observers may sometimes be present. Cognitive techniques are generally used to understand the question-response process, whereas usability is generally used to understand respondent reactions to the features of an electronic survey instrument, for instance, its display and navigation. In concurrent interviews, respondents are asked to think aloud as they actually answer the survey. In retrospective interviews, respondents answer the survey as they would normally, then ‘think aloud’ afterwards. Other techniques, which are described in the literature and which will be employed as appropriate include: follow-up probing, memory cue tasks, paraphrasing, confidence rating, response latency measurements, free and dimensional sort classification tasks, and vignette classifications. The objective of all of these techniques is to aid in the development of surveys that work with respondents’ thought processes, thus reducing response error and burden. These techniques are generally very useful for studying and revising a pre-existing questionnaire. NCSES has used cognitive and usability testing in previous generic clearance projects to improve existing survey items, to develop and refine new content on existing surveys, and to explore content for new surveys.
Exploratory Interviews. A technique where interviews are conducted with individuals to gather information about a topical area. These may be used in the very early stages of developing a new survey or new data delivery mechanism. They may cover discussions related to administrative records, subject matter, definitions, functionality, etc. Exploratory interviews may also be used to investigate whether there are sufficient issues related to an existing data collection to consider a redesign. NCSES has used such interviews extensively in recordkeeping studies with respondents to several of its establishment surveys to determine both what types of records institutions keep (and therefore what types of information they can supply), as well as where and in what format such records are kept.
Respondent Debriefing. A technique in which individuals are queried about how they have responded to a particular survey, question, or series of questions. The purpose of the debriefing is to determine if the original survey questions are understood as intended, to learn about respondents’ form filling behavior and recordkeeping systems, or to elicit respondents’ satisfaction with the survey. This information can then be used (especially if it is triangulated with other information) to improve the survey. This technique can be used as a qualitative or quantitative measurement, depending on how it is administered. This technique has been employed in NCSES generic clearance projects to identify potential problems with existing survey items both quantitatively and qualitatively.
Pilot Studies/Pretests. These methodologies are typically used to test a preliminary version of the data collection instrument, as was done with the Early Career Doctorate Project. Pretests are used to gather data and assess reliability, validity, or other measurement issues. Pilot studies are also used to test aspects of implementation procedures. The sample may be purposive in nature, or limited to particular groups for whom the information is most needed. Alternatively, small samples can be selected to statistically represent at least some aspect of the survey population.
Split Panel Tests. A technique for controlled experimental testing of alternatives. Thus, they allow one to choose from among competing questions, questionnaires, definitions, error messages, surveys, or survey improvement methodologies with greater confidence than other methods alone. Split panel tests conducted during the actual fielding of the survey are superior in that they support both internal validity (controlled comparisons of variable under investigation) and external validity (represent the population under study). Nearly any of the previously mentioned survey improvement methods can be strengthened when teamed with this method.
Behavior Coding. A quantitative technique in which a standard set of codes is systematically applied to respondent/interviewer interactions in interviewer-administered surveys or respondent/questionnaire interactions in self-administered surveys. Though this technique can quantifiably identify problems with the wording of questions, it does not necessarily illuminate the underlying causes.
Customer Satisfaction Surveys. These surveys are typically brief, asking a few questions about particular aspects of, for example, a survey or a website. NCSES expects to use this technique primarily to evaluate data dissemination tools and methods.
The NSF National Center for Science and Engineering Statistics (NCSES) is responsible for collecting, analyzing, evaluating and disseminating information on science, engineering and technology employment, work force, and education, as well as research and development (R&D) funding and performance. In accordance with Sec. 3(a)(6) of the National Science Foundation Act of 1950, as amended, and Section 505 of the America COMPETES Act (public law 110-69), the National Science Foundation (NSF) is directed to “serve as a central Federal clearinghouse for the collection, interpretation, analysis, and dissemination of objective data on science, engineering, technology, and research and development…that is relevant and useful to practitioners, researchers, policymakers, and the public.” NCSES publishes data in individual survey reports and in such general reports as Science and Engineering Indicators and Women, Minorities and Persons with Disabilities in Science and Engineering. NCSES also releases data in a variety of formats including data tables, data tools, interactive web tools, and public use files.
An extension to NCSES’ previously-granted generic clearance is requested for several reasons. As a federal statistical agency, NCSES is engaged in a process of continuous improvement in the data collections it conducts and in the way it provides access to data and information. Critical to the improvement in existing surveys is the ability to engage in small scale projects to test alternatives to current approaches. Generic clearance authority substantially enhances NCSES’ ability to engage in such exploration, testing and evaluation. Furthermore, as the world continues to change, NCSES must continuously evaluate its surveys and data dissemination tools and methods in light of these changes. Respondent behaviors will change (e.g., response rates decrease over time); technology will change (e.g., the web quickly became a data collection option and is the primary mode for dissemination); and the S&E enterprise will change (e.g., today’s students increasingly pursue multi/interdisciplinary studies rather than a single discipline). Similarly, the understanding of how to improve surveys continues to evolve (e.g., today’s research continues to update the interpretation of the best implementation for web surveys).
Thus, NCSES requests an OMB generic clearance structure to continue improving the overall quality of its statistical surveys, reduce the burden on respondents to NCSES surveys, shorten the time required for NCSES to update and improve its data collections, and redesign and improve its dissemination tools and methods.
The information obtained from these efforts will be used to develop new NSF surveys and improve current ones. Specifically, the information will be used to reduce respondent burden and to improve the quality of the data collected in these surveys. These objectives are met when respondents are presented with plain, coherent, and unambiguous questionnaires asking for data compatible with respondents’ memory and/or current reporting and recordkeeping practices. The purpose of the survey improvement projects will be to ensure that NCSES surveys are continuously attempting to meet these standards of excellence. In addition, the information obtained from data dissemination improvement efforts will be used to help design new data access tools or improve existing dissemination methods. Improved data access will help policymakers, researchers, and the general public by easing and streamlining the way they find the information they are seeking.
Improved NSF surveys will help policy makers in decisions on R&D funding, graduate education, scientific and technical workforce, innovation, as well as contribute to increased agency efficiency and reduced survey costs. In addition, methodological findings have broader implications for survey research and may be presented in technical papers at conferences or published in the proceedings of conferences or in journals.
NCSES will employ information technology, as appropriate, to reduce the burden of respondents who agree to participate in its improvement projects. Many respondents of current NCSES surveys supply email addresses that can be used to recruit respondents for survey improvement projects. This allows respondents to communicate with NCSES at their convenience. Respondents to current NCSES surveys of academic institutions can often provide addresses for websites with additional information (e.g., about their schools), once again reducing their workload. NCSES will continue to explore state-of-the-art technology to find ways to reduce burden on respondents to both individual and establishment surveys. For example, NCSES used desktop sharing and teleconferencing software to conduct usability testing in remote locations to gather information about the functionality of its data dissemination tools. By using this software to digitally record both comments and web screen interactions, NCSES had a complete record of each session that made it unlikely that there would be a need to call respondents back to clarify notes of the sessions.
Web surveys facilitate accurate data by providing respondents with automated tabulations and feedback on inconsistent answers. These features potentially reduce the need for followup contact with respondents. However, the success of these features resides in their being well designed to ensure that respondents are aware of these features. Thus, one focus of NCSES improvement activities is improving the usability of NCSES web surveys. NCSES improvement projects help ensure that respondents are presented with the most user friendly and least burdensome survey instruments possible. In addition, NCSES continues to explore the adoption of innovative methods that could reduce respondent burden and provide easier access to data and information. Enhanced data dissemination tools help users find, organize, and download data faster and more accurately.
Survey improvement projects will be conducted both to improve existing surveys as well as to develop new NCSES data collections. The NCSES data collections themselves are subject to great scrutiny to ensure there is no duplication of other efforts. Likewise, the projects conducted under the generic clearance authority will be structured in order not to duplicate other efforts. Furthermore, generic clearance activities help avoid repetitious efforts to hone implementation for the full-scale surveys.
One goal of NCSES’ efforts to improve its surveys is to minimize the burden on the small organizations that respond to NCSES surveys. By learning about organizational and recordkeeping practices of small, medium, and large organizations, NCSES is in a better position to design surveys and data collection procedures that minimize the burden for various types of respondents, especially small and very small entities. For example, NCSES has investigated methods for surveying companies with fewer than ten employees about their R&D activities.
In the case of pilot studies or split-panel tests, if probability samples are utilized, sampling rates proportional to size are often used to make sure that a large institution has a higher probability of being selected than a small institution. This ensures that a high proportion of the attribute of interest—U.S. S&E funding, performance, employment, or education—is captured while minimizing the burden on small entities.
Numerous consequences would result if NCSES could not conduct the survey and data dissemination improvement projects requested in this document. The quality of the data collected could decrease because the current surveys would not be systematically evaluated and updated to better reflect the current state of S&E. Over time, surveys and data dissemination tools that are currently well designed would eventually become obsolete. New survey items and procedures could not be implemented without adequate testing and refinement. Advances in understanding how organizations or individuals answer surveys or provide alternative forms of data, and how NCSES can better serve respondents, would be curtailed. Finally, NCSES’ ability to develop timely, new, and well-designed surveys and data dissemination tools would be diminished.
Under this clearance, NCSES will explain any circumstances that would result in respondents being required to:
Report information to the agency more often than quarterly;
Prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Submit more than an original and two copies of any document;
Retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
Respond to a statistical survey in a manner that is not designed to produce valid and reliable results, i.e., results that can be generalized to the universe of study;
Use a statistical data classification that differs from one approved by OMB;
Respond in a manner that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of compatible data with other agencies for confidential use;
Submit proprietary trade secret or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
Comments on this data collection effort were solicited in the Federal Register (84 FR 8905) March 12, 2019 (see Attachment A). NCSES received one comment related to this request, via web on April 19, 2019. The comment consisted of a paper concerning access to engineering education and accreditation, especially for minorities and working-class adults. NSF believes the comment does not pertain to the collection of information under the generic clearance, which is focused on survey improvement activities.
The primary objectives of the survey and data collection improvement projects include involving respondents in the development of new survey content, soliciting respondent feedback to current surveys, observing respondent navigation of survey questionnaires, exploring how respondents’ recordkeeping systems work, and eliciting feedback from data users on dissemination methods and tools. These objectives focus on consultation with respondents to reach the goals of understanding (1) how to minimize the time and effort to complete survey and data collection tasks, (2) how to reduce other aspects of burden such as concerns about the use of the survey data, (3) how to motivate respondents to provide survey answers that have the highest quality and most accuracy, and (4) respondents’ ability to access and use the data disseminated by NCSES.
NCSES and its contractors sometimes provide compensation to participants in improvement projects. In some cases, the compensation covers travel costs only. In other cases, compensation is offered in the range of $25-$75 for activities such as focus groups or cognitive interviews. This practice has proven necessary and effective in recruiting some types of subjects to participate in this small-scale research, and is also employed by other federal agencies. Testing activities are sometimes conducted in contractors’ cognitive laboratories or other comparable facilities.
Unless otherwise specified and with approval granted by OMB, such incentives will be limited to no more than $40 for participation in a cognitive interview and no more than $75 for participation in a focus group. Respondents for field test activities such as split sample tests, behavior coding of interviewer/respondent interaction, and respondent debriefings will receive payment only when there are extenuating circumstances that warrant it. Generic clearance packages for projects offering participant compensation will explain the rationale and describe the amounts of compensation.
Respondents in the improvement projects will be advised that their participation is voluntary. In focus groups, interviews, and other respondent activities, NCSES may ask respondents for permission to record sessions via audio or video recording. Such recordings are conducted to provide project staff, including those not present at the activity, with a complete and accurate record to supplement note taking. Recording the session also allows staff to focus more on what is taking place during the session rather than on the completeness of their notes. In some cases, recordings may be used to train others to conduct this type of research or for illustrative purposes in presentations to professional audiences. For sessions that are recorded, respondents will be asked for their consent to the audio or video recording. They will be notified if there is any chance that a session may be played for audiences for research purposes.
Confidentiality will be pledged in some cases, such as when the information being requested about an individual or an organization may be sensitive. The pledge of confidentiality will be made under the Privacy Act (where applicable) and the National Science Foundation Act of 1950 (as amended). Specifically, when confidentiality is pledged to individuals, the pledge used will be the following:
The information is solicited under authority of the National Science Foundation Act of 1950 as amended. All information you provide is protected under the NSF Act as amended and the Privacy Act of 1974 as amended and will only be used for research or statistical purposes. Any information publicly released such as statistical summaries will be in a form that does not personally identify you.
When confidentiality is pledged to organizations, such as businesses, colleges and universities, and other non-profit organizations, the pledge used will be the following:
The information is solicited under authority of the National Science Foundation Act of 1950 as amended. All information you provide is protected under the NSF Act as amended and will only be used for research or statistical purposes. Any information publicly released such as statistical summaries will be in a form that does not personally identify you or your organization.
No questions of a sensitive nature are anticipated in work conducted under this generic clearance. However, the nature of the exploration of survey content may include asking respondents whether items might be considered sensitive in the context of data collection.
NCSES estimates that a total reporting burden of 15,610 hours over the three years of the requested generic clearance will result from working to evaluate or improve existing surveys, to develop new surveys or data collections, or to evaluate data dissemination methods. This includes both the burden placed on respondents participating in each activity as well as burden imposed on potential respondents during screening activities. Table 1 provides a list of potential surveys for which generic clearance activities might be conducted, along with estimates of the number of respondents and burden hours that might be involved in each.
Table 1: Potential surveys for improvement projects, with the number of respondents and burden hours
|
Number of Respondents |
Number of Hours |
Early Career Doctorate Survey |
500 |
500 |
Survey of Earned Doctorates |
1,200 |
600 |
Other surveys of the science and engineering workforce |
1,600 |
450 |
Higher Education Research & Development Survey |
300 |
500 |
Federally-Funded Research & Development Centers (FFRDC) Survey |
60 |
120 |
Federal Labs Survey (possible future survey) |
275 |
525 |
State Government Research & Development Survey |
150 |
225 |
Survey of Nonprofit Research Activities |
225 |
550 |
Business Research & Development Survey |
50 |
150 |
Annual Business Survey |
300 |
450 |
Survey of Scientific & Engineering Facilities |
300 |
300 |
Public Understanding of Science & Engineering Survey |
550 |
125 |
Data dissemination tools and mechanisms |
100 |
100 |
Other surveys and projects not specified |
10,000 |
3,000 |
Total |
15,610 |
7,595 |
The cost to respondents generated by the list of potential projects is estimated to be $757,866 over the three years of the clearance. No one year’s cost would exceed $757,866 , so if all work were done in one year, costs in that one year would be $757,866 and the costs in each of the other 2 years would be zero. As in previous requests for generic clearance authority, the total cost was estimated by summing all the hours that might be used on all projects over the three years (15,610) and multiplying that figure by the hourly wage $48.55) of the level of employee who typically answers NCSES’ questionnaires or attends NCSES workshops. This wage amount is the May 2018 national cross-industry estimate of the mean hourly wage for a financial analyst, or Job Category 13-2051, by the Bureau of Statistics (http://www.bls.gov/oes/current/oes132051.htm, accessed on 4/19/2019). The total hours are based on similar NCSES projects over the past few years.
There are no planned capital, startup, operation or maintenance costs to the respondents, recordkeepers, or data users involved in these improvement projects. Some explorations involving the use of alternative sources of data (e.g., converting respondents’ records into a standard format for upload) may entail some costs; in such events, details would be provided in the materials associated with that burden request.
The 3-year cost to the Federal government generated by the improvement projects is estimated to be approximately $2,800,000 (estimate based on $200,000 per line in Table 1). The main components of these costs are contractor costs and staff time. There are no startup, equipment, operations or maintenance costs. Bidders on the NCSES contracts are required to have all software, licenses, and hardware needed to complete the survey improvement projects. The costs generated by future data collections will be described in the clearance request for each specific data collection.
The request for 15,610 burden hours is higher than the current generic clearance (11,180 hours). Because NCSES anticipates taking advantage of new ways of recruiting large numbers of participants for small-scale studies (e.g., Amazon Mechanical Turk, or MTurk), the agency has requested a higher number of respondents and hours, compared to the current generic clearance.
Data will be collected to develop new surveys, improve the content or methodology of current surveys, explore alternative sources of data, or evaluate data dissemination mechanisms. Methodological findings from improvement projects may be referenced in the technical notes for published data, in methodology reports, in technical papers presented at conferences, in the proceedings of conferences, or in journals. Generic clearance activities will not be used to calculate substantive results or estimates that will be released.
A17. OMB Approval Expiration Date
NCSES will display the expiration date for OMB approval of the information collection on survey instruments.
A18. Exceptions to the Certification Statement
No exceptions to the Certification Statement should be required. If so, OMB approval will be requested in advance of conducting the survey or data collection.
0 NSF Information Quality Guidelines are available on http://www.nsf.gov/policies/infoqual.jsp. OMB Information Quality Guidelines are available on http://www.whitehouse.gov/omb/inforeg/infopoltech.html. OMB standards and guidelines for statistical surveys are available on http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |