Report on 2010 Generic Clearance Activities

National Science Foundation
4201 Wilson Boulevard
Arlington, Virginia 22230

May 27, 2011

Ms. Shelly Wilkie Martinez

Office of Management and Budget

New Executive Office Building

Room 10201

Washington, DC 20503

Dear Shelly:

In May 2010, the National Science Foundation’s Division of Science Resources Statistics (SRS) generic clearance for survey improvement projects (OMB Number 3145-0174) was extended for three years, with the stipulation that reports be submitted annually, containing indicators of the work that was conducted under the clearance. Between January 1, 2010 and December 31, 2010, SRS, now the National Center for Science and Engineering Statistics (NCSES), conducted three studies under projects approved by OMB for the SRS generic clearance for this three-year period. Projects covered redesign issues for two current surveys and exploratory work for the Microbusiness Innovation Science and Technology Survey. The report that follows describes the work conducted for studies during calendar year 2010.

The methods employed in these generic clearance activities included face-to-face interviews, a short survey conducted via email, and interviews over the phone. The face-to-face interviews utilized cognitive testing methods to elicit unbiased information about how respondents perceive items such as definitions, instructions, questions, and response categories.

The number of respondent burden hours used during 2010 in the three studies described below totaled 57 hours. A breakdown by method shows the following: 34 hours for face-to-face interviews, 14 hours by phone, and 9 hours by email. These hours cover the 51 individuals who participated in these studies. Participation for these individuals ranged from 30 minutes for a survey by email to two hours for face-to-face interviews during cognitive testing. The estimate of 57 hours includes 7 hours for recruiting.

The 2010 calendar year was a period of low activity for our generic clearance. Nevertheless, the generic clearance allowed us to make improvements for two current surveys and to make progress on the design of a new survey. This work included testing the web survey before launching our redesigned survey for higher education R&D. We conducted a small set of cognitive interviews for an ongoing survey. For the new survey, we requested standardized information prior to a workshop activity. Our staff continued to pursue improvements in the quality of current and potential NCSES data collections. Future activities under the generic clearance will continue to contribute to increasing data quality and improving the survey experience for respondents.

Brief descriptions and findings of the 2010 generic clearance activities by survey or initiative follow.

Higher Education Research and Development (R&D) Survey

In the summer of 2010, SRS and Westat staff received OMB generic clearance to conduct 15 remote web usability tests for the Higher Education R&D Survey. These tests focused on the navigational changes made to the redesigned survey after the FY 2009 pilot test with 40 institutions. Current survey respondents were tested over the phone using web conferencing software. This software provided a visual display of the respondent’s computer activity as well as speakerphone contact during the test. Results of these interviews led to numerous revisions to the survey website. These changes were incorporated before the rollout to the full population of 700 institutions for the new Higher Education R&D Survey for FY 2010.

Due to limited time and resources, a total of only 6 web usability tests were conducted during calendar year 2010. The tests averaged 2 hours in duration with a total of 6 individuals, for an estimated 12 burden hours. The recruiting contacts averaged 15 minutes considering the initial email and a follow-up phone call to arrange an acceptable time and explain the testing procedures. Total recruitment burden is estimated to be 2 hours for the 6 respondents that we contacted. Total burden for this study was 14 hours (12 hours for the survey and 2 hours for recruitment).

Microbusiness R&D Survey

SRS conducted a data user workshop in September 2010 to discuss issues for a future survey of companies with less than five employees (microbusinesses). The objectives of the data user workshop were twofold: (1) gain a better perspective on data user needs and priorities of needs among users and (2) understand how microbusiness data might be used. Workshop participants were provided with a draft questionnaire that was the result of previous testing with 43 U.S.-based small businesses.

In order to better inform the workshop discussion areas, SRS requested clearance to send to the workshop invitees a brief survey regarding their data use and needs. This survey consisted of seven questions and was e-mailed to workshop participants prior to the workshop. Results from the survey were presented at the workshop.

The time to complete the survey was estimated at 0.5 hours for each of 15 respondents for a total of 8 hours of burden. The recruiting contacts were estimated to average 2 minutes for each respondent in order to read the e-mail they received. For the 23 respondents contacted, there was a total of 1 hour of burden for recruiting. The total burden associated with the MIST pre-workshop survey was 9 burden hours (8 for completing the survey and 1 hour for the recruitment).

Survey of Earned Doctorates (SED)

The SED, conducted annually since 1957, collects data on all individuals receiving research doctoral degrees from accredited U.S. institutions. These data are used to assess characteristics of doctorate recipients and trends in doctorate education and degrees. In 2010, SRS conducted 30 cognitive interviews with doctoral candidates to evaluate two alternate versions of questions measuring the respondent’s field of research. These interviews were planned to evaluate how this question might measure interdisciplinary research. Of the 30 respondents, 20 were from departments considered either very likely or somewhat likely to encourage interdisciplinary research. Each respondent was given a $30 honorarium. These interviews helped us understand how respondents define the two concepts of interdisciplinary and multidisciplinary research. Furthermore, we explored to what extent, if any, respondents relate these two concepts to their own research activities.

The cognitive interviews showed that the question on dissertation research field is a reasonable indicator of interdisciplinary research, but there is room for improvement. While all 30 respondents had a general understanding of the terms interdisciplinary and multidisciplinary, there were substantial variations. The interviews showed that the question on dissertation research field might be problematic for individuals in doctoral programs that are explicitly interdisciplinary. NCSES will use the results of the cognitive interviews when evaluating revisions to the SED prior to the next survey round.

The cognitive interviews averaged 60 minutes in duration for 30 individuals, for an estimated burden of 30 hours. SRS contacted 31 interviewees to recruit these 30 individuals. The recruiting contacts averaged 7 minutes including 5 minutes for a phone call to explain the interview process and scheduling and 2 minutes for one or more follow-up emails. Total recruitment burden is estimated to be 4 hours (31 respondents times 7 minutes is 217 minutes, or 3 hours and 37 minutes). Total burden for this project was 34 hours (30 hours for the interviews and 4 hours for recruitment).

Sincerely yours,

Fran Featherston

National Center for Science and Engineering Statistics

National Science Foundation

cc:

S. Plimpton (IRM/DAS)

L. Carlson (SBE/NCSES)

J. Gawalt ( ‘’ )

J. Jankowski ( ‘’ )

E. Rivers ( ‘’ )

S. Cohen ( ‘’ )