3145-0174 Annual Report - 2012 Clearances

National Science Foundation
4201 Wilson Boulevard
Arlington, Virginia 22230

May 24, 2013

Ms. Shelly Wilkie Martinez

Office of Management and Budget

New Executive Office Building

Room 10201

Washington, DC 20503

Dear Shelly:

In May 2010, the National Science Foundation’s National Center for Science and Engineering Statistics (NCSES) generic clearance for survey improvement projects (OMB Number 3145-0174) was extended for three years, with the stipulation that reports be submitted annually, containing indicators of the work that was conducted under the clearance. Between January 1, 2012 and December 31, 2012 NCSES, conducted five studies under projects approved by OMB for the SRS generic clearance for this three-year period. These five projects covered redesign issues for two current surveys and exploratory work for the Microbusiness Innovation Science and Technology Survey and the Early Career Doctorates Study. The report that follows describes the work conducted for studies during calendar year 2012.

The methods employed in these generic clearance activities included interviews, web surveys, and usability testing of another web survey. The interviews were conducted face-to-face or over the phone, and utilized semi-structured protocols to elicit unbiased information about how respondents perceived items such as definitions, instructions, questions, and response categories. The usability testing was conducted by phone using conferencing software to view respondents’ computer screens as they completed the survey tasks.

The number of respondent burden hours used during 2012 in the five studies described below totaled approximately 1410 hours, which includes 30 hours for recruiting participants for these projects. Of the remaining 1380 hours, the largest share - 1331.5 hours - were used for self-administered survey tests, while 11 were for face-to-face interviews, and 38 were used for phone interviews. These hours cover the 1945 individuals who participated in these studies.

The generic clearance allowed us to make improvements for two current surveys and to make progress on the design of two new surveys. We explored barriers to online reporting for our Survey of Earned Doctorates. For designing a new survey on microbusiness R&D, we worked with respondents to refine survey items and explore the feasibility of collecting data on a full-scale survey. For designing a new survey on early career doctorates, we tested contact strategies to increase response. In 2012, NCSES staff continued to use the generic clearance to improve the quality of current and potential NCSES data collections. Future activities under the generic clearance will continue to contribute to improving the survey experience for respondents as well as increasing the quality of NCSES data collections.

Brief descriptions and findings of the 2012 generic clearance activities by survey or initiative follow.

Early Career Doctorates Study (ECDS)

In 2012, NCSES received OMB generic clearance to test a data collection strategy that uses institutional contacts as the conduit for questionnaire dissemination to reach early career doctorates (ECD). The primary goals of the research were to a) test methodologies for building a sample frame of ECD, b) contact a sample of ECD to complete a web survey, and c) to assess strategies for recruiting ECD. A secondary goal of the research was to test the ECD questionnaire, which had been revised based on results from the 2009 questionnaire pretest as well as stakeholder feedback.

RTI, contractor for the ECDS, conducted the methodological study. Since the data collection phase recently ended on February 25, 2013, NCSES has not thoroughly completed the data analysis yet. However, initial findings related to the primary goals indicate that having institutions contact the ECD for the pre-notification of the survey as well as the final appeal yields better response rates than having all of the contacts come from NSF (via RTI).

The total burden for the ECDS was 1,304 hours, split between two stages. The burden hours used were less than initially estimated due to the fact that only 56 of the 74 institutions that agreed to participate in the survey submitted lists of ECD. During Stage 1, where the focus was creating the frame for the survey, two of the 74 Points of Contact (POC) did not respond to our contacts. On average it took the POC 5.6 hours to compile the ECD list resulting in approximately 400 burden hours (72 POC x 5.56 hours = 400 hours). The burden hours for the High Authority (HA, typically a president, who is contacted in order to inform them of the project, seek their participation, and request a point of contact) in Stage 1 was approximately 61 hours (82 HA x 44.5 minutes = 60.8 hours). A total of 461 burden hours were used in Stage 1.

During Stage 2, where the focus shifted to collecting data from ECD, we sampled 2,115 individuals from 56 institutions that provided the ECD lists; 1,519 individuals responded either through the survey or by directly contacting NSF or RTI. Most of the sampled ECD (1,453) started the survey and spent an average of 33.4 minutes. Of those that completed the survey, the average time to complete was 36.4 minutes. An additional 66 people responded outside of the web survey system to indicate that they would not complete the survey; we estimated the burden at 10 minutes per person bringing the overall burden per responding ECD to 32.4 minutes. The burden hours for the ECD was 820 hours (1,519 ECD x 32.4 minutes = 820 hours). The burden hours for the High Authority was 23 hours (29 HA x 48 minutes = 23 hours). Therefore, 843 total burden hours were used in Stage 2.

Microbusiness R&D Survey

1. Cognitive interviews

NCSES received OMB generic clearance to conduct cognitive interviews with microbusinesses (fewer than five employees) in 2011. These interviews focused on the ability of these very small companies to answer selected research and development (R&D) and innovation related questions. During 2012, NCSES and Westat (the survey contractor) conducted 45 interviews. Eleven interviews were conducted in person, and 34 were conducted over the telephone using WebEx. Web Ex is a web conferencing tool that allows the respondent to view a PDF version of the questionnaire from the interviewer’s computer screen while talking on the phone. Respondents for both modes completed the questionnaire via a read aloud process.

Each interview lasted an average of one hour for a total of 45 hours of burden. To recruit these respondents, approximately 290 microbusinesses were contacted. These recruiting calls lasted an average of two minutes resulting in almost 10 burden hours (290 contacts x 2 minutes = 9.7 hours). The overall burden for this project was approximately 55 burden hours (45 hours for completing the cognitive interviews and 10 hours for recruiting).

2. Preliminary test of Data Collection Instrument

In a separate request, NCSES received OMB generic clearance to conduct a preliminary test of the questionnaire to gather data to refine our questionnaire items. This test allowed NCSES to administer the survey to a wider set of respondents from more industry sectors than was possible for the cognitive interviews. During 2012, 74 questionnaires were completed with microbusinesses. Respondents completed a self-administered questionnaire after being recruited by telephone or e-mail.

Completing the questionnaire took approximately 15 minutes for a total of 18.5 hours. To recruit these respondents, approximately 350 microbusinesses were contacted via telephone or mail. Both the recruiting calls and the time needed to review the advance letter took an average of two minutes resulting in almost 12 hours (350 x 2 minutes = 11.7 hours). The overall burden for this project was just over 30 hours (18.5 hours for completing the questionnaire and 11.7 hours for recruiting).

Survey of Earned Doctorates (SED)

The SED is conducted annually to collect data on individuals who received research doctorates from accredited U.S. institutions. For the past several years, SED has offered a web version of the survey, which has proven to be an efficient and cost-effective means of collecting data. Some institutions have been reluctant to move to this mode of administration, however.

In 2012, NCSES received OMB generic clearance to conduct semi-structured interviews to understand and identify the barriers institutions face in adopting the web survey mode for the SED. The National Opinion Research Center (NORC) at the University of Chicago, the current contractor for the SED, conducted the interviews, and made the following three recommendations:

1. Differentiate between the web survey and the PDF version. Some institutions confused the PDF version of the SED, which respondents and print and complete on paper before submitting to their institution, with the web version of the survey, which is administered online.

2. Retool the materials that are sent with the SED, paying attention to their messaging and appearance, so that they are more clear.

3. Expand and tailor the initial contact strategy to ensure that the SED information is conveyed to both the appropriate authority figure within the institution (typically, a dean) as well as the staff member tasked with coordinating the effort. Tailor the timing of the contact to the graduation schedule of the institution.

Sixteen telephone interviews were conducted by NORC. The time to complete each interview averaged 15 minutes. The recruiting contacts (conducted via e-mail) averaged 2 minutes to contact 22 institutions. The total burden for this study was approximately 5 hours (44 minutes for recruiting and 4 hours for interviewing).

Survey of Science and Engineering Research Facilities

The Survey of Science and Engineering Research Facilities is a congressionally mandated, biennial survey that is the primary source of information about science and engineering research facilities located at U.S. research-performing colleges and universities. In response to a concern about possible double-counting of facilities in the survey, especially in cases where there are multiple campuses within a system, NCSES received OMB generic clearance to conduct a methodological study to clarify which campuses were included in the responses provided during the FY2011 Facilities Survey.

Two hundred and twelve of the 541 institutions that participated in the FY2011 Facilities Survey were identified as having multiple campuses, locations, or other entities. NCSES’s contractor for the Facilities Survey, Westat, e-mailed the established survey contacts at those 212 institutions and provided a link to a web questionnaire that provided a list of the campuses, schools, branches, and other entities that were part of their larger institutional structure. Institutions were asked to clarify whether or not research space for those entities was reported in the FY2011 survey cycle.

Because of the study, NCSES discovered that institutions generally reported correctly. However, as a precautionary measure, some of the language in the introductory instructions may be modified for the FY2013 Facilities Survey, which will begin in October 2013.

Of the 212 institutions contacted, 108 responded, and the questionnaire generally took less than 5 minutes to complete, which resulted in an overall burden of approximately 9 hours (108 institutions x 5 minutes = 9 hours). To recruit these respondents, 212 institutions were contacted via e-mail, which took an average of two minutes, resulting in a burden of approximately 7 hours (212 x 2 minutes = 7 hours).

Sincerely yours,

Rebecca L. Morrison

National Center for Science and Engineering Statistics

National Science Foundation

cc:

S. Plimpton (IRM/DAS)

J. Gawalt (SBE/NCSES)

S. Cohen ( ‘’ )

1