Redesign of the NSF Survey of Research and Development Expenditures at Universities and Colleges

Featherston/Britt

November 29, 2007

Questions for Universities

Interviews for Ethnographic Survey System Evaluation

Academic R&D Survey Redesign

Introduction

We are visiting universities to find out what practices and approaches they use to collect and report R&D data, both within the institution as well as to other organizations such as NSF. We will be using this information to find out what we are doing right and what needs improvement for the Academic R&D Expenditures Survey.

We would like to tape this interview so that we have all your comments. Would that be okay? [Have respondents sign consent forms.]

Today, we would like to walk through the entire process used at your institution for R&D data. We would like to discuss everything from generating the data for your information systems to downloading the data for your survey response. We are especially interested in the topics from the survey, but we also are interested in R&D data that you may collect for other purposes.

We are conducting these interviews to help us make decisions for the redesign of the Academic R&D Expenditures Survey. It will be extremely helpful to that effort if we can better understand the variety of ways that institutions collect and use R&D data.

In our e-mail, we asked to meet with you as well as representatives from other places in the university. I'd like to get a sense of who is here for this first meeting and who we might visit with either later today or at another time.

[Read list of roles and check for who is and is not present.]

1. Vice President for Research or someone from that office (to discuss the measurement of basic research, institutional uses of the data, and so forth)

2. Sponsored Research office (to discuss the pre- and post-award administration process, classification of fields of science and sources of funding)

3. Accounting or Budget office (to discuss the classification of sources of funding and reporting of indirect costs and equipment expenditures)

4. Institutional Research office (to discuss the uses of R&D data for comparisons with peer institutions)

I have a series of questions that I’ll be referring to as we go along. Since these are the first interviews we’re doing for this project, please let me know if there are any other issues that you think we should be covering to find out about R&D data at your institution. OK?

I. High level questions

Walk through a typical research grant

1. Before we talk specifically about the survey, could you take 5 or 10 minutes and walk me through the grant process at this institution? I’d like to hear about the various offices at your institution and how they are involved in this process.

2. Could you tell us about the indirect costs? How do the indirect costs for a grant relate to the expenditures that you track?

3. Do any of the indirect costs recovered go into accounts that are later used to fund other research projects? Can you tell how much of the indirect costs are used this way?

4. Next, we’d like to ask about how institutions answer the question on basic research in Item 1. Does your institution use a form or a database to record information on whether research is basic or applied?

5. If we decided to publish our information on expenditures for basic research by institution, would institutions be willing for us to show a total and federal figure?

I. High level questions (continued)

Uses of R&D information

6. Many institutions call us and ask for our R&D data so that they can use it to compare themselves against peer institutions. Could you tell us about comparisons that are important to your institution?

-- What kind of R&D comparisons, if any, do you make between your institution and other institutions?

-- What kind of R&D comparisons, if any, do you make between departments at your institution and similar departments at other institutions?

-- What information do you use to make R&D comparisons within your institution?

-- What kind of comparisons, if any, do you make among departments at your institution?

7. When you’ve done analyses in the past, were there ever specific types of R&D information that you wanted but did not have on a national basis?

--Could you tell me more about what type of information you wanted?

8. As we may have already mentioned, we held a meeting of experts to discuss what issues we should consider as we redesign the Academic R&D Survey. In our next several questions, we’d like to talk with you about some options this expert panel raised. [Hand out copies of Table for Respondent’s state.] I’m giving you a copy of the page from our state table to give you an idea of the other institutions that answer the survey in your state. It might serve as a reference as we discuss some of the issues.

9. The expert panel suggested that there is a “research enterprise” that could be surveyed for each institution. It might even include financially separate organizations. If I asked you to describe the research enterprise for this institution, what would you include?

10. At the expert panel meeting, they discussed which organizational units get included in the survey responses for an institution. Currently, we include units only if their research expenditures flow through the university. Do you have any situations here with faculty performing sponsored research that does not flow through the institution’s accounting system?

If yes:

--Could you tell me the types of organizational units where this occurs? (As necessary:) How many units would you have like that?

-- Do the faculty members have joint appointments at the other institution?

-- Are there faculty members who have joint appointments or research collaborations with industry?

-- Are there faculty members who work on sponsored project with PIs from another institution? How does the funding get tracked in that situation?

-- Could you tell me more about how the research dollars are handled in these situations?

11. We are currently exploring information available from IRS for non-profit institutions—501(c)3 organizations— other than academic institutions. Do you know how we might distinguish these two types of nonprofits?

-- Some universities have set up separate research foundations to manage their sponsored research funds. Do you know how we might distinguish this type of nonprofit as an academic institution?

12. What is your opinion about which research expenditures we should ask your institution to report for the survey?

--[As needed:] Which research locations might you include?

13. Another issue is the type of institutions that we include in the survey. Our survey title says “Universities and Colleges.” However, there are some institutions such as the RAND Graduate School and Scripps Research Institute that grant degrees but are not usually thought of as a university or college. We also include non-degree granting institutions such as Woods Hole Oceanographic Institute and the Desert Research Institute, because they are comprised of students and faculty from nearby universities and colleges. What is your opinion about how we should define which institutions are included in our survey’s population?

14. [Hand out/Refer to survey’s Item 2.] I’d like to show you our list of science and engineering fields. Our expert panel suggested that we also collect information on emerging and interdisciplinary fields. What fields would you add that are not on this list? (As needed: )They could be fields that overlap with our current list.

-- In order to give us an idea of possible emerging fields, could you tell us about any new departments, programs or centers your institution has added within the last few years?

15. Our expert panel recommended that we start including R&D that is NOT science and engineering in our totals for R&D. Our current categories for these expenditures include education, library science, law, social work, communications, arts, business, and the humanities. What is your opinion of including these non-science and engineering expenditures in some of our ranking tables and all other totals for R&D?

-- Do you think there is a substantial amount of non-S&E R&D being performed at your institution?

16. Next, we’d like to talk about the term, “subrecipients,” which we use in Questions 1A and 1B. Could you tell me in your own words what you consider to be a “subrecipient?”

17. Now, could you look at Handout 1? It has some definitions of subrecipient from our questionnaire as well as a definition from OMB Circular A-133. Which parts of these definitions would be helpful for Questions 1A and 1B?

18. Our expert panel recommended that we might consider collecting separate data on direct and indirect costs to help institutions compare themselves with each other. What is your reaction to that?

--How would the comparison of direct costs for research differ from comparing all research costs?

19. If we did ask institutions to report direct and indirect costs, which breakouts would be most useful to you?

Probe as needed:

Source of funds? (Show Handout 2)

Field of S&E?

Federal and Total by Field of S&E?

Federal agency?

Federal agency by Field of S&E?

Closing (as needed for high level officials)

20. Are there any other areas of the survey or instructions that we haven’t covered that you’d like to discuss?

II. Respondent questions

People and offices involved in R&D data

21. I’d like to start by asking you to take a few minutes to describe the people involved in gathering the information for the questionnaire. Tell me about anyone that is involved in the process of collecting and reporting the survey answers to us.

Which person or office helps you with the Academic R&D Survey data?	How does this person help you complete the survey?	(If applicable:) Which data do they provide?

22. Could you tell us [more] about databases and software you use to come up with the answers to the survey?

Data systems	Items on survey	Description of use

II. Respondent questions (continued)

Methods Used

23. Now we’d like to talk about the process you use when you are working on the survey—how you go about working on the survey. Which item do you usually start with?

24. Do you enter data directly into the web survey or do you fill out a paper version first?

25. Do you have information that is institution-wide for all the survey items? Or, do you have to compile information from separate components and entities that have different information systems for some items?

(As needed :) Which items require you to use different sources for different units within your institution?

26. What time of year are the data first available? And, how does that fit the survey’s current due date of late January?

II. Respondent questions (continued)

27. We’ve talked about the various sources of funding for R&D across your institution. Now we’d like to ask about the types of units that you include to arrive at your total R&D expenditures.

What about…	(IF “YES”) Is it included in your answer to Item 1? How do you find out their expenditures?
	Yes	No	Do not have	Comments
Research centers that are tied to specific departments
Research foundation that is associated with the university
Hospitals owned by the university
Hospitals NOT owned by the university
Agricultural experiment stations
Other nonprofit organizations related to your institution
Nonuniversity locations where faculty are performing research as part of a joint appointment

28. [Institutions that report medical sciences only:] If we asked you a new question about the medical school R&D expenditures, how would you go about deciding what to report?

-- Which departments of the medical school would you include?

II. Respondent questions (continued)

29. [Hand out survey form as needed.] Next we’d like to talk about Item 1. As you probably know, Item 1 asks about various sources of funds for research. Could you tell us how you decide what to put in each category? [Use table below to record answers]. The first one is the federal government.

As needed:

-- Where do you report funding from other universities?

-- Where do you report funding from state governments that are not your own state?

-- Where do you report funding from nonprofit organizations?

-- Where do you report funding from a private university?

-- Where do you report funding from foreign sources?

Source of funds	-- What types of funds do you report in [name of category]?
Federal
State and local governments
Industry
Institutional funds
(1) Institutionally-financed organized research
(2) Unreimbursed indirect costs	-- How do you calculate row d(2), “unreimbursed indirect costs and related sponsored research?
All other funds
Total

30. Our expert panel suggested that we consider another category for nonprofit organizations. If we added such a category, how would that change the way you currently report your expenditures?

31. Are there any other changes you would suggest for the categories we have for sources of funding?

32. Currently we say that the breakout for institutional funds is confidential, but we were asked to consider whether or not to change that. If it were not confidential, we could publish numbers for each institution. However, some institutions might be hesitant to report the breakout if it were to be published. What is your reaction on whether this information should be published by institution or should be kept confidential?

-- If we ask for breakouts for indirect costs, would universities be willing for us to publish the totals for indirect costs for their institutions?

33. The numbers that institutions report for basic research are also confidential, and we would also like to get your reaction to whether or not to change that. Again, the tradeoff is that confidential information cannot be published for each institution, but institutions might not report basic research if it were published. What is your reaction on whether basic research should be published by institution or should be kept confidential?

34. Next we’d like to talk about sources of information that you use for fields of science and engineering in Item 2. How do you generate the actual survey number?

(As necessary:) Do you pull numbers from spreadsheet, extract number from database, copy number from paper report, or something else?

35. What do you do if a grant has multiple investigators who are in different fields of science?

-- What do you do if a grant has a principal investigator that has joint appointments in several departments?

-- What do you do if the grant is in a multidisciplinary research center?

36. If we asked you to report all the sources of funding in Question 1 by field of science and engineering, what extra steps would you need to take?

37. If we collected information using different fields such as biotechnology, what extra steps would you need to take?

-- What about nanotechnology?

38. Next, we’d like to talk about Questions 1A and 1B. What are you currently reporting for these questions?

39. Next, we’d like to talk about the term, “subrecipients,” which we use in Questions 1A and 1B. Could you tell me in your own words what you consider to be a “subrecipient?”

40. Now, could you look at Handout 1? It has some definitions of subrecipient from our questionnaire as well as a definition from OMB Circular A-133. Which parts of these definitions would be helpful for Questions 1A and 1B?

41. Next we’d like to talk about sources of information that you use to determine the amount spent on R&D equipment in Item 3. What is your basis for classifying items as “equipment”?

--(As necessary:) What is your dollar basis for capitalizing equipment?

--(As necessary:) Do you have a number of years for the equipment’s life expectancy that you use for capitalizing equipment?

42. Would you report equipment that was purchased under a specific equipment-only grant?

43. Is there a maximum dollar amount above which an item would not be considered equipment or would need to be purchased through a non-operating account? [As needed:] If it would not be classified as equipment, what would it be classified as?

44. Other possible issues for interview (if time permits):

Question 2B (federal agencies, federal R&D)

Question 2A (non S&E R&D)

Closing

45. Are there any other areas of the survey or instructions that we haven’t covered that you’d like to discuss?

Handout 1

Definitions of Subrecipient

NSF Survey Items 1A and 1B instructions:

“For Federal awards, subrecipient means the non-Federal entity that expends Federal awards received from a passthrough entity to carry out a Federal program, but does not include an individual that is a beneficiary of such a program. A subrecipient may also be a recipient of other Federal awards directly from a Federal awarding agency. —OMB Circular A-133, Section .105 (revised June 27, 2003)

“For awards from non-Federal sources, the subrecipient definition is analogous to the Federal one.”

Additional instructions within NSF Survey Questions and Answers section:

“For Federal awards, a subrecipient is an entity that receives Federal financial assistance from the State or any other entity to administer a program (OMB Circular A-133, Section .210 (revised)). The subrecipient actually administers or controls the program, as opposed to the subcontractor who contracts for a specific service on a per-unit basis.

“A key factor in determining if a subrecipient arrangement exists is determining if the entity assumes the responsibility to administer the program. Subrecipients tend to be the co-authors of publications, writers of technical reports discussing findings, inventors, etc.

“Unlike a subrecipient relationship, a subcontract is a procurement of goods and/or services. Payments to subcontractors are expenditures for services, not expenditures for research, and are different from pass-through funds to subrecipients.”

OMB Circular A-133 instructions

Subrecipient means a non-Federal entity that expends Federal awards received from a pass-through entity to carry out a Federal program, but does not include an individual that is a beneficiary of such a program. A subrecipient may also be a recipient of other Federal awards directly from a Federal awarding agency. Guidance on distinguishing between a subrecipient and a vendor is provided in §___.210.

Vendor means a dealer, distributor, merchant, or other seller providing goods or services that are required for the conduct of a Federal program. These goods or services may be for an organization's own use or for the use of beneficiaries of the Federal program. Additional guidance on distinguishing between a subrecipient and a vendor is provided in §___.210.

§___.210 Subrecipient and vendor determinations.

(a) General. An auditee may be a recipient, a subrecipient, and a vendor. Federal awards expended as a recipient or a subrecipient would be subject to audit under this part. The payments received for goods or services provided as a vendor would not be considered Federal awards. The guidance in paragraphs (b) and (c) of this section should be considered in determining whether payments constitute a Federal award or a payment for goods and services.

(b) Federal award. Characteristics indicative of a Federal award received by a subrecipient are when the organization:

(1) Determines who is eligible to receive what Federal financial assistance;

(2) Has its performance measured against whether the objectives of the Federal program are met;

(3) Has responsibility for programmatic decision making;

(4) Has responsibility for adherence to applicable Federal program compliance requirements; and

(5) Uses the Federal funds to carry out a program of the organization as compared to providing goods or services for a program of the pass-through entity.

(c) Payment for goods and services. Characteristics indicative of a payment for goods and services received by a vendor are when the organization:

(1) Provides the goods and services within normal business operations;

(2) Provides similar goods or services to many different purchasers;

(3) Operates in a competitive environment;

(4) Provides goods or services that are ancillary to the operation of the Federal program; and

(5) Is not subject to compliance requirements of the Federal program.

(d) Use of judgment in making determination. There may be unusual circumstances or exceptions to the listed characteristics. In making the determination of whether a subrecipient or vendor relationship exists, the substance of the relationship is more important than the form of the agreement. It is not expected that all of the characteristics will be present and judgment should be used in determining whether an entity is a subrecipient or vendor.

(e) For-profit subrecipient. Since this part does not apply to for-profit subrecipients, the pass-through entity is responsible for establishing requirements, as necessary, to ensure compliance by for-profit subrecipients. The contract with the for-profit subrecipient should describe applicable compliance requirements and the for-profit subrecipient's compliance responsibility. Methods to ensure compliance for Federal awards made to for-profit subrecipients may include pre-award audits, monitoring during the contract, and post-award audits.

(f) Compliance responsibility for vendors. In most cases, the auditee's compliance responsibility for vendors is only to ensure that the procurement, receipt, and payment for goods and services comply with laws, regulations, and the provisions of contracts or grant agreements. Program compliance requirements normally do not pass through to vendors. However, the auditee is responsible for ensuring compliance for vendor transactions which are structured such that the vendor is responsible for program compliance or the vendor's records must be reviewed to determine program compliance. Also, when these vendor transactions relate to a major program, the scope of the audit shall include determining whether these transactions are in compliance with laws, regulations, and the provisions of contracts or grant agreements.

Handout 2

Example of Cost Breakouts

	Type of Costs
Source of Funds	Direct	Indirect
Federal
State and local
Industry
Institutional
All other sources

14