Technical Report 10-010

Technical Report 10-010

Are Individuals Able To Accurately Report Whether Their Work Has Been Supported by the 2009 American Recovery and Reinvestment Act?

March 2010

A Report to the National Science Foundation

Division of Science Resources Statistics

By

Morgan M. Millar

Benjamin L. Messer

Shaun S. Genter

Don A. Dillman

SESRC

Social & Economic Sciences Research Center (SESRC)
P.O. Box 644014
Washington State University
Pullman, Washington 99164-4014
Telephone: (509) 335-1511
Fax: (509) 335-0116

Are Individuals Able To Accurately Report Whether Their Work Has Been Supported by the 2009 American Recovery and Reinvestment Act?¹

March 2010

A Report to the National Science Foundation

Division of Science Resources Statistics

By

Morgan M. Millar, Benjamin L. Messer, Shaun S. Genter, and Don A. Dillman

Social and Economic Sciences Research Center

Washington State University

Pullman, WA 99164-4014

Table of Contents

INTRODUCTION ………………………………………………………………………………………………. ………………. 6

RESULTS FROM A GENERAL HOUSEHOLD SURVEY …………………………………………………………….. 6

Study Design …………………………………………………………………………………………………………………………. 6

ARRA Questions in the Household Survey …………………………………………………………………………….. 7

How Many Respondents Reported Support for Their Employment? …………………………………….. 7

Conclusions and Implications ……………………………………………………………………………………………….. 12

RESULTS FROM COGNITIVE INTERVIEWS WITH FACULTY, POST-DOCTORATES,

AND RESEARCH ASSISTANTS …………………………………………………………………………………………….. 12

The ARRA Questions ……………………………………………………………………………………………………………… 12

Procedures ……………………………………………………………………………………………………………………………. 13

Sample ………………………………………………………………………………………………………………………………….. 13

Results ………………………………………………………………………………………………………………………………….. 14

Interpretation ……………………………………………………………………………………………………………………….. 21

Conclusions and Implications ………………………………………………………………………………………………… 24

Potential Problems with Other Survey Questions ………………………………………………………………….. 25

CONCLUDING STATEMENT ………………………………………………………………………………………………... 31

APPENDICES ………………………………………………………………………………………………………………………. 33

Appendix A: Questionnaire Used in Cognitive Interviews ……………………………………………………….. 33

Appendix B: Other Materials Used in Cognitive Interviews …………………………………………………….. 38

Are Individuals Able To Accurately Report Whether Their Work Has Been Supported by the 2009 American Recovery and Reinvestment Act?

INTRODUCTION

The National Science Foundation Division of Science Resources Statistics requested that we evaluate whether people can provide an accurate answer to questions such as the following, which is being considered for inclusion in the 2010 surveys of college graduates and doctoral recipients:

“During 2009, was any of your work supported by the Federal stimulus bill (American Recovery and Reinvestment Act of 2009)?

Did not work in 2009

No

Yes

Obtaining an accurate answer to questions of this nature, inserted into random sample surveys, is important for being able to accurately evaluate the employment impacts of the American Recovery and Reinvestment Act (ARRA) passed by Congress and signed by the President in early 2009.

In the following, we briefly report the results from including an ARRA question in a statewide survey of 3900 general public households in Washington sampled from the U.S. Postal Service (USPS) Deliver Sequence File (DSF) of residential addresses for that state. In addition, we report the results from 20 cognitive interviews designed to test the accuracy of ARRA questions and conducted with faculty, post-doctorates, and research assistants who have degrees in a science or engineering field of study and who are currently employed at Washington State University.

Based upon the results of these experiments we draw conclusions about the validity of results obtained from asking ARRA survey questions and the possible limitations associated with their use in national surveys. Because of the quite different nature of these tests, the procedures and results will be discussed individually, beginning with results from the general public survey.

RESULTS FROM A GENERAL HOUSEHOLD SURVEY

Study Design

During the fall of 2009, we surveyed a random sample of Washington State households by mail and web, using a mixed-mode design. The survey, titled the Washington Economic Survey (WES) and sponsored by Washington State University, was conducted over the course of three months, from October 12, 2009, approximately one year after the effects of the economic recession began impacting the state, to the end of data collection on January 12, 2010. A random sample of residential postal addresses was obtained from the U.S. Postal Service’s (USPS) Delivery Sequence File (DSF), a database with near complete coverage of households in the U.S., and was weighted by urban-rural counties to ensure a sufficient number of responses from rural households.

Five contacts were mailed to respondents: 1) a pre-notification letter, 2) the survey invitation with a $5 incentive, 3) a reminder letter, 4) a follow-up survey invitation to nonrespondents, and 5) a second reminder letter to nonrespondents. As part of methodological experiments embedded in the survey design, some respondents received a web survey invitation in the second contact while others received mail; respondents that received the web invitation were sent a mail response option in the fourth contact. The web and mail versions of the survey were designed as similarly as possible to minimize measurement error resulting from using two survey modes. Also, some respondents received the fourth contact via USPS Priority Mail (vs. standard First Class) and some of these respondents also received a second $5 incentive.

The WES paper questionnaire included 12 pages of questions that asked how the respondent’s household had been impacted by the economic recession during the previous year (since October 2008). It asked about changes in employment, savings, retirement plans, health care, housing, and household expenditures, Internet and cell phone service, and demographic characteristics. On page six of the survey, questions 25 to 28 asked whether and how the 2009 American Recovery and Reinvestment Act (ARRA) affected households across the state.

The total response rate for the WES was 55.7% (2,009 responses / 3,605 sample size). Thirty-one percent (630/2009) responded over the web and the remaining 69% responded by mail. Results from web and paper responses have been combined for the analysis that follows:

ARRA Questions in the Household Survey

The WES contained these questions regarding if and how the ARRA has impacted Washington households.

Q25. In February 2009, an economic stimulus bill known as the American Recovery and Reinvestment Act (ARRA) was passed by the U.S. Congress to help stimulate the nation’s economy. An important part of this survey is to better understand if and how the stimulus bill has affected households throughout Washington.

During 2009, was any of your employment supported by the stimulus bill (American Recovery and Reinvestment Act of 2009) in some way?

N o  Skip to Q26

Y es

D oes not apply, I have not worked since February 2009  Skip to Q26

Respondents answering ‘Yes’ to Question 25 were directed to a follow-up open-ended question:

Q25a. (If yes) please explain in your own words how the stimulus bill (ARRA) has supported your employment. This question is very important, so please tell us all you can.

How Many Respondents Reported Support for Their Employment?

A total of 2,009 households responded to the WES. Of these, 1,947 provided an answer to Question 25 (Q25); the remaining 62 respondents skipped Q25. One hundred and thirty three respondents of the 1,947, or 6.8%, answered ‘Yes’ on Q25 while 73.7% answered ‘No’ and 19.5% answered ‘Does not apply’.

A total of 134 respondents provided an answer in Question 25a (Q25a). One hundred and nineteen of the 133 respondents who answered ‘Yes’ on Q25 also provided an answer to Q25a. Six respondents who answered “No” on Q25 provided an answer to Q25a. Three respondents who answered “Does not apply” on Q25 provided an answer to Q25a. And, six of the 62 respondents who did not answer Q25 provided an answer to Q25a.

Answers to Q25a were coded into five categories: Direct Support, Indirect Support, Not Sure, Indeterminable/Not Applicable, and Excluded. Below is an outline and a figure of the coding categories and results. Percentages are taken from an N of 119 (134 Total minus 15 Excluded) and the number of coded responses exceeded 119 because five respondents indicated being affected in two different ways.

1) Received direct support, N=30 (25.2%): Respondent indicated the type and/or place of their employment and explained that ARRA directly supported their employment. Below are eight examples from respondents who indicated directly receiving ARRA employment support:

R1: I was unemployed until March 11, 2009 and due to the (ARRA) I was hired by a Hanford sub-contractor for a job that is supposed to last 2.5 yrs. maybe longer. So the stimulus was a life saver for me! I just hope the cleanup funds at Hanford continue long past the 2.5 years!

R2: I was an engineer on a project funded by this program.

R3: My employer was already looking to pay for as much commuting cost for his employees as he could. The stimulus allowed him to pay 100% of transit passes instead of maxing out at $120 per month. (My monthly pass is $153, so it saved me $33 per month. GO STIMULUS!)

R4: I am employed half-time through a University Medical Center. Part of the stipend for my training was supported through the ARRA.

R5: I would have been employed .7 time, however, with the stimulus funds, I am working .8

R6: (I think it is a yes) The school district received money that put my job back on the table. Without the money, the school districts cuts would have meant no job for me.

R7: Position supported through a $15,000 grant, by the mayor's office of Arts and Public Affairs.

R8: As a youth employment counselor, my program has used ARRA funds to subsidize jobs for low-income youth. A significant (2/3) of my income has been paid for by ARRA funds during the summer months.

2) Received indirect support, N=31 (26.1%): Respondent indicated that they received ARRA support indirectly, in one of three ways:

a) N=12 (38.7%) Respondent’s employer’s customers or contractors received ARRA funds, which indirectly impacted their employment through increased sales, less layoffs, building improvements, etc. Below are five examples:

R9: Even though we are both retired, we drive cars for an auto dealer occasionally. The stimulus package did affect the car sales.

R10: We both work as managers in public organizations (health and higher education). Although our positions are not directly funded by ARRA, our organizations have received money to keep facilities open and improve physical facilities. As managers, we have received indirect benefit.

R11: My company sells information regarding government spending (all levels not just federal). Our success, or lack of failure, this year can be attributed to increased demand for the information we sell, demand stemming from the stimulus bill.

R12: Cash for clunkers. I am an Automotive Technician for a Nissan dealership.

R13: I am a mortgage consultant. The stimulus helped my refinance business by allowing some people to refinance that might not have been able. So I would say this helped my job/income be sustained. But the only part that helped was the refinance plan. I think it's called Home Affordability Refinance Plan. Other than that I received no benefit from stimulus.

b) N=9 (29.0%) Respondent received extended and/or higher unemployment benefits due to ARRA funds. Below are three examples:

R14: Received emergency unemployment compensation

R15: Extended unemployment benefit.

R16: Yes ARRA has been an unexpected blessing $25.00 per week with U.I. gross amount payment of $200.00. I was laid off my minimum wage job of five years in July 2009. Benefits are almost at end. ARRA $25.00 is bus transportation for job search and food bank trips, hospital and AA.

c) N=10 (32.3%) Respondent received an income tax break or rebate via ARRA funds.

R17: Received a tax refund to help stimulate the economy putting more money in pocket of the public to spend.

R18: Less taxes taken out of my pay checks.

3) Not sure if support was direct and/or indirect, N=28 (23.5%): Respondent indicated only that they were employed in a sector which presumably received ARRA funds (e.g. public education) but failed to mention or was unsure if and how ARRA affected their employment. Below are nine examples:

R19: I'm unsure about my department, however I work for a school district and know the stimulus has helped to keep some teachers employed. I think in some way it must have helped the transportation department.

R20: I am a K-12 educator, my wife is an educator at a local community college. Both the K-12 system in our state and the community college received money to reduce the impact of our state-level funding cuts.

R22: The stimulus bill has helped our institution a chance to provide more business to the Small business owners.

R23: I work for a non-profit that provides weatherization for low income families and some of the ARRA money is coming to our agency--- not only is this job security for myself but we're also looking to increase our staff.

R24: I work in education. Stimulus money was given to education to help support particular programs.

R25: I work road construction, there has been a lot more road work in this area of Washington state.

R26: I am a teacher in a public school that received stimulus funds.

R27: Funds made available to Head Start programs benefited my agency.

R28: We do work with the military, and I believe they have received money from ARRA, thus giving us more work.

4) Indeterminable if support was received or Not applicable to employment, N=35 (29.4%): The respondent indicated one of four ways in which ARRA support was not applicable to their employment or it was indeterminable as to whether they received any support at all:

a) N=19 (54.3%) Respondent indicated only their employment sector without mentioning ARRA or only the ARRA without mentioning their employment. Below are eight examples:

R29: I work at a public school.

R30: The stimulus money was given to the state for education

R31: Kept teachers in their classrooms, and gave new teaching curriculum to the school district.

R32: My work involves businesses. Any added monetary benefit helps maintain my company's ability to do business.

R33: Some money has been promised to utilities across the United States which has promoted some project work on their end, but as much as potentially could occur if the government would release more of that promised money.

R34: I work in the service industry, so anyone's economic boon improves my income.

R35: The fund flows through our State to support Medicare and Non-Medicare program.

R36: I really don't know what extent if any ARRA has supported my employment. I work in the hospital as a staff nurse.

b) N=3 (8.6%) Respondent entered a complaint with the ARRA or some other entry that does not apply. Below are three examples:

R37: This is not applicable to me. I am 90 years old and just surviving on widows Social Security.

R38: That stimulus is bull shit. I got nothing and I pay in the highest tax bracket, yet only make enough to own a home as a single male. I have no extra cash. The government wants to give everyone who doesn't want to work free money on my highest tax rate dollar. (WTF)

R39: Our retirement is based on interest from CD's and other interest bearing accounts. With the banks some of which I have had accounts going broke I don't receive the same return on my money as before.

c) N=9 (25.7%) Respondent indicated that they received ARRA support unrelated to their employment (e.g. home improvements, transportation, etc.). Below are four examples:

R40: I am confused by this question as it relates to employment. But in reference to the stimulus plan, I was able to purchase insulation for my home with the $600 stimulus check I received, which will save me money in heating and AC.

R41: The bill helped us because instead of using the money we were saving for other bills we used the stimulus to pay for some things. It was a big help, we were able to save a little money for at least 2 months.

R42: Cash for Clunker Program

R43: Home buyers credit.

d) N=3 (8.6%) Respondent indicated that they worked for a bank that received TARP funds. Below are two examples:

R44: I work for a bank that received bail out funds.

R45: I work for a bank that received tarp money

e) N=1 (2.8%) Respondent indicated that they were a Federal employee. Below is the only example:

R46: I'm a federal employee and manage ARRA grants for new Americorps programs.

5) Excluded based on answer to Question 25 (No, Does not apply, or Skip), N=15: These respondents provided a negative answer on Q25 but also answered Q25a. All but one respondent indicated an ambiguous comment or a reason why they chose their answer to Question 25. The one exception indicated a false negative, noting in Q25a that ARRA provided extended unemployment compensation.

The results suggest that a little over half, 51.3%, answered Q25 correctly and specifically indicated how ARRA supported their employment in Q25a, either directly or indirectly. An additional 23.5% of respondents potentially answered Q25 correctly and in Q25a indicated that ARRA could have supported their employment but did not or were unable to disclose a direct or indirect effect. Finally, 29.4% of respondents appeared to have answered Q25 incorrectly, judging from their responses in Q25a, in which respondents confused ARRA with TARP funds, received ARRA funds for things unrelated to their employment, or only expressed general concerns about the ARRA.

Finally, of the 15 Excluded respondents, those who answered Q25 negatively or not at all but still provided an answer in Q25a, only one resulted in a false negative. A respondent answering “No” on Q25 noted that they received extended unemployment benefits due to ARRA, an indirect effect. All other Excluded respondents provided answers that did not suggest ARRA was related to their employment or other area of life.

Conclusions and Implications

Methodologically, these results suggests that Q25 may obtain more positive responses than is warranted, leading to conclusions that the ARRA has had more of impact on employment than is actually the case. Moreover, the open-ended Q25a may be a necessary follow-up to Q25 in order to better identify the false positives (or false negatives). Implementing a ‘Not Sure’ answer category in Q25 could also provide respondents who are unsure an alternative, more accurate answer category that might prevent some incorrect answers. Finally, it appears that respondents may have misinterpreted or broadly interpreted “any of your employment” in Q25, with some respondents linking non-employment related ARRA funds to their employment situation. This misinterpretation is also apparent in the results from the cognitive interviews presented in the following section.

RESULTS FROM COGNITIVE INTERVIEWS WITH UNIVERSITY FACULTY, POST-DOCTORATES, AND RESEARCH ASSISTANTS

The National Science Foundation’s Division of Science Resource Statistics’ surveys are primarily focused on populations of college graduates in the science and engineering fields. However, due to the small number of respondents answering the ARRA question in the WES, and the limited demographic information available about these respondents, with these data we are unable to provide extensive analyses of differences in the answers provided by people considered to be in the nation’s science and engineering labor force. In response to this problem we conducted more highly focused interviews with people employed only in science and engineering work.

The ARRA Questions

A series of cognitive interviews were performed to evaluate a question about ARRA support among college graduates in the sciences and engineering. The question, as used in the cognitive interviews, is the original version of the question proposed by NSF. It was as follows:

Q14. During 2009, was any of your work supported by the Federal stimulus bill (American Recovery and Reinvestment Act of 2009)?

D id not work in 2009

N o

Y es  Go to Q15

In order to elicit more information about how respondents produced answers to this question, we asked those who responded “yes” to Question 14 a follow-up open-ended question. The follow-up question read as follows:

Q15. (If Yes) please explain in your own words how the stimulus bill (ARRA) has supported your work. This question is very important, so please tell us all you can.

For testing purposes, these items were placed at the end of a short questionnaire consisting of 13 other questions regarding employment, which were taken from the current NSF National Survey of College Graduates. These other questions were used to establish a context for answering the ARRA questions. They were chosen because they are items that would likely precede the ARRA items in the questionnaire, should the ARRA questions be adopted. The complete questionnaire used in the Cognitive interviews is included as Appendix A of this report.

Procedures

To evaluate the questionnaire we conducted cognitive interviews. A protocol was developed for administering the questionnaire and structuring the interviews. This protocol is included in Appendix B of this report. The interviews were conducted by Don Dillman, and each lasted between 30 and 45 minutes. During each interview, two of the remaining three authors served as note-takers.

At the beginning of the interviews, interviewees were first informed of the general purpose of the interviews (to evaluate questions considered for use in NSF’s surveys) and asked to sign a consent form (see Appendix B). Interviewees were then given a “practice questionnaire,” which contained an unrelated item, in order to illustrate how the questionnaire was to be answered and the purpose of the process (see Appendix B). They were then instructed to fill out the questionnaire, and to think aloud as they came up with their answers. Respondents were asked to point out any problems they had in answering questions, and explain why these questions were problematic. If the interviewer sensed any hesitation or confusion on the part of a respondent, he would ask them what they were thinking and probe them to provide more detail about their thought processes.

When providing responses to questions 14 and 15, respondents were asked extra probing questions to ensure that the researchers fully understood the meaning of respondents’ answers. For examples of probing questions, see list of probes in Appendix B. The interviewer made an effort to assess the degree of certainty in respondents’ answers, and to understand exactly how each individual interpreted the questions. Further information was gathered as needed about respondents’ funding situations in order to add context to the responses.

A total of twenty interviews were conducted by the authors during the weeks of January 11-15 and January 25-29, 2010. All interviewees were employees of Washington State University (WSU); they included research assistants, postdoctoral researchers, research faculty, tenure track faculty, and administrators.

Sample

We aimed to interview ten respondents who we believed to have been supported by stimulus funds as well as ten respondents who were known to have not received ARRA support. In order to recruit individuals supported under ARRA, we contacted the Director of the Sponsored Programs Services at WSU, who is in charge of ARRA reporting, and the Authorized Institutional Official and Director of the Office of Grant and Research Development. They provided a list of individuals paid from ARRA funding based on the open and transparent principles in this Act.

To create a list of potential respondents, Sponsored Program Services at WSU, authorized as noted above, compiled a list of every WSU budget-project that was funded with stimulus dollars for the period of 4/1/09-10/31/09.  WSU’s first stimulus award had a start date of 4/1/09.  This list was cross-referenced with a report from the Office of Grant and Research Development of every award received by WSU that was funded with stimulus dollars. This list was then compared to the WSU payroll system to verify salaries and wages paid on stimulus award funds.  The list of names included contact information and WSU department.

To schedule interviews, a department was randomly selected; we continued calling within that department until we were able to schedule an interview. The goal was to have the widest range of disciplines as possible. Also, a concerted attempt was made to obtain a mixture of tenure-track faculty, non tenure-track (temporary faculty), post doctorates, and graduate research assistants. Individuals were called and asked to participate in a project headed by Dr. Don A. Dillman, and sponsored by the National Science Foundation. They were asked to be interviewed for approximately 30 minutes in order to assess questions to be included in NSF questionnaires of college graduates, and were informed that they would receive a cash incentive of $35.00 to show appreciation for their time and assistance. Those who agreed to participate were sent an email to confirm the appointment.

Our first ten participants were recruited from this list, and were thus considered to be ARRA supported. Respondents 11-20, however, were not listed as having received salary under ARRA, so we considered these respondents to be not supported by ARRA. We included individuals who were not thought to have ARRA support in order to assess the likelihood of “false positive” reports, which are individuals who might indicate they received ARRA support when in fact they did not. After the first ten interviews were completed, preliminary results were discussed with the National Science Foundation. Since the preliminary results indicated that graduate research assistants had the most trouble accurately answering the question (see results below), the focus of this question was shifted to possible inclusion only in the Survey of Doctorate Recipients. Based on this discussion we were asked by NSF to switch from recruiting a wide array of college graduates to just those with a doctoral degree. We made a concerted effort to recruit postdoctoral researchers as well as faculty whom we believed had not been funded under ARRA for the final ten interviews.

Upon completion of each interview, the note-takers summarized the findings and made an assessment of whether respondents answered the ARRA question correctly or incorrectly. These assessments are summarized in the results section below.

Results

In these interviews we focused on the extent to which each surveyed individual could accurately report if their work had been supported by ARRA funding. Table 1 summarizes the positions of the respondents along with the main findings of the interviews. In this table, we document whether the individual was known to have ARRA support, whether they reported ARRA support, and our estimation of the accuracy of their responses. Also, we have indicated in the table whether the respondents expressed any uncertainty about either the intended meaning of the ARRA question or about the accuracy of their responses.

Table 1.	Summary of Interviews
Interview Number	Positiona	Evidence of ARRA funding?b	Response to ARRA question	Correct or incorrect response?	Uncertain about question meaning or response?c
1	Faculty	Yes	Yes	Correct	Yes
2	Research Assistant	Yes	Yes	Correct	Yes
3	Non-tenure Research Faculty	Yes	Yes	Correct	Yes
4	Research Assistant	Yes	Yes	Correct	Yes
5	Faculty	Yes	No	Incorrect	Yes
6	Research Assistant	Yes	No	Incorrect	No
7	Faculty	Yes	Yes	Correct	Yes
8	Research Assistant	Yes	No	Incorrect	No
9	Research Assistant	Yes	No	Incorrect	No
10	Research Assistant	Yes	No	Incorrect	Yes
11	Administrator	No	No	Correct	No
12	Faculty	No	Yes	Correct	No
13	Faculty	No	No	Correct	No
14	Faculty	No	No	Correct	Yes
15	Faculty	No	No	Correct	No
16	Postdoctoral Researcher	No	No	Correct	Yes
17	Faculty	No	No	Correct	Yes
18	Administrator/ Faculty	No	No	Correct	No
19	Non-tenure Research Faculty/ Director	No	Yes	Correct	No
20	Postdoctoral Researcher	No	No	Correct	Yes

a. Faculty members are tenured or on tenure track unless otherwise specified. Research assistants are graduate students

b. Whether or not respondent was identified as having received salary from ARRA funding

c. Whether respondent indicated uncertainty about what the question was asking for, or about the accuracy of his/her response to the question

Below we report brief summaries from each interview to provide evidence on how each respondent interpreted and responded to the ARRA question(s), and whether an accurate answer was obtained.

R1 Tenure-track Faculty, Evidence of ARRA funding, Correct Response.

This respondent replied “yes,” that her work was supported by ARRA, and explained in detail receiving the grant as a co-principal investigator. When asked a follow-up question of whether she received salary from the ARRA funded project, she then questioned her answer and said that maybe she had misinterpreted the question. She had worked on the project in October 2008, but did not believe that she would receive personal pay until the summer of 2010. We judge her original answer to be accurate, in the sense that she has received the grant. However, she does not believe that she has been paid from these funds. She was able to list off other faculty members whose pay was coming from ARRA, though. She illustrated an uncertainty as to whether the question was asking in regards to projects she is working on or about her compensation. She advised that the question more clearly explain what is meant by “your work.” She explained that the previous question, which asked about contracts or grants in 2008, encouraged her to think about receiving grants when answering the question about ARRA funds, not necessarily about whether her salary had been paid with those grants. She was not certain that her graduate students, who are being supported under special ARRA-funded positions, would know that ARRA was involved in their support.

R2 Graduate Research Assistant, Evidence of ARRA Funding, Correct Response.

This respondent knew he was paid from an NSF grant obtained by his professor. He had been verbally told that he was being paid from ARRA. Initially he seemed quite confident that ARRA was involved in the funding of the project, and that his own salary was paid with the ARRA-funded grant. He seemed to have a good deal of knowledge, for a graduate student, about how the grant money was being spent. However, upon being probed further, indicated he was not completely sure whether it was the specific ARRA grant that was funding him specifically, because his professor had multiple grants from NSF, and he had been working on those grants for four years. He thought a way to determine this would be to look at his employment documents, although he was not sure whether this would confirm if ARRA was involved. He indicated that since he was involved in an ARRA grant, if answering the questionnaire alone he might have marked “Yes without thinking” even though he was not completely certain. We believe his “yes” answer to be accurate, but he indicated that he was not entirely certain of the accuracy of his response.

R3 Non-tenure-track Research Faculty, Evidence of ARRA Funding, Correct Response.

This respondent is a Research Scientist who works in an institute at the university. His verbal answer for this question was, “Ummm. I think, yes. I think it was.” He went on to explain that his institute has ARRA funding. He was aware that there was a grant proposal that was initially turned down, but then was accepted quickly after stimulus money became available. As he recalled these details, he became more confident that ARRA funding was behind the grant which came to his institute. He was less certain of the specifics of how the money was spent and how it directly related to his pay. He is on “soft (grant) money,” so he observed somewhat cautiously that at least a few months of his salary came from these funds. He mentioned that the project started in July and he believes the money was spent right away and has been spent. He was a little uncertain as to whether he was actually paid from these funds. It seems likely he gave a correct answer, but lacks certainty in his own mind about from which funds in his institute he was paid.

R4 Graduate Research Assistant, Evidence of ARRA Funding, Correct Response.

This respondent has worked in a research assistant position since the Fall of 2009. He was not sure whether the money for his research assistantship came from ARRA, and indicated he needed a “not sure” option to mark on the questionnaire. He was certain that he is funded by NSF, because his advisor only has NSF grants, but simply wasn’t sure if the money was from ARRA. He commented that to answer this question he would need to know where NSF’s money came from, and he didn’t think a student would have knowledge to answer this question. He was not aware of any other students who were funded with ARRA money. He said that if he were answering this questionnaire at home, he would probably just guess and say “yes” because he knew the grant was awarded around the time that the stimulus money was dispersed. This is the answer he chose, but apparently he had no basis for choosing it.

R5 Tenure-track Faculty, Evidence of ARRA Funding, Incorrect Response.

This respondent indicated that she received ARRA funding, and it was being spent during the Fall on a graduate student RA and laboratory materials. However, she answered “no” because she was not receiving a salary from these monies. She explained that she would likely use the ARRA funds for her summer salary in 2010, because she has a nine-month appointment. Thus, the word “work” was being interpreted in a more restrictive way than was done by other respondents. She was aware her grant was supported by ARRA, but did not respond ‘yes’ because she did not think her personal salary was being paid with this money. However, our records indicated that she was paid with ARRA funds, so she not only had trouble interpreting the question, but she also was not aware that the money had been used to some extent to fund her salary. She mentioned that her RA only knew the funding for the project came from the NSF, but probably would not know that the funding was “special ARRA money.”

R6 Graduate Research Assistant, Evidence of ARRA Funding, Incorrect Response.

This respondent came to WSU in the Fall of 2009, when he was on a research assistantship. He is now on a teaching assistantship. He knew that his Fall 2009 work was supported by government funding, indicating that his professor had a grant from a federal agency. He said “No” without hesitation to the ARRA question in part because he thought the priority of the ARRA was to go to American citizens, and being an international student, he did not think he could get any of the funding. He said that his advisor had funding before ARRA was passed into law. After more discussion about ARRA he said with some confidence, “No, I don’t think so.”

R7 Tenure-track Faculty, Evidence of ARRA Funding, Correct Response.

This respondent replied yes, but qualified his answer by saying “a little bit.” He said he had received $25K, but actually has not yet spent any of that money, which he considered to be the definition of “supporting his work.” After some discussion he said that he had ordered some materials and done some preliminary preparation that would be needed for the work to be done under the ARRA grant. Thus, there was some ambiguity here with regard to how he should interpret the question and answer it. Basically, the “yes” meant he had been awarded the money, but none had actually been spent. And the time he had spent on it was in anticipation of doing the expected work. It was clear from his answers that he is supported by multiple grants so it took some effort to recall what had happened on this grant in particular. Although he was certain that he received an ARRA-supported grant, he was not aware that any of his salary may have been paid with these funds.

R8 Graduate Research Assistant, Evidence of ARRA Funding, Incorrect Response.

This respondent answered “no,” and clearly did not believe any of his funding came from the ARRA funds. He explained his funding was through his advisor and is from NIH. He said the advisor got the money 1- 1½ years ago. He had heard of the stimulus bill, but was pretty certain that there was not any chance his advisor was supporting him through ARRA funds. He said that students in his department are generally informed of the source of their funding.

R9 Graduate Research Assistant, Evidence of ARRA Funding, Incorrect Response.

This respondent answered “no” and said his support is through a WSU project that is state funded. He said his advisor has several grants, but that his advisor had not specified which grant he was funded by, and he had not asked. His advisor has four other graduate students and he didn’t know whether the others were funded by ARRA funds. He also said he wasn’t sure what the ARRA is, but explained it as U.S. government money going out to different organizations to help create jobs.

R10 Graduate Research Assistant, Evidence of ARRA Funding, Incorrect Response.

This respondent was certain that his funding comes from an NSF grant, but was not sure if ARRA was involved. He said he was “pretty sure” that ARRA was not involved, yet would still mark “unsure” in the answer category if that option was available. He indicated that his advisor tells him where his money comes from. His advisor said she had a grant (awarded in spring 2009) and he was funded under it in the Fall of 2009.

R11 Administrator, No Evidence of ARRA Funding, Correct Response.

The respondent answered correctly, and with confidence, “Unfortunately, no, all of my applications got turned down.” When asked why he thought some people might answer ‘no’ when they actually had received ARRA funds, the respondent replied that the word ‘work’ is ambiguous, but that he interpreted it to mean “his research” and not “his salary,” implying that some might interpret it differently. When asked about whether some respondents might answer “yes” when they did not directly receive ARRA funds because the institution has received ARRA funds, the respondent replied, “I had not thought about it that way, but I would answer ‘yes’ if the question asked about the university instead of my work specifically.” Finally, when asked about whether grad students would know if they were being funded by ARRA, the respondent answered that his grad students probably would not know because he gets most of his funds from NIH and most of his students would assume their funding came from this source.

R12 Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

We did not expect this respondent to have ARRA funding because he was not listed as having received salary funds from an ARRA grant. However, he responded “yes” and informed us of a grant funded by ARRA that he received specifically for equipment. He seemed very knowledgeable about the grant and therefore we believe his answer to be correct. Even though he did not receive a salary from the grant, his research was supported through ARRA funds. He believes that the post-docs working with the equipment most certainly would know that the money came from ARRA, although he was a bit more doubtful if his graduate students were aware of this fact, or if they would know much about the specifics of which grants pay their salaries. He pointed out the difficulty of keeping track of such matters, since there are often multiple grants and the sources of students’ salaries could change from month to month. This suggests that while faculty who are involved in the grant writing process are familiar with the funding sources, people who are employed though the grants may have less understanding of where their support comes from.

R13 Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

This respondent answered “no” to the ARRA question, which is accurate. This respondent said he gets his grants from the same source every year and has done so for most of his academic career. He seemed certain of the accuracy of this response. In response to a follow-up question asking about his research assistants, the respondent noted that his graduate level assistants would know the source of his funding. Elaborating, he said his graduate students would know, partly because they help write up final reports and are otherwise closely connected to the research. He did say his undergraduate and high-school mentees might not know the source of his funding.

R14 Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

The respondent answered no, which we believe is correct. She said “I don’t think so…I don’t really know how it’s affected the university, but I didn’t get any grants through it.” The respondent initially seemed confident in her response, as she interpreted the question to apply to her personal grant funding. However, as she thought more about it she became unsure. She began considering the general stimulus funds that were given to the university, and whether those should be considered. She admitted she had little knowledge of how those funds were dispersed, so she felt unsure about how they might have affected individuals such as herself. She explained that if she were an RA, she doubts she would know how to answer this question, because graduate students probably don’t know how the university has used these monies. She indicated that a “don’t know” option was not available, and felt that would be a helpful addition. It seems like she probably would have marked “don’t know” if it was available. When asked whether she thinks that someone without grant money might answer yes, she indicated she thought this was possible, depending on how the university distributed its stimulus money and how much a person knows about how the university distributed the money. Because she was unsure of how the university used its money, she really was unsure of whether somebody might provide a “false positive.”

R15 Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

The respondent did not receive ARRA funds. He answered, “No, because I assume [the question is referring to] direct support but WSU did get ARRA money that saved jobs, so it could have supported me indirectly.” He further elaborated, “I answered the question as a direct benefit. If the question said ‘direct or indirect support’ I would have said ‘yes’ because WSU got ARRA money and it may have benefited me.” When asked if others at the university might answer the question ‘yes,’ assuming indirect support, the respondent replied, “Some might, particularly staff or other professionals [at the university] that are more worried about budget cuts and their jobs.”

R16 Postdoctoral Researcher, No Evidence of ARRA Funding, Correct Response.

We believe this respondent answered this question correctly. His response was “no,” however, he did not know with much certainty. After reading the question, his first response was “I have no idea.” He paused before saying that he is “guessing no, actually” because the grant he is funded with is a 4-year grant that was granted before 2009. He explained that the principal investigators of the projects deal with where the grants come from, and the postdocs just work under it (implying postdocs have less knowledge about the funding sources and grant writing process). He became a bit more certain of his response the longer he thought about it, because he reasoned that any funding from ARRA would have been distributed more recently, and the PI on his project got the money before 2009. When asked about his familiarity with ARRA, the respondent confused it with the Toxic Asset Relief Program (TARP). He said he knew about it, just from what he’s “gathered from the news…that money was thrown at the banks…and some went into science.” When asked about people responding yes based on knowledge that the university got funding from ARRA, he said he would not have considered this because he thinks only of the grants he is funded from.

R17 Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

The respondent did not receive ARRA funds. After reading the ARRA question, the respondent hesitated for some time before being prompted, “What are you thinking?” The respondent replied, “I’m thinking if WSU got [ARRA] money for the budget, and I think they did, but I’m not sure if any went to me [directly]. The way it [the question] is written, with ‘my work’, I would say ‘No’ but the university may have gotten some money. I would look at the follow-up question [respondent looked at Question 15] and I would say ‘No’ because I cannot answer question 15. It is hard for me to say if and how it supported my work.” When asked if she was sure her grant money has not come from ARRA, the respondent replied, “I don’t know, all my grants come from the USDA and have been going on for years. I did not think about my salary but more about WSU budgets and funds and my grants.” The respondent did not answer Question 15. Thus, although this respondent answered correctly, she seemed very uncertain as to how to interpret the question and what factors to consider when formulating her response.

R18 Administrator / Tenure-track Faculty, No Evidence of ARRA Funding, Correct Response.

The respondent answered the question correctly. He replied “no,” saying he never did get any of this funding. He was certain of this response. He explained that he didn’t apply for any because his work is not eligible for it because he conducts his research is overseas. When asked about the potential for people to report yes even if they had not received stimulus money (based on the fact that the university got state funding through ARRA, which benefitted the university in a general sense), he said he was not aware of whether people would be inclined to answer this way, but could understand how he might have answered differently if he was speaking from another position. Since part of his appointment is administrative, he may be more inclined to have thought about this possibility if he were answering from that perspective (but he was filling out the questionnaire about the other job title he holds, so he did not think about it in this way until he was probed to do so).

R19 Non-tenure track Research Faculty / Director, No Evidence of ARRA Funding, Correct Response.

The respondent answered “yes.” Based on her explanation, we believe her answer to be correct in the sense that she is working on getting an ARRA-funded project underway, although she has not yet received salary from these funds. The respondent works at a research center on campus, and the project in question would be through a contract with an outside organization. In an unequivocal voice she said, “Clearly some of our work is funded with stimulus money,” though exactly how the money was used and when is more complicated. She noted that oftentimes researchers aren’t exactly sure about their source of funding since her research center is funded by a variety of sources, both public and private. At the time of the interview the respondent was preparing to work on a project for an organization that has been awarded stimulus funds for this specific project. However, the contract for her to work on the project had not officially been signed, so no money had yet been passed on to her or her center. However, she believed that once the contract was signed that ARRA funds would be use retroactively to compensate her. The story became even more complicated when she informed us that the center had been working with this organization on this type of research in the past, since before stimulus funds were available, but she was certain that that the current project would be funded with ARRA money. Already, at the time of the interview, the respondent had received some ARRA funds to cover travel expenses to visit the organization. The question was somewhat difficult because of the nature of indirect funding. She knew ARRA funding was used only because she was aware that the organization was funding their contract with the research center using this money.

R20 Postdoctoral Researcher, No Evidence of ARRA Funding, Correct Response.

The respondent answered “no,” indicating that the funding source for the project he was working on did not come from ARRA. And, though the respondent was unsure of exactly where funding for the project came from (as in many other cases) he did name a specific organization through which he thought the funding for the project he was working on came from. However, he thought that this organization in turn was funded with ARRA funds. Probably most notably though is that the respondent said “no” when asked if he knew what ARRA funding was. On the other hand he did say that a friend he knew was funded with ARRA money through the USDA.

Interpretation

To the best of our knowledge, 15 out of the 20 respondents answered the question about ARRA support for their work correctly. All of those who responded incorrectly were individuals known to have ARRA support, yet they answered “no.” Notably, most of these individuals were graduate student research assistants. The one faculty member who answered incorrectly was aware of her ARRA-supported grant, but was confused by the wording of the question. The fact that most of those who responded incorrectly were graduate students who were unaware that ARRA played a role in their support suggests that unless individuals are directly involved in procuring the funding for their research, there is a good chance that they may be unaware of the role ARRA has played. These findings suggest that, as it stands, this question could result in a significant underestimation of the number of university employees who are supported by ARRA funds.

It does not appear as if we received any “false positive” reports of ARRA support. However, it should be noted that we stopped interviewing graduate students as we entered into the portion of the interviews that were with individuals thought not to have ARRA funding. Since students were most likely to be uncertain about funding sources, it is possible that we could have seen different results if non-ARRA funded graduate students were interviewed. Perhaps some of these students would have provided a “false positive” response. Also, several respondents who were not ARRA-supported mentioned that the university had received stimulus money, and wondered whether they should consider these funds as supporting their work. Thus, while these few individuals ultimately decided that the university’s general stimulus funds did not apply for this question, it is possible that others who are surveyed may do so.

While a majority of those interviewed provided correct responses, what is perhaps more important in assessing the validity of this question is that a majority of respondents expressed some degree of uncertainty about the question’s meaning or the accuracy of their responses. Eleven of the 20 respondents verbally expressed some level of uncertainty. In fact, some respondents seemed so uncertain that their responses were merely guesses. While most of these individuals guessed correctly, they seemed to have no idea of whether their work had been supported by ARRA. This leads us to believe that in a larger sample, there would be a greater likelihood that some individuals’ guesses would be incorrect. The following instances illustrate this uncertainty.

Respondent 4, a graduate research assistant, said he did not know whether his work as supported by ARRA. He was hesitant to say yes or no, instead indicating that he needed a “not sure” option.

Another graduate student, respondent 10, also said that he would have chosen “unsure” if that answer category had been an option.

A faculty member (R14) remarked that a “don’t know” option isn’t available, and thought it would be helpful to have.

A postdoctoral researcher’s (R16) first response to the question was “I have no idea.”

We believe this uncertainty stems from numerous potential sources, which can be broadly grouped into two categories. The first category of uncertainty involves confusion about the meaning of the question. Multiple respondents were unclear about what specifically the question was asking of them. Several commented that the wording of the question, particularly the phrase “your work,” was vague and/or confusing:

Respondent 1 was confused as to whether “your work” referred to the “work that [she] does” or her “salary,” which, according to her, would elicit two different responses.

Respondent 15 commented on how he interpreted the question to be applying to direct support for his work, but he was uncertain about whether the question also was interested in eliciting information about indirect support. He said, “I answered the question as a direct benefit. If the question said ‘direct or indirect support’ I would have said ‘yes’ because WSU got ARRA money and it may have benefitted me.” This implies that the respondent very well could have answered the question differently, and since the question did not specify, he was forced to simply make a decision about how to interpret it. Another respondent, R17, also was concerned about whether ARRA funds directed at the university at large would count as supporting “her work.”

Respondent 7 hesitated about whether to answer yes or no because, although he had received a grant, he had not yet spent it. He believed that for it to count as “supporting his work” that he would have needed to have spent the money already.

Respondent 11 commented that the word “work” is “ambiguous,” and could mean research, or salary.

Respondent 5 knew that she had received a grant through ARRA, but answered “no” because she interpreted “your work” to mean her salary. Thus, she provided an incorrect response because she was forced to interpret the meaning of the question, and she did so in a restricted way.

The other type of uncertainty stems from individuals’ particular circumstances and a lack of knowledge of the details of their employment and funding. For example, some individuals, particularly students and those who are not principal investigators of grant projects, do not necessarily know the details of the grants. Several graduate students remarked that they knew that their advisors had funding through a particular agency or organization, but they did not know the specifics of the grants and where the money stemmed from. For example,

One graduate student, #4, remarked that he knew his advisor received NSF grants, but he felt that to be able to answer the question about ARRA involvement that he “would need to know where NSF’s money came from, and [he didn’t] think a student would have the knowledge to answer this question.”

Respondent 10 echoed this, saying he was certain NSF provided the grant, but was “not sure if ARRA is involved.”

Several graduate students were rather certain that there was no possibility that their advisors’ grants were related to ARRA, even though they were. It was clear in these cases that advisors never discussed ARRA funding with their students, so they were unprepared to provide an accurate response.

Indeed, advisors often mentioned that they did not believe that their students would know whether ARRA was involved, although others remarked that their students would know this information.

Respondent 5 said her research assistant only knew the funding came from NSF, but probably would not know that the funding was “special ARRA money.”

Respondent 1 believed her graduate students would know that their stipends were the result of special funding, but was doubtful that they were aware that this funding was the result of ARRA.

Respondent 12 doubted whether graduate students would know ARRA was involved, or if they would know much about what grants were involved at all.

Graduate students’ knowledge of the sources of their funding is likely to be particularly limited when their advisors hold multiple grants.

Respondent 2 began to doubt his answer when he realized that his advisor has “multiple NSF grants” and he has been funded under these for four years. He realized he was not certain which grant paid his salary.

Respondent 9 knew his advisor had several grants, and admitted he had not asked his advisor which grant pays his salary.

Graduate students were not the only respondents who expressed uncertainty about the details of grant money and their salaries. In fact, several faculty members who had received grants funded through ARRA were not aware that they had received salary from these funds:

Respondent 5 answered incorrectly because she did not believe she was receiving salary from her ARRA-funded grant.

Respondents 1 and 7 also believed they were not being paid salary from their ARRA funded grants.

Another source of confusion for both students and faculty was the fact that many researchers have multiple grants, and it is very difficult to keep track of how they are all being spent and which grants are being used at which times.

Respondent 12 explained the complexities of keeping track of the funding and how multiple grants are being utilized. He doubted that many graduate students in this type of situation would know which specific grant was funding their work, since the source of their salaries is likely to change from month-to-month. He himself admitted a lack of knowledge of the specifics, as he holds a lot of grants and has accounting employees manage them.

Respondent 3 also expressed uncertainty about the specific source of his funding. He was pretty sure ARRA money had come into his institute, but seemed to be merely assuming that these monies directly contributed to his personal salary. He admitted that he is not involved in the grant writing process and so he does not know the specifics of the funding allocation.

Respondent 7 knew he had received ARRA money, but because he has multiple grants, had a bit of trouble recalling the timing of this grant in relation to others, and what specifically had been done with the money from this grant in particular.

Thus, another observation from these interviews is that among those with multiple grants, work that is funded by one particular grant is typically related to work done for another grant. This makes it especially difficult for individuals to know which specific grant is being charged for a particular time period. During any given week, the researchers may be charging work to several grants. However, whether or not respondents know specifically when ARRA grants were used will likely be less of a problem if and when this question is actually asked in the Survey of Doctorate Recipients. Due to the timing of the cognitive interviews, we asked respondents to only think of how ARRA might have affected their work in 2009, which for some individuals only included a few months of having potential access to ARRA grant funds. Thus, they were faced with recalling exactly when they started spending the funds. The question, if included in the SDR, will include both 2009 and 2010. This, we think, may eliminate some confusion about whether particular funds have been utilized yet or not, which was a problem affecting several respondents’ ability to confidently answer the question.

Conclusions and Implications

The results just summarized suggest that there are multiple problems that can contribute to incorrect responses to questions regarding ARRA support for one’s work. Based on these results we conclude that this question will not reliably produce accurate results in a general population sample or in a sample of college graduates. Graduate research assistants had the most trouble when answering this question. If an individual is not directly involved in grant writing, it is unlikely he/she will have enough knowledge to answer this type of question. Once we limited our focus only to individuals with Ph.D.s, we saw a reduction in the level of uncertainty about funding sources.

Thus, if the question is only to be asked of doctorate recipients, then a good deal of the inaccuracy we observed may be eliminated. Nearly all of the faculty, post-doctoral researchers, and administrators we interviewed provided correct answers to the question. In this context the question has more potential, but also still suffers from some substantial limitations. Although most doctorate-holding respondents could answer the question correctly, many of these individuals were unsure of the accuracy of their responses. Changes to the question could alleviate some of the sources of confusion and uncertainty.

Our first recommendation is to clearly and explicitly define what is meant by “your work.” Numerous respondents mentioned that this wording was vague, unclear, or confusing. The question should spell out what respondents should consider to be “their work.” The question should direct respondents how to account for salary, grants, and/or indirect effects as a result of funding provided to their employers.

It might also be helpful to include an open-ended follow-up question such as the one used in the cognitive interview questionnaire to help determine what aspects of work respondents are considering when answering the question.

There are two primary ways to reduce the ambiguity that exists in “your work” in the question. First, the words could be changed to indicate something more specific and less ambiguous. For example, we tried “your employment” in the WES, but also encountered problems at least partially related to ambiguity. Other suggestions include, from most to least ambiguous: “your research” or “your job,” to “any of the work you perform for/on your [principal] job” or “any of the pay you received for your [principal] work/job”, or some variations thereof. However, even these latter, more specific phrases can be somewhat ambiguous and/or confusing without addressing the second way to reduce ambiguity: placement of the question in the questionnaire.

In both surveys we conducted with the ARRA question – the WES and the cognitive interviews – the question was located somewhat removed from other questions concerning the respondent’s principal job. In the WES, we transitioned from asking about the respondent’s debt situation compared to a year ago and expectations about the future to asking about whether their employment was supported by ARRA. There is quite a disjuncture here. In the cognitive interviews, we transitioned from job satisfaction, to hours worked/week, to benefits available, to work supported by grants in 2008, and finally to whether their work was supported by ARRA during 2009. Again, there is somewhat of a disjuncture. A way to reduce ambiguity, almost regardless of the words used in the question (e.g. ‘your work,’ ‘your employment,’ etc.), is to place the question directly or closely following other questions regarding the respondents employment situation and principal job. So, for example, in the WES the ARRA question might have seemed less ambiguous if it followed the employment Qs 5-7. In the cognitive interview questionnaire, it might have been less ambiguous it followed the question about the respondent’s principal job (Q5 or Q7). Moreover, we think this would be particularly feasible if only one or two questions were asked regarding ARRA support. They would distract the respondent’s attention from other questions concerning their work or employment. If a whole section of ARRA questions is desired, then this strategy may be more difficult to pursue, as was the case with the WES, which may call for very specific wording in the question, as indicated above.

Changes such as these can improve the likelihood that the question will elicit accurate responses, or at least allow researchers to better gauge the accuracy of responses. However, it must also be noted that even with a clearly written question, numerous respondents are likely to be unaware of the details of the funding for their work, and thus may not have the knowledge needed to answer this question. One partial solution to this problem is to include a “not sure” or “don’t know” option for respondents to select if they are not certain of ARRA funding. Several of our respondents wanted this option, and provided an answer based on speculation because they were not given this. Because we did not survey a random sample of college graduates, we cannot say with much authority how prevalent this lack of knowledge may be. It might be the case that a good deal of respondents would select “not sure” if this option was provided, in which case the usefulness of the question altogether might be reduced.

Potential Problems with Other Survey Questions

The questionnaire we used in the cognitive interviews (See Appendix A) included thirteen additional questions from NSF’s National Survey of College Graduates. As we conducted the cognitive interviews it became clear that respondents had difficulties answering several of these questions. In this section we identify which questions posed problems and briefly outline some of the issues we encountered in hopes that this information might be helpful for future surveys.

Q1. Were you working for pay or profit during the week of October 1, 2009?

Working includes being a student on paid work-study, self employed, or on any type of paid or unpaid leave, including vacation.

Use an X to mark your answer.

Yes

No

The wording of Question 1 caused some confusion for a few respondents. Specifically, the word “profit” was unclear. Some respondents were confused about whether or not they worked for “profit”. We think the confusion stems from respondents being employed at a university, in which “profit” is not part of most university employees’ pay. Also, it seemed that respondents interpreted the subject as “both pay and profit” instead of “either pay or profit.” However, each respondent ultimately answered the question accurately. One respondent, R17, resolved her confusion to the question by focusing on the “pay” element, which did apply to her.

Several respondents also ignored or did not register the instructions that asked them to mark their responses with an “X.” Some respondents filled in the bubbles, comparing it to a multiple choice exam, and others used check marks. A few respondents realized at some point later in the questionnaire that they were not sure how they were supposed to mark their answers, which suggests to us that the instructions were never processed.

Another issue encountered in Question 1 was the use of a specific date of reference (October 1, 2009). In fact, this was a recurring issue throughout the questionnaire. Some respondents indicated that they had a difficult time recalling October 1, 2009. These respondents seemed troubled when they could not recall the specific events of that week, which made answering questions more difficult. For example, one respondent reflected on the date for a moment before saying, “well, even if I were on vacation I suppose I would be working anyway!” Though this respondent answered the question correctly, his statement suggests that he was not only troubled by the need to recall the specific events of the week of October 1, but that he also may have misinterpreted the instructions pertaining to vacation time (he seemed to be thinking he was supposed to mark ‘no’ if he had been on vacation). Asking respondents to recall a particular week led at least two respondents to first try to remember that week, but then decide to think more generally rather than specifically. R18 told us that he would think of October 1, 2009 as a typical week, where the 1^st would have been a Monday. Likewise, R15 preferred to think about the time period generically as “the fall” 2009 since pinpointing the exact date was difficult and seemed pointless. There were indications that these respondents were not completely confident that this generalized approach was appropriate. One respondent, R4, explicitly said it was strange that the questionnaire was concerned about one particular week, because that week may or may not have been representative of one’s typical experiences. He explained that he thought it would make more sense if the questionnaire asked for general experiences, rather than the specifics of one week.

Q2. Who was your principal employer during the week of October 1, 2009?

If you had more than one job, report the one for which you worked the most hours that week.

If your employer had more than one location, report the location that employed you.

If you worked for a contracting or consulting company, report the name of that company, not the client organization.

Employer Name

Department/Division

City/Town

State

ZIP Code

Overall, this question appeared rather burdensome for many respondents. We sensed some individuals were slightly overwhelmed by the amount of instructions they needed to read and process. This may have caused some errors. For example, R19 entered her Department/ Division in the “Employer Name” space, and the name of her employer in the “Department/ Division” space. When asked about this she said she entered the information the same way she would have written her work address. As she put it,

I didn’t think about the university hierarchy. I guess I just put it down this way because my ‘default’ is to think about my employer in the same order I would write their mailing address.

Perhaps the burdensome nature of the question is also one reason why numerous respondents abbreviated their employer name and/or their Department/Division without much thought. It seemed that it was not until we pointed it out that many realized they had used abbreviations. When asked about it, R3 said he abbreviated because “he didn’t feel like writing it all out.” When asked if they would abbreviate at home most said they would not, but based on the interview data, it is difficult to know if this is the case. In fact, R12 specifically said he would not completely spell out his department even if alone; he wrote the full name of his college, but not the department because he “[didn’t] like the name because it is too long.” He said he probably would not spell it out because it is too burdensome. To help avoid this, R16 suggested that we include an instruction asking the respondent to avoid abbreviations.

Q4. During the week of October 1, 2009, what type of academic position(s) did you hold at this institution?

Mark Yes or No for each item.

Yes No

1 President, Provost, or Chancellor (any level)

2 Dean (any level), department head or chair

3 Research faculty, scientist, associate or fellow

4 Teaching faculty

5 Adjunct faculty

6 Postdoc (e.g., postdoctoral fellow or associate)

7 Research assistant

8 Teaching assistant

9 Other position – Specify

Question 4 was particularly confusing for a number of reasons. One problem we encountered multiple times relates to the distinction between “teaching faculty” and “research faculty.” On several occasions tenure-track faculty were unsure about whether or not to mark “research faculty” (item 3) or “teaching faculty” (item 4). Based on their responses to probing questions it became clear that tenure-track faculty – even though a majority of their appointment requires them to conduct research – were reluctant to enter “research faculty” because this status is usually associated with non-tenure track positions. “Teaching faculty,” on the other hand, is associated with tenure-track positions, even if teaching is a relatively small proportion of their overall appointment. For example, upon being probed about his answer to this question, R11 said:

The question is somewhat ambiguous because it [research faculty] implies a non-tenure track position, but because I am not teaching classes I answered “no” to “teaching faculty” (even though this respondent did have tenure at the time of the interview).

Another problem with this question relates to the status of post-doctoral researchers at WSU. R16 ultimately answered Question 4 correctly, that he was a post-doctoral researcher, but this conclusion was difficult for him to reach. He explained that at WSU post-doctoral researchers are not technically labeled as such, but rather have the job title “research associate.” This led him to write “research associate” on Question 5. His different responses apparently caused him some cognitive dissonance, though he ultimately justified his responses by noting that his “title” (as asked in Question 5) is in fact “research associate,” while he is also a post-doctoral researcher (as he indicated in Question 4).

One final concern with this question is the instructions, which ask the respondent to “Mark Yes or No for each item.” Several interviewees only marked “yes” for the item they felt best applied to them, but did not mark “no” for those that did not. It is possible that this is because the format we used was somewhat different from the actual NSF survey; our survey had comparatively more space between the “yes” and “no” columns, a distance that might have placed the “no” column outside of the respondent’s foveal view.

Q5. What was the title of the principal job you held during the week of October 1, 2009?

Example: Financial analyst

In itself this question did not pose many difficulties for respondents, but we think it caused some problems because we believe it became an anchor for Question 7 (see below).

Q6. Did your duties on this job require the technical expertise of a bachelor’s degree or higher in…

Mark Yes or No for each item.

Yes No

1 Engineering, computer science, math, or the

natural sciences

2 The social sciences

3 Some other field (e.g., health, business, or

education) – Specify

Several respondents answered this question incorrectly because they did not limit themselves to just thinking about the actual degrees that are required for their job, but rather degrees and/or skills that are useful for it. For example, R18, a research faculty member with a partial administrative appointment, marked yes for option 1, which is reflective of the field of his doctorate, and his research. However, he also marked yes to option 3 and wrote in “administration.” When probed about this, he said that administrative “skills” were needed for his job. It was confirmed that he did not hold any degree in administration, but was answering the question simply in reference to the skills he felt were necessary. Similarly, respondent 12 debated marking option 2 in addition to option 1 because he has a bachelor’s degree in the social sciences. He reasoned that this degree was not required, but it was useful for what he does. He ultimately marked no for this option. Interestingly, though, he then went on to mark yes for option 3, because he felt that a degree in medicine is required for his job, but he does not have a medical degree.

Q7. During what month and year did you start this job (that is, the principal job you held during the week of October 1, 2009)?

Month Year

PRINCIPAL JOB STARTED

As mentioned above, we believe Question 5 served as an anchor that affected how people responded to Question 7. We came to this conclusion because many respondents asked the interviewer if Question 7 referred to the response they gave in Question 5 (their current title) or the length of time they’ve been employed at the institution. R13, for example, wrote that he had been working at his job since 1985, even though he had been promoted to the rank of full professor (his current job title) sometime after this date. On the other hand, some respondents answered this question based on their current job title, even though they had been doing the same work at the institution since before obtaining their current title. R15 noted that he had been working in his current position (professor and chair) since 2004, even though he had been at WSU for much longer.

Q8. To what extent was your work on your principal job related to your highest degree? Was it…

Mark one answer.

Closely related

Somewhat related

Not related

Some PhD students answered this question incorrectly. At least one research assistant said that his assistantship (his ‘work’) was somewhat related to his dissertation work, which is a degree which he has not yet completed. Since this respondent had not yet completed his doctoral degree he should have answered based on how related his master’s degree is to his assistantship rather than how related his dissertation research is to his research assistantship.

Q9. Thinking about your principal job held during the week of October 1, 2009, please rate your satisfaction with that job’s…

Mark one answer for each item.

Very Somewhat Somewhat Very

Satisfied Satisfied Dissatisfied Dissatisfied

1 Salary

2 Benefits

3 Job security

4 Job location

5 Opportunities for

advancement

6 Intellectual challenge

7 Level of responsibility

8 Degree of independence

Question 9, item four asked the respondent to consider their satisfaction level with their “job location.” The responses to this question varied from very satisfied to very dissatisfied, though we have reason to believe that this variation in responses is the result of measurement error rather than actual levels of satisfaction. After being probed about what each respondent considered to be their “job location” we found that the interviewees conceptualized their job location in a variety of ways. Some respondents reflected on their office, campus, or the building(s) they work in, others thought about the location of their office in relation to their home or the rest of the campus, while others thought about Pullman and/or the surrounding area and geographic region. R12, for example, said that he was very dissatisfied with his job location because his work was spread across a number of buildings on campus. R20 also reported being very dissatisfied with his principle job location because he was required to travel to a branch campus at least once a week. Another respondent was dissatisfied that the location of his office was located off-campus. Numerous respondents thought about their level of satisfaction or dissatisfaction with Pullman, the region, and the location of Pullman to larger cities and airports. Some respondents, after being asked about another aspect of job location than they had considered, admitted that if they were to consider other aspects of job location they would have responded more negatively or positively than they had. For example R16 (who answered somewhat satisfied) noted:

I was thinking of the university, and the towns/surrounding area. I like the university, but I’m less enthusiastic about the isolated nature of the towns. I did not consider my building, but if I had I might have changed my response to something more negative.

Another issue with this question is that some respondents felt that certain items did not apply to them, so they had difficulty responding. Some indicated that they wished there was a “not applicable” option.

Q11. During a typical week on your principal job, how many hours did you work?

NUMBER OF HOURS WORKED PER WEEK

When answering this question many respondents wondered if it referred to hours worked or hours paid. This issue was particularly salient for research assistants with 20 hour/week appointments. Oftentimes these individuals considered themselves to have “worked” for 40 or more hours a week, but they are only paid for 20.

This illustrated how “work” was defined differently for different respondents, which resulted in a wide range of methods for answering this question. Some respondents only included official time spent at the office or lab, while others included time spent doing other things that might be tangential or supportive of their primary “work.” For example, R12 included time spent walking, because he does a lot of his thinking about research while walking. Another respondent, R11, included time spent at university social functions, because although these are not related to his actual work, he is expected to be at them.

Q12. Concerning your principal job during the week of October 1, 2009, were any of the following benefits available to you, even if you chose not to take them?

Mark Yes or No for each item.

Yes No

1 Health insurance that was at least partially paid

by your employer?

2 A pension plan or a retirement plan to which your

employer contributed?

3 A profit-sharing plan?

4 Paid vacation, sick or personal days?

Question 12, item 3 asks if “a profit-sharing plan” is available to the respondent. Though ultimately none of the interviewees answered the question incorrectly, many respondents hesitated before answering. In only one instance did a respondent think of a way that a University researcher might be part of a profit sharing plan with a University: patents. There were no interviewees that responded “yes” to this question, but several seemed confused about what this would apply to, which resulted in uncertainty about their responses.

Q13. Thinking back now to 2008, was any of your work during 2008 supported by contracts or grants from the U.S. government?

FEDERAL EMPLOYEES: Please answer “No.”

Mark one answer.

Did not work in 2008

Yes

No

Don’t know

A number of interviewees responded to this question incorrectly. R13, for example, confused funding that the university procures from the state with federal funding. As such, this respondent answered incorrectly – the funding he had in mind when responding to the question came from a university. This similar confusion was also the case with R10. R16 answered the question correctly, but wondered if his federal grants qualified him as a federal employee. Though this respondent decided this did not make him a federal employee, it is possible that others might make a different judgment and answer incorrectly.

CONCLUDING STATEMENT

While collecting the data analyzed here, particularly in early February, 2010, we became aware of numerous news reports attempting to assess the impact of the American Recovery and Reinvestment Act on the creation of jobs. It is noteworthy that estimates of the effects of ARRA have varied widely, and that most analysts report that it is often difficult for employees to know whether their employment has been affected by these stimulus funds. The analysis presented here tends to confirm these media reports, and provides some insight about the reasons why determining the employment effects of ARRA is so difficult.

Responses to both the statewide survey and the cognitive interviews of science and engineering university employees suggest that it is often unclear whether one’s work is supported by the stimulus funding. Some individuals are simply unaware or unsure of whether ARRA has played a role in their work or not. Some people’s work is supported by multiple sources, and this can make it difficult to recall one specific funding source. ARRA funds may have been received, but not yet spent. Also, such funds may be spent on equipment or activities that “support” one’s work, but is not personally received as salary or wages. Additionally, effects may be direct or indirect, which further complicates individuals’ understandings of how ARRA may play a role in their employment. Our interviews made it clear to us that respondents who seemed quite conscientious and knowledgeable about their work and sources of financial support found it difficult to answer these questions.

Our general conclusion is that there is clear evidence that some people, in both the general public and University samples, have received ARRA funding and realize that it is providing employment or other support of their work. Others are less clear on whether this is the case, and some simply do not know. Thus, it would seem to us that asking questions of this nature in NSF surveys can be used as a general indicator of respondents’ work being supported by ARRA funding, but it will not be a precise indicator. Like many other indicators, e.g. using income as an indicator of economic status or people’s responses to a question about whether their health is “excellent, good, fair or poor” as an indicator of their health status, there is likely to be some error in people’s answers. This error may be greater in a non-university setting, where grant funding is not prevalent and employees may know less about how their work is financed. Nonetheless, improvements could be made to alleviate some of the problems we encountered with the questions. Thus, as one piece of information that might be used in order to understand who is impacted, and how, by ARRA stimulus funding, the questions evaluated here would seem to have some utility. It is important to recognize that even though the questions tested here are imperfect, they may still provide useful information when used in analyses along with other variables in order to understand the effects of ARRA funding on scientific and engineering work now occurring in the United States.

Finally, it seems important to reflect on the timing of any studies that include questions about ARRA funding. In our cognitive interviews we asked about funding for 2009. When asking respondents whether ARRA supported their work in surveys to be conducted in late 2010, the question would have a broader time horizon. Some of the difficulties our respondents had of knowing how they were paid during October would not exist with this longer time horizon.

However, it’s also important to recognize that the more time that elapses between the ARRA and when people fill out the survey, the greater the problems of recall. Not only will ARRA become a distant memory for some people, it seems possible that there will be some potential for respondents to confuse ARRA funds with those from other sources, and even a new stimulus bill, should one become law. Thus, we suggest that NSF proceed cautiously in deciding when and how questions are asked to assess the impact of ARRA funds on the employment of scientists and engineers. It is at best a difficult concept to measure with precision, and is likely to become even more difficult to measure as time goes by.

Appendix A: Questionnaire Used in Cognitive Interviews

Questions Being Considered for

2010 National Science Foundation

Surveys of College Graduates

INSTRUCTIONS

Please answer each of the questions and tell me what you are thinking about each of the questions as you provide your answers.

Q 1. Were you working for pay or profit during the week of October 1, 2009?

Working includes being a student on paid work-study, self employed, or on any type of paid or unpaid leave, including vacation.

Use an X to mark your answer.

Yes

No

Q2. Who was your principal employer during the week of October 1, 2009?

If you had more than one job, report the one for which you worked the most hours that week.

If your employer had more than one location, report the location that employed you.

If you worked for a contracting or consulting company, report the name of that company, not the client organization.

Employer Name

Department/Division

City/Town

State

ZIP Code

Q3. Was your principal employer an educational institution?

Yes

No

Q4. During the week of October 1, 2009, what type of academic position(s) did you hold at this institution?

Mark Yes or No for each item.

Yes No

1 President, Provost, or Chancellor (any level)

2 Dean (any level), department head or chair

3 Research faculty, scientist, associate or fellow

4 Teaching faculty

5 Adjunct faculty

6 Postdoc (e.g., postdoctoral fellow or associate)

7 Research assistant

8 Teaching assistant

9 Other position – Specify

Q 5. What was the title of the principal job you held during the week of October 1, 2009?

Example: Financial analyst

Q6. Did your duties on this job require the technical expertise of a bachelor’s degree or higher in…

Mark Yes or No for each item.

Yes No

1 Engineering, computer science, math, or the

natural sciences

2 The social sciences

3 Some other field (e.g., health, business, or

education) – Specify

Q7. During what month and year did you start this job (that is, the principal job you held during the week of October 1, 2009)?

Month Year

PRINCIPAL JOB STARTED

Q8. To what extent was your work on your principal job related to your highest degree? Was it…

Mark one answer.

Closely related

Somewhat related

Not related

Q9. Thinking about your principal job held during the week of October 1, 2009, please rate your satisfaction with that job’s…

Mark one answer for each item.

Very Somewhat Somewhat Very

Satisfied Satisfied Dissatisfied Dissatisfied

1 Salary

2 Benefits

3 Job security

4 Job location

5 Opportunities for

advancement

6 Intellectual challenge

7 Level of responsibility

8 Degree of independence

Q 10. How would you rate your overall job satisfaction with the principal job you held during the week of October 1, 2009?

Mark one answer.

Very satisfied

Somewhat satisfied

Somewhat dissatisfied

Very dissatisfied

Q11. During a typical week on your principal job, how many hours did you work?

NUMBER OF HOURS WORKED PER WEEK

Q12. Concerning your principal job during the week of October 1, 2009, were any of the following benefits available to you, even if you chose not to take them?

Mark Yes or No for each item.

Yes No

1 Health insurance that was at least partially paid

by your employer?

2 A pension plan or a retirement plan to which your

employer contributed?

3 A profit-sharing plan?

4 Paid vacation, sick or personal days?

Q13. Thinking back now to 2008, was any of your work during 2008 supported by contracts or grants from the U.S. government?

FEDERAL EMPLOYEES: Please answer “No.”

Mark one answer.

Did not work in 2008

Yes

No

Don’t know

Q14. During 2009, was any of your work supported by the Federal stimulus bill (American Recovery and Reinvestment Act of 2009)?

FEDERAL EMPLOYEES: Please answer “No.”

Mark one answer.

Did not work in 2009

No

Yes  Go to Q15

Q15. (If yes) please explain in your own words how the stimulus bill (ARRA) has supported your work. This question is very important, so please tell us all you can.

Appendix B: Other Materials Used in Cognitive Interviews

ARRA Interview Protocol

1. Welcome each participant.

2. Introduce team members in the room ad explain why each of us is there.

3. Explain task—to help the NSF with a questionnaire they will be send to a large sample of College graduates and people working in colleges and universities in 2010.

4. The NSF wants to know how people are interpreting questions in the questionnaire so they get insight into whether they are collecting the information that is intended.

5. I will ask you to fill out a questionnaire, but to do it by talking out loud and telling me what you are thinking about as you answer the questions.

6. But first I need to ask you to sign a consent form for the interview.

7. It’s a little unusual to talk out loud while completing a questionnaire, so we have a short practice question to help explain what we mean.

8. Ask respondent to complete the practice questionnaire.

9. Emphasize the key idea—by asking people to answer questions and tell us what they are thinking it helps us learn what might be wrong with the questions and how we can go about improving them.

Interview Consent for National Science Foundation Study

Each year, the National Science Foundation conducts a survey of national samples of people employed in scientific work. These surveys concern the nature of the work they do. Currently, NSF is conducting research to improve the current versions of these surveys, and I have been asked to help with that research.

I have volunteered to participate in this study regarding questions about my education and work. By completing a brief questionnaire and offering my opinions about various features of these forms, I will be assisting the research team and the National Science Foundation (NSF).

At no time during my volunteer assistance am I obligated to offer personal information if I do not wish to do so. If I prefer not to answer a particular question asked of me when completing a questionnaire or by the interviewer, I am free not to respond at my discretion.

This information is solicited under the authority of the National Science Foundation Act of 1950, as amended. All information collected will be confidential and used only for research purposes. Your identifying information will be removed from any results or reports of data. We want to assure you that anything you tell us or any information that you give us will not be attributed to you individually. The information will only be shared with the research team. We consider the information you choose to share with us personal and confidential. You have the right to refuse to answer any question, and to refuse to participate. However, we expect the outcome of this survey to benefit you in that the results will be used to improve the quality of the data that NSF produces.

This information is being collected by a research team from the Social and Economic Sciences Research Center at Washington State University. This study has been approved by U.S. Office of Management and Budget (OMB). The OMB control number for this collection is 3145-0174. The WSU Institutional Review Board has determined that the study satisfies the criteria for Exempt Research at 45 CFR 46.101(b)(2).

If you have any questions or comments about this research project, please contact the Washington State University Social and Economic Sciences Research Center (509.335.1511).

_____________________________ ______________________________

Participant’s Signature Researcher’s Signature

_____________________________ ______________________________

Printed Name Printed Name

_____________________________ ______________________________

Date Signed Date Signed

Practice questionnaire

Please read this question out loud and tell me what you are thinking as you come up with your answer.

Q-1. How many windows are in the apartment, mobile home or house where you live?

________ Number of windows

Probes that may be used, particularly when answering the ARRA related questions.

Probes for Specific Answers

Q15: Could you explain a little more about the reason you answered _______ (Y/N)?

If respondent answered “No”:

NOQ15: The last question is a particularly important question in the survey. Is it possible that part of your work may be supported in some way by the ARRA stimulus bill, but that you are not aware of that support?

Yes

No

If respondent answered “Yes” on Q15:

YESQ15a: Could you explain a little more about the reason you answered ________ (Y/N)?

YESQ15a: The last two questions are particularly important questions in this survey. Is it at all possible that your work is NOT supported in any way by the ARRA stimulus bill?

Yes

No

We will probe further as needed on these questions to see if respondents are interpreting the question appropriately, and particularly if they appear to have interpreted the question as asking for indirect effects.

“Tell me what you’re thinking as you answer this question.”

“Are you familiar with the ARRA or the Federal Stimulus Bill? What do you know about the ARRA or Federal stimulus bill?”

“You have indicated yes/no. Can you tell me why you chose this answer?”

“How certain are you about your answer? How do you know that your work has/has not been supported by ARRA?”

“What aspects of your work are you thinking about when coming up with your answer?”

“What do you think the question refers to when asking about ‘your work’?”

Are you aware of whether some other employees in the University are now being paid at least in part by ARRA (stimulus funds)?”

“To the best of your knowledge, are you paid by one source of funds, i.e. one grant or contract, or multiple sources of funds? “

“Has your supervisor, told you that some or all of your funding comes from ARRA? (And, is it some or all?)”

If yes to receiving ARRA funds “What proportion of your appointment comes from ARRA?”

“We are wondering if when we ask that question, “During 2009, was any of your work supported by the Federal Stimulus bill?” that a respondent might have said yes, because of money allocated to the University by the State of Washington to support the University budget, even though they were not employed specifically on ARRA funds through a specific grant or contract?   Do you think it’s likely a person would have said yes, even though they personally, are not being paid specifically from stimulus act (ARRA) funds?”

To the best of your knowledge, has the stimulus bill (ARRA) provided you with: a tax refund or benefit; s upport for home purchase or improvements; support for buying an automobile, including ‘Cash for Clunkers’; i mproved public services; health care services; c hild care services; support for education.

1 Morgan M. Millar, Benjamin L. Messer, and Shaun S. Genter are graduate research assistants in the Department of Sociology and the Social and Economic Sciences Research Center at Washington State University. Don A. Dillman is Regents’ Professor in the Social and Economic Sciences Research Center and Department of Sociology, Washington State University. Research reported here was conducted by the Washington State University Social and Economic Sciences Research Center (SESRC) with support from USDA-National Agricultural Statistics Service and the National Science Foundation Division of Science Resources Statistics under cooperative agreement no. 43-3AEU-5-80039.

14