Rationale for Salary Question

Attachment 1

Rationale for a combination open ended/closed ended question for recipients of research doctorates in the Survey of Earned Doctorates (SED 2008)

The Federal sponsors of the Survey of Earned Doctorates believe that it would be desirable to have an item on the survey that assesses expected salaries of new research doctorate recipients. As a part of the preparation for this item, NSF commissioned work to plan, execute, and evaluate a field test of SED respondents; this project -- the Salary Experiment – was designed to provide information to determine the optimal format for an expected salary item.

From a data quality perspective, the ideal format to collect salary data is an open-ended format that allows respondents to provide point-data. The open-ended format has the advantage of allowing for very precise measurement of salaries by groups of interest. Open-ended data also allow for accurate trend data over time as salaries increase. Further, due to the continuous nature of data from an open-ended salary item, the results can be subjected to statistical operations that allow data users to examine a number of relationships. However, open-ended salary items have been shown to be burdensome and potentially intrusive in some populations. The added cognitive burden and perceived intrusiveness of such items can lead to item non-response. Indeed, data from the Salary Experiment reveal that of the four item formats tested, the open-ended format had an item nonresponse rate that was more than twice as high as any of the other formats.

Closed-ended items have the advantages of decreased cognitive burden and perceived intrusiveness, but come with a cost of decreased precision and decreased ability to compute trend data, as the ranges on closed-ended salary items necessarily change over time. Data from the Salary Experiment suggest that some respondents may be more likely to provide salary data on closed-ended items than on open-ended items. Closed-ended item formats had an item nonresponse rate of 4 percent versus 10 percent for the open-ended format.

These two competing aims, the desire for high quality open-ended salary data and a high item response rate, led to the creation of another format, the combination format. In this format, respondents were presented with an open-ended item and were also given the option of responding to a follow-up closed-ended item. This approach does little to ease the cognitive burden on respondents, but we believe that it decreases the perceived intrusiveness of the item by an appreciable amount. Data from the Salary Experiment support this belief. The combination format had a similar item nonresponse rate (4.3 percent) to the closed-ended formats, but had the benefit of continuous, open-ended data for most respondents in this condition (80.4 percent). Fifteen percent of respondents in the combination format responded to the closed-ended portion of this item; less than 5 percent did not provide a response to the question.

The fact that the item nonresponse was no lower in the combination format than in either of the closed-ended formats leads the NSF/SRS to recommend this format for the salary item on the Survey of Earned Doctorates. The combination format does not compromise the quantity of salary data gathered, and the quality of salary data from this format is markedly better than salary data gathered from closed-ended formats.

However, this format is not without cost. Among respondents presented with the combination format, 80 percent chose to respond to the open-ended portion of the item. This contrasts with 90 percent who responded to the open-ended item when no closed-ended option was available. Therefore, one can conclude that the presence of the closed-ended format may have influenced some (about 10 percent, assuming the groups are comparable) of those who might otherwise have responded to the open-ended item to respond to the closed-ended version instead. That is -- to the degree that the respondents in the two experimental conditions were comparable -- one can conclude that the presence of the closed-ended option in the combination format decreased the response rate on the highest quality (e.g., open-ended) data. At the same time, the presence of the closed-ended options may have influenced some who would not have responded to an open-ended only format of the salary item to respond to the closed-ended format and provide some useful salary data.

Taken as a whole, we believe that the best balance is having actual salary data from the majority of respondents (80%), while also having salary data -- from those who may balk at providing exact salary data -- from ranges that the closed-ended formats provides.

Rationale for proposed salary ranges

Data from multiple sources (Finn SSA study and Salary Experiment data discussed in Attachment 2) suggest that there is a wide variance in the salaries of new doctorate recipients primarily as a function of field of doctorate.

New doctorate recipients in highly technical fields (e.g., computer science and electrical/computer engineering) may earn up to and over $100,000 annually in their first year out of graduate school, while new doctorate recipients in less traditionally technology-based fields (e.g., humanities) and fields where postdoctoral study appointments are common (e.g. biological sciences) often earn less than $40,000 annually. Because of this wide variation in expected salaries for newly minted doctorate recipients, any item that is designed to measure starting salaries must be very robust, with both a low floor and a high ceiling.

Drawing on data from the Salary Experiment, we recommend ranges for the closed-ended portion of this item that will be sensitive to both the low and high earning members of the new doctorate cohort. During the conduct of the study, two salary ranges were tested on actual doctorate recipients; these ranges will be referred to as low-range and high-range. The low-range values began at an expected salary of $30,000 or less and went up to a ceiling of more than $90,000. The high-range values began at an expected salary of $50,000 or less and went to a ceiling of $120,000 or more. (Table 1)

Table 1. Salary ranges as tested

Low Range	High Range
$30,000 or less	$50,000 or less
$30,001 to $35,000	$50,001 to $60,000
$35,001 to $40,000	$60,001 to $70,000
$40,001 to $50,000	$70,001 to $80,000
$50,001 to $60,000	$80,001 to $90,000
$60,001 to $70,000	$90,001 to $100,000
$70,001 to $80,000	$100,001 to $110,000
$80,001 to $90,000	$110,001 to $120,000
More than $90,000	More than $120,000

Data from the study revealed that the lower floor was definitely needed as many new doctorate recipients' first post-graduation salary comes from a postdoctoral study appointment. Salaries for such appointments are frequently in the $30,000 to $40,000 per year range and are sometimes lower. In addition, in many non-technical fields, entry-level assistant professors earn salaries that are comparable to doctorate recipients who hold postdoctoral study appointments. Indeed, 7.3 percent of all respondents with definite post-graduation plans who responded to the open-ended salary item reported an expected salary of $30,000 or less¹. Additionally, 25.1 percent of respondents with definite post-graduation plans reported an expected salary of $30,001 to $40,000. Nearly one in five (18.1 percent) reported an expected salary between $40,001 and $50,000. In sum, over half (50.5 percent) of all new doctorate recipients expect to earn $50,000 or less in the first year after graduation. The need for a floor that is lower than the lowest category in the experimental "high-range" condition is apparent. See Table 2.

Additionally, one in nine (11.0 percent) new doctorate recipients who responded to the open-ended salary item expected a salary of more than $90,000 in their first year after graduation. About four percent (4.1 percent) of new doctorate recipients expected to earn between $90,001 and $100,000. An additional 2.1 percent of new doctorate recipients expected to earn a salary between $100,001 and $110,000. About one in twenty (4.9 percent) new doctorate recipients indicated an expected salary of more than $110,000. As time progresses and salaries are subject to inflation, this percentage will undoubtedly increase. Therefore, a ceiling that is higher than the highest category in the experimental “low-range” test item is needed. (Table 2).

Table 2. Responses to open-ended questions on actual salary for respondents with definite post-graduation plans, by question format
Range	Both open only and combined			Open only			Combined
	Count	Percent		Count	Percent		Count	Percent
Total	537	100.0		275	100.0		262	100.0
$20,000 or less	13	2.4		6	2.2		7	2.7
$20,001 to $30,000	26	4.8		12	4.4		14	5.4
$30,001 to $40,000	135	25.1		70	25.5		65	24.8
$40,001 to $50,000	97	18.1		54	19.6		43	16.4
$50,001 to $60,000	74	13.8		36	13.1		38	14.5
$60,001 to $70,000	44	8.2		22	8.0		22	8.4
$70,001 to $80,000	50	9.3		20	7.3		30	11.5
$80,001 to $90,000	39	7.3		22	8.0		17	6.5
$90,001 to $100,000	22	4.1		13	4.7		9	3.4
$100,001 to $110,000	11	2.1		7	2.5		4	1.5
$110,001 to $120,000	11	2.1		7	2.5		4	1.5
$120,001 or more	15	2.8		6	2.2		9	3.4

NOTE: Open refers to the question that asks for actual salary only.

Combined refers to the question format that asks for actual salary in an open-ended format, but also offers the option to respond to a closed-ended item.

We recommend that the response options to the salary question incorporate the low floor of the low-range option but also some of the higher ranges from the high-range option. The proposed question uses $5,000 ranges at the very low end of the distribution ($30,000 to $40,000) to retain adequate sensitivity to the lower-earners in the doctorate cohort and $10,000 ranges in the middle and upper portions of the distribution (i.e., $40,000 and up) with a ceiling of $110,000 or more. We feel this range most appropriately balances the demands for sensitivity at both ends of the earning distribution. The recommended ranges will satisfactorily measure current salaries of new doctorate recipients and allows some room for growth in expected salaries as time progresses. Table 3 below shows the recommended categories for the closed range part of the new expected salary item, along with the counts and percentages of open-ended respondents that would have fallen into each category based on Salary Experiment data.

Table 3. Recommended closed-ended salary ranges for expected salary item with frequencies from open-ended respondents
Range	Count	Percent
$30,000 or less	39	7.3
$30,001 to $35,000	41	7.6
$35,001 to $40,000	94	17.5
$40,001 to $50,000	97	18.1
$50,001 to $60,000	74	13.8
$60,001 to $70,000	44	8.2
$70,001 to $80,000	50	9.3
$80,001 to $90,000	39	7.3
$90,001 to $100,000	22	4.1
$100,001 to $110,000	11	2.0
$110,000 or above	26	4.8

The NSF believes that a salary item with these ranges in a closed-ended format following the open-ended option will allow for the collection of the maximum amount of usable data as well as a great deal of high-quality open-ended continuous data.

The Proposed SED Salary Question

B8. What will be your basic annual salary for this principal job (in the next year)? Do not include bonuses or additional compensation for summertime teaching or research. If you are not salaried, please estimate your earned income.

$_________________________

If you prefer not to report an exact amount, please indicate into which range you expect your salary to fall:

0	$30,000		or less		6	$ 70,001	-	$ 80,000
1	$30,001	-	$35,000		7	$ 80,001	-	$ 90,000
2	$35,001	-	$40,000		8	$ 90,001	-	$100,000
3	$40,001	-	$50,000		9	$100,001	-	$110,000
4	$50,001	-	$60,000		10	$110,001		or above
5	$60,001	-	$70,000		11	Don’t		Know

B9. How many months does this salary cover?

number of months |__| |__|

1 Figures based only on respondents with definite plans for post-graduation study or employment who provided expected salary data in one of the two open-ended formats (i.e., open-ended only or open-ended followed by closed-ended). Respondents in these two conditions represented 42 percent (537 of 1,264) of respondents with definite plans for post-graduation study or employment. Respondents to the paper survey were randomly assigned to the 5 versions of the salary question (including a control version). The characteristics of respondents in all 5 treatments were similar.

- 6 -