NCSES - Request for approval of two tests for the Survey of Doctorate Recipients (SDR)

Attachment E: DIS Pilot Study Response Analysis Survey (RAS)
Questionnaire
RAS Introduction

The next few questions ask about your experience completing the Survey of Doctorate Recipients
pilot study. Your answers will be very helpful in guiding future cycles of the survey.
RAS1. How would you rate the speed at which you completed today’s survey?
Very slow
Somewhat slow
Somewhat fast
Very fast
RAS2. How would you describe today’s survey relative to other SDR surveys you’ve completed?
Very similar
Somewhat similar
Somewhat dissimilar
Very dissimilar
RAS3. To what extent did you enjoy completing today’s survey?
Did not enjoy at all
Enjoyed a little
Enjoyed somewhat
Enjoyed a great deal
RAS4. How sensitive did you think the questions on this survey were?
Not sensitive at all
A little sensitive
Somewhat sensitive
Very sensitive
RAS5. How confident are you that NCSES will protect your answers?
Not at all confident
A little confident
Somewhat confident
Very confident
RAS6. (DI ONLY) Today’s survey referenced the 201x Survey of Doctorate Recipients. Do you
remember completing that survey?
Yes
No
93

RAS7. (DI ONLY) Prior to answering today’s survey, did you remember what your responses to the
201x Survey of Doctorate Recipients had been?
Yes
No
RAS8. (DI ONLY) Have you ever participated in a survey in which historical information about you had
been pre-filled for you to confirm or update?
Yes
No
RAS9.
(DI) One of the goals of this survey is to understand people’s reactions to seeing their prior answers
pre-filled on the survey.
Please share your own reactions to seeing your 201x responses to the Survey of Doctorate Recipients
pre-filled in today’s survey.
(INDI) We are interested in understanding how people might react if some of their answers from the
last time they responded to the Survey of Doctorate Recipients were pre-filled in today’s survey.
People would then confirm or update the answers (rather than having to re-enter information that
had not changed).
Please share how you might have reacted if your most recent responses to the Survey of Doctorate
Recipients had been pre-filled in today’s survey.
[Randomize the order of these items.]
a.
b.
c.
d.
e.
f.
g.

Surprised
Confused
Appreciative
Comfortable
Annoyed
Concerned
Relieved

Very much
1
1
1
1
1
1
1

A little
2
2
2
2
2
2
2

Not at all
3
3
3
3
3
3
3

RAS10. Please use this space to describe any other reactions you (DI: had/RM: might have), or to
explain the reactions you indicated in the prior question.
[open-ended verbatim response]

94

RAS11.
(DI) Another goal of this survey is to determine whether people are more or less accurate with their
answers when their prior answers are pre-filled.
Do you think that pre-filling some of your answers from 201x helped your answers to today’s survey
to be…
(INDI) We are also interested in whether people will be more or less accurate with their answers
when their prior answers are pre-filled.
Do you think that pre-filling your most recent answers to the Survey of Doctorate Recipients and
asking you to confirm or update them would lead to your answers being…
More accurate
Less accurate
No impact
RAS12. (DI ONLY) Were there any questions in today’s survey where you felt that the answer
displayed from 201x was “accurate enough” and you decided to leave it as-is, rather than updating it
with potentially more accurate information?
Yes for one question
Yes for more than one question
No
RAS12a. [IF YES IN RAS12] Please use this space to help us understand your reasoning for how you
determined when to leave answers as-is, rather than providing more accurate information.
[open-ended verbatim response]
RAS13. (DI ONLY) Were there any questions in today’s survey where the answer displayed from 201x
was no longer true, and you decided to leave it as-is, rather than updating it with current
information?
Yes for one question
Yes for more than one question
No
RAS13a. [IF YES IN RAS13] Please use this space to help us understand your reason for how you
determined when to leave answers as-is, rather than providing current information.
[open-ended verbatim response]

95

RAS14.
(DI) Do you think that pre-filling some of your answers from 201x and asking you to confirm or update
them made this survey…
(INDI) If this survey had pre-filled your most recent answers and asked you to confirm or update
them, would that have made this survey…
1
2
3
4
5

Much more burdensome
A little more burdensome
Neither more or less burdensome
A little less burdensome
Much less burdensome

RAS15. Do you think pre-filling your most recent answers to Survey of Doctorate Recipients is a…
1
2
3
4
5

Very bad idea
Somewhat bad idea
Neither a good nor a bad idea
Somewhat good idea
Very good idea

RAS16. Did you use records (such as a resume, curriculum vitae, pay stubs, or tax records) to answer
any of the questions on today’s survey?
1 Yes
2 No
IF RAS17=1, CONTINUE. ELSE GO TO RAS17.
RAS16a. For which questions did you use records?
[open-ended verbatim response]

RAS17. Do you have any other suggestions for how to improve the way we ask about employment on
the Survey of Doctorate Recipients?
[open-ended verbatim response]

Thank you

Thank you for participating in the Survey of Doctorate Recipients pilot study. We appreciate your
time!

96