Download:
pdf |
pdfATTACHMENT C: COGNITIVE INTERVIEW PROTOCOL
NSF Cognitive Interview Protocol
Introduction
Thank you for coming in today. Have you been with us here before? Let me tell you a little about
Westat and what we’re going to do today.
Westat is a company that conducts research on many topics, for different organizations. This
particular project is for the National Science Foundation. Westat has been asked to conduct a
survey about what people know about science. For this survey thousands of people will be asked to
complete a survey on the Internet. Before that happens though we like to test the questions out on
people like yourself.
At this stage we are still developing the questionnaire and trying to figure out how to make it as easy
as possible for people to understand and fill out correctly. So that is what you’ll be helping us with.
Informed Consent
Before we get started, I want to let you know that everything we cover in the interview will be kept
private - only the few people actually working on the project will have access to the information you
share with us. If there are any questions you don’t want to answer that is perfectly fine - We’re really
more interested in how you interpret the questions.
If it’s OK with you, I would like to tape our conversation, just so that I can review it later when
we’re analyzing all the interview data.
IF NEEDED: And finally, some of our colleagues from the National Science Foundation are here
today – behind the mirror - to observe and see for themselves how the questions work.
Here is a form I must ask you to look over and sign – it basically covers the points I’ve just gone
over with you and indicates you have agreed to take part.
HAVE RESPONDENT SIGN INFORMED CONSENT. GIVE SECOND COPY TO RESPONDENT TO
KEEP.
[IF RESPONDENT AGREES TO RECORDING]
I’m going to start the recorder. Today is ___________________. Now that the recorder is running,
let me ask again: is it okay with you if I record this interview?
Procedures
So let’s move on. I’d like you to fill this out the best you can, but let me say a few things first. I’d
like you to read aloud as you go along – just read the question and whatever your answer is. I’m
going to be following along and filling in your answers on my copy. As much as possible, just read
and respond to this the way you think you normally would if you were at home doing this on your
own. Again, some people might read practically every word they see, while others might be inclined
to skip stuff here and there – just do whatever feels right to you.
Please share whatever reactions come to mind while you’re filling it out. At any time you should feel
free to let me know if:
•
•
•
•
there are words you don’t understand, or
the question doesn’t make sense to you, or
you could interpret it more than one way, or
if the answer you are looking for is not provided.
Don’t worry about criticizing the questions. I didn’t write them so don’t be concerned about
hurting my feelings or anything like that.
Finally, we’ll pause now and then to discuss something about a question.
For example I may ask you to explain your answer in more detail or I might ask questions such as:
•
•
•
how did you come up with your answer, or
how did you remember that, or
what did it sound like the question meant to you?
These follow up questions may seem strange, but bear with me. They help me understand how well
the questions are working. Do you have any questions before we begin?
Section A.
HAVE RESPONDENT COMPLETE ENTIRE SECTION BY READING ITEMS ALOUD AND
ANSWERING. BEFORE MOVING ON TO NEXT SECTION, RETROSPECTIVELY PROBE ON THE
FOLLOWING ITEMS.
True/False Versions of the Questions
HALF OF THE RESPONDENTS WILL RECEIVE THE TRUE/FALSE VERSION.
HALF OF THE RESPONDENTS WILL RECEIVE THE FORCED CHOICE VERSION.
Now, I would like to ask you a few short questions like those you might see on a
television game show. For each statement that I read, please select true or false.
If you don't know or aren't sure, select don’t know and go to the next question.
A1.
1. First, the center of the Earth is very hot.
A2.
2. All radioactivity is man-made.
A3.
True
False
Don’t know
True
False
Don’t know
3. It is a chromosome from the father’s genes that decides whether the baby
is a boy or a girl.
True
False
Don’t know
27
A4.
4. Lasers work by focusing sound waves.
True
False
Don’t know
How did you come up with your answer? How certain are you of your answer?
IF DON’T KNOW, Could you take your best guess? How confident would you feel about
your guess?
A5.
5. Electrons are smaller than atoms.
A6.
6. Antibiotics kill both viruses and bacteria.
True
False
Don’t know
True
False
Don’t know
28
A7.
7. The universe began with a huge explosion.
True
False
Don’t know
How certain are you of your answer?
IF TRUE, How did you know this was true?
IF FALSE, Tell me more about why this is false. How would you rewrite this sentence so
that it was true?
IF DON’T KNOW, Did you learn anything about this in school?
A7_1.
8. The universe has been expanding for billions of years.
True
False
Don’t know
Tell me more about your answer. IF FALSE, Tell me what’s wrong with this sentence.
IF DON’T KNOW, Did you learn anything about this in school?
29
A7_2
I’d like to show you another question that is similar. DISPLAY SHOWCARD 1.
SHOWCARD 1
The universe has been expanding ever since it began.
True
False
Don’t know
In your opinion, are these two questions the same or different? What makes them the
same/different? IF NEEDED, How would you answer this second question?
30
A8.
9. The continents on Earth have been moving for millions of years and will
continue to move in the future.
True
False
Don’t know
A9_1.
10. Mice descended from earlier species of animals.
True
False
Don’t know
In your own words, what is this question asking?
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
A10.
11. The Earth goes around the Sun.
True
False
Don’t know
31
A9.
12. Human beings descended from earlier species of animals.
True
False
Don’t know
In your own words, what is this question asking?
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
IF FALSE, How would you reword this sentence so that it is true?
IF RESPONDENT ANSWERS FALSE TO THIS ITEM BUT TRUE TO Q10,
Tell me more about why your answers to these two questions were different.
NOTE IF RESPONDENT REVISITS Q10 AFTER READING THIS ITEM.
AFTER COMPLETING PROBING FOR ENTIRE SECTION,
What did you think of this series of questions?
IF NEEDED, Were any terms difficult to understand?
32
Forced Choice Versions of the Questions
HALF OF THE RESPONDENTS WILL RECEIVE THE TRUE/FALSE VERSION.
HALF OF THE RESPONDENTS WILL RECEIVE THE FORCED CHOICE VERSION.
Now, I would like to ask you a few short questions like those you might see on a
television game show. For each statement that I read, please select the best
answer. If you don't know or aren't sure, select don’t know and go to the next
question.
AA1.
1. First, is the center of the Earth very hot or is it not very hot?
AA2.
Very Hot
Not Very Hot
Don’t know
2. Is all radioactivity man-made or does some occur naturally?
All man-made
Some natural
Don’t know
AA3.
3. What decides whether a baby is a boy or a girl? Is it a chromosome from
the father’s genes, from the mother’s genes, from both, or from neither?
Father’s chromosome
Mother’s chromosome
Both
Neither
Don’t know
33
AA4_1.
4. Do lasers work by focusing sound waves or do they work in some other
way?
Sound waves
Some other way
Don’t know
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
AA4.
I’d like to show you another question that is similar. DISPLAY SHOWCARD 2.
SHOWCARD 2
Do lasers work by focusing sound waves, light waves, or do they work in
some other way?
Sound waves
Light waves
Some other way
Don’t know
How would you answer this second question?
IF RESPONDENT ANSWERS “SOUND WAVES” TO FIRST AND “LIGHT WAVES” TO
SECOND, Tell me more about why your answer changed.
Which version of the question do you prefer? Why?
34
AA5.
5. Are electrons smaller, larger, or the same size as atoms?
AA6.
Smaller
Larger
The same size
Don’t know
6. Do antibiotics kill bacteria only, viruses only, or both viruses and bacteria?
AA7.
Bacteria only
Viruses only
Both bacteria and viruses
Don’t know
7. Did the universe begin with a huge explosion or did it not begin this way?
With a huge explosion
Did not begin this way
Don’t know
How certain are you of your answer?
IF DID NOT BEGIN THIS WAY, How did the universe begin?
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
35
AA7_1
8. Has the universe been expanding, shrinking, or has it been pretty much the
same size since it began?
Expanding
Shrinking
Same size
Don’t know
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
AA7_ 2.
I’d like to show you another question that is similar. DISPLAY SHOWCARD 3.
SHOWCARD 3
Has the universe been expanding, shrinking, or has it been pretty much the
same size for billions of years?
Expanding
Shrinking
Same size
Don’t know
In your opinion, are these two questions the same or different? What makes them the
{same/different}? IF NEEDED, How would you answer this second question?
36
AA8.
9. Have the continents on Earth been moving for millions of years or have the
continents pretty much stayed in place?
Moving
Stayed in place
Don’t know
AA9_1.
10. Did mice descend from earlier species of animals or did they not descend
from earlier species of animals?
Did descend from earlier species
Did not descend from earlier species
Don’t know
In your own words, what is this question asking?
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
AA10.
11. Now, does the Earth go around the Sun, or does the Sun go around the
Earth?
Earth goes around the Sun
Sun goes around the Earth
Don’t know
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
37
AA9.
12. Did human beings descend from earlier species of animals or did they not
descend from earlier species of animals?
Did descend from earlier species
Did not descend from earlier species
Don’t know
In your own words, what is this question asking?
IF DID NOT DESCEND, How did human beings appear on earth?
IF RESPONDENT ANSWERS “DID NOT” TO THIS ITEM BUT “DID” TO Q10
Tell me more about why your answers to these two questions were different.
IF DON’T KNOW, Did you learn anything about this in school?
Could you take your best guess? How confident would you feel about your guess?
NOTE IF RESPONDENT REVISITS Q10AFTER READING THIS ITEM.
AFTER COMPLETING PROBING FOR ENTIRE SECTION,
What did you think of this series of questions?
IF NEEDED, Were any terms difficult to understand?
38
Section B.
HAVE RESPONDENT COMPLETE ENTIRE SECTION BY READING ITEMS ALOUD AND
ANSWERING. BEFORE MOVING ON TO NEXT SECTION, RETROSPECTIVELY PROBE ON THE
FOLLOWING ITEMS.
The next questions are not about what you think, but are about what most
scientists think. Please tell us whether each of the following is True or False
according to modern biology.
In your own words, who are “most scientists”? Do you feel you know what “most
scientists” think? IF YES, Tell me more about how you know.
To you, what is “modern biology”?
According to modern biology…
True
False
Modern biology
doesn’t say
True
False
Modern biology
doesn’t say
A. Individual members of a species are
equally able to find food and
reproduce.
According to modern biology…
B. Even species that look different such as
cats and worms can be similar
genetically.
In your own words, what is this saying?
How did you decide on your answer? How confident of your answer are you?
39
According to modern biology…
True
Modern biology
doesn’t say
False
C. Some bacteria may inherit a
gene that helps them survive
the effects of drugs. These
bacteria will be more likely to
live when exposed to the
drugs.
According to modern biology…
True
False
Modern biology
doesn’t say
D. All plants and animals, including
humans, share a common ancestor that
lived a long time ago.
What does “common ancestor” mean to you?
Tell me more about your answer. IF FALSE, Tell me what’s wrong with this statement.
IF FALSE, Let’s say this question asked whether the following was true or false
according to the theory of evolution. How would you answer?
IF DIFFERENT, Why is your answer different?
40
According to modern biology…
True
False
Modern biology
doesn’t say
E. Physical characteristics such as big
muscles that an animal acquires during
its lifetime will be passed on to its
offspring.
In your own words, what is this saying?
What does it mean for a characteristic to be acquired during the lifetime of an organism?
Can you give me an example?
According to modern biology…
True
False
Modern biology
doesn’t say
F. People of some races have better
genetic traits than people of other races.
In your own words, what is this saying? IF NEEDED, What does it mean by “better
genetic traits”? Can you give an example?
How did you decide on your answer?
41
According to modern biology…
True
False
Modern biology
doesn’t say
G. Evolution means that all species are
slowly evolving towards perfection.
Are you familiar with the theory of evolution? IF YES, In your own words, what is the
theory of evolution?
How did you decide on your answer?
In your own words, what does “slowly evolving toward perfection” mean? Can you give
an example?
According to modern biology…
True
False
H. All dogs and cats share a common
ancestor that lived a long time ago.
IF FALSE AT D AND TRUE AT H, Tell me more about your answer.
IF NEEDED, Tell me about the differences between item D and this one.
42
Modern biology
doesn’t say
According to modern biology…
True
False
Modern biology
doesn’t say
I. During their lifetimes, plants develop
traits like growing toward the sun that
they need in order to live.
Tell me more about your answer.
What does it mean to “develop traits during their lifetimes”? Can you give me an
example?
According to modern biology…
True
False
Modern biology
doesn’t say
True
False
Modern biology
doesn’t say
J. When a predator is introduced into a
population of lizards, the lizards with
traits that help them avoid the predator
are more likely to survive and reproduce
than the others.
According to modern biology…
K. Pigeons and frogs share a common
ancestor that lived a long time ago.
IF FALSE, Let’s say this question asked whether the following was true or false
according to the theory of evolution. How would you answer? IF DIFFERENT, Why is
your answer different?
43
According to modern biology…
True
False
Modern biology
doesn’t say
L. A population of birds that lives in two
different places may slowly adapt to
those places and over many generations
become two different species.
IF FALSE, Let’s say this question asked whether the following was true or false
according to the theory of evolution. How would you answer?
IF DIFFERENT, Why is your answer different?
According to modern biology…
True
False
M. People’s fates are determined by their
genes and not by free will.
Tell me more about your answer.
What does it mean to say “people’s fates are determined by their genes”?
44
Modern biology
doesn’t say
AFTER COMPLETING PROBING FOR ENTIRE SECTION,
What did you think of this series of questions?
IF NEEDED, Were any terms difficult to understand?
FOR ALL ITEMS WHERE RESPONDENT ANSWERED ‘BIOLOGY DOESN’T SAY,’
Tell me more about your answer here.
DETERMINE IF RESPONDENT IS USING THIS CATEGORY AS “DON’T KNOW.”
ASK RESPONDENTS TO COMPARE D, H, AND K.
These questions all ask about a common ancestor.
To you, are these questions asking the same thing or different things?
IF DIFFERENT, What is different about these questions?
45
Section C.
HAVE RESPONDENT COMPLETE ENTIRE SECTION BY READING ITEMS ALOUD AND
ANSWERING. BEFORE MOVING ON TO NEXT SECTION, RETROSPECTIVELY PROBE ON THE
FOLLOWING ITEMS.
C1
2. For the following items, please indicate your agreement/disagreement with
the given statements using the following scale:
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
N. The modern theory of
evolution is just
speculation.
Are you familiar with the theory of evolution?
IF YES, What do you about the theory of evolution? IF NEEDED, Where did you learn
about it?
What does “speculation” mean?
IF AGREE OR STRONGLY AGREE, Tell me more about your answer. How do you know it
is based on speculation?
Strongly
Agree
Agree
O. Strong scientific
evidence shows that
evolution has actually
occurred.
What is “scientific evidence”?
46
Undecided
Disagree
Strongly
Disagree
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
P. The earth is less than
10,000 years old.
Tell me more about your answer. Where did you learn about the age of the earth?
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
Q. Mice today are
basically the same as
mice have always
been.
R. The Earth is at least 4
billion years old.
S. The modern theory of
evolution cannot be
correct because it
disagrees with what
the Bible says.
Are you familiar with the Bible’s account of creation? Can you briefly describe it? How
did you learn about it?
47
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
Strongly
Agree
Agree
Undecided
Disagree
Strongly
Disagree
Agree
Undecided
Disagree
Strongly
Disagree
T. Humans exist today in
the same form they
always have
U. Most scientists doubt
whether evolution
occurs.
Tell me more about your answer.
Strongly
Agree
V. Over time, many new
types of plants and
animals have come
into existence.
48
C2.
13. Do you, personally, think that the theory of evolution is correct, do you
think it is incorrect, or don’t you have an opinion either way?
Think the theory of evolution is correct
Think the theory of evolution is incorrect GO TO QUESTION 4
Don’t have an opinion GO TO QUESTION 5
C2A.
14. Why do you think the modern theory of evolution is correct?
15. Why do you think the modern theory of evolution is incorrect?
PROBE ON RESPONDENT’S EXPLANATION.
49
C3.
16. How much evidence do you think supports the modern theory of
evolution?
Overwhelming evidence
A lot of evidence
Some evidence
A little evidence
No evidence
Tell me what you know about the theory of evolution. How did you learn about it?
C5.
17. Does the modern theory of evolution claim that human beings are just
another species of animal with nothing really special about them?
Yes
No
What is this question asking? IF NEEDED, What does “just another species of animal”
mean exactly?
Tell me more about your answer.
How easy or difficult was it to answer this question? What made it {easy/difficult}?
50
C7.
18. Does the theory of evolution lead children to believe that there is no God or
does it not lead to this belief?
Does lead to this belief
Does not lead to this belief
What is this question asking about?
How easy or difficult was it to answer this question? What made it {easy/difficult}?
C9.
19. How much does the theory of evolution conflict with a belief in morality?
Not at all
A little
Some
A lot
Completely
What is this question asking about? IF NEEDED, What is morality?
Tell me more about your answer.
51
Closing and Incentive
IF OBSERVERS ARE PRESENT, CHECK TO SEE IF THEY HAVE FURTHER QUESTIONS.
That is all the questions I have for you.
Is there anything we haven't discussed that you would like to mention?
DISCUSS ANY RESPONDENT COMMENTS.
Thank you for your time.
STOP TAPE RECORDER.
GIVE INCENTIVE AND HAVE RESPONDENT SIGN RECEIPT.
52
| File Type | application/pdf |
| Author | Aaron Maitland |
| File Modified | 2013-03-05 |
| File Created | 2013-03-05 |