Supporting Memo

MEMORANDUM
February 27, 2013
To:

Shelly Martinez, Desk Officer
Office of Management and Budget

From:

John Gawalt, Director
National Center for Science and Engineering Statistics

Via:

Suzanne Plimpton, Clearance Officer
National Science Foundation

Subject:

Notification of information collection under generic clearance: Science and Engineering
Indicators knowledge of science questions

The purpose of this memorandum is to inform you that under the generic clearance for survey
improvement projects (OMB number 3145-0101) the National Center for Science and Engineering
Statistics (NCSES) of the National Science Foundation (NSF) plans to investigate the effect of science
content, question wording, and response format on questions about basic science knowledge.
NCSES is seeking clearance for cognitive testing of survey questions about knowledge of science.
BACKGROUND
The National Science Foundation reports information about public understanding of science in Science
and Engineering Indicators (SEI), a publication of the National Science Board (NSB). The quality of the
reported information is dependent on the validity of the knowledge questions as measures of basic factual
knowledge of science, the wording of the questions, and the question format.
NSB members have raised specific concerns about two true-false survey items that have been part of the
SEI basic science knowledge scale for many years: “Human beings, as we know them today, developed
from earlier species of animals” and “The universe began with a huge explosion.” One concern,
particularly with the first item, is that the item may confound religious belief with understanding of the
relevant science. For example, some with certain religious beliefs would respond “false” or “don’t know”
to these statements even if they are aware of the scientific consensus about evolution. A second concern,
mainly with the second item, was that it presented a seriously flawed summary of relevant aspects of the
accepted scientific account of the origins of the universe.
NSF’s Directorate for Social, Behavioral and Economic Sciences (SBE) held two workshops to consider
the conceptualization and measurement of adult science knowledge in response to these concerns. In
outlining a revised framework for research on public understanding of science, the workshops urged
further research on public understanding of the various dimensions of knowledge and acceptance of
evolution and inquiry into public understanding of science as an institution.
In light of the workshop recommendations, NSF is undertaking exploratory work centered on public
scientific knowledge related to the topics covered by the two survey items. The immediate purpose of
this work is to aid in interpreting responses to the existing items that are used in SEI to track trends in
factual knowledge. NSF envisions that this research may, in the long run, contribute to developing

alternate items that might replace the current ones, but this project is not primarily designed to produce
new survey questions for continued use.
NCSES has contracted with the survey research firm Westat to conduct cognitive testing that address the
research questions listed below.
•

•
•

•

To what degree and in what ways do responses to questions testing knowledge of human
evolution differ from responses to questions about the theory of evolution that do not involve
specific reference to humans (e.g., questions about natural selection, questions about inheritance
processes)?
How is respondents’ belief in/acceptance of the theory of evolution related to their likelihood of
correctly responding to questions at different levels of difficulty that test knowledge of/familiarity
with tenets of the theory of evolution?
For questions about the origins of the universe, to what degree and in what way do question
variants (e.g., mention of the age of the universe; focus on a continuous process of expansion and
cooling since the universe originated, rather than on an originating event such as a “big bang” or
“huge explosion”) affect response patterns?
Are there question format variations that demonstrably affect substantive results or measurement
quality for the two true-false items in question and for other true-false science knowledge items?

RESEARCH PLAN
In order to address the four research questions raised above, we propose a combination of expert review
and cognitive interviewing to evaluate survey questions on knowledge of science. 1
Expert review
Experts in biology, the sociology of religion, and astronomy will be critical for ensuring that the questions
address appropriate aspects of the theory of evolution and the origins of the universe. Thus, we propose
crafting draft survey questions and then discussing them with substantive experts. These expert reviews
may be done before and after the cognitive interviews. These expert interviews do not need OMB review
as we are interviewing fewer than nine individuals (some of whom are federal employees) and asking
them individualized questions keyed to their areas of special expertise.
Cognitive interviews
Cognitive interviewing involves administering a draft survey question and obtaining additional oral
information about how the respondents answered the survey questions. This is typically done by
encouraging respondents to think out loud while answering the questions or using follow-up probes after
the respondent answers a survey question. For example, the interviewer may probe a respondent by
asking them “how did you come up with your answer?” The verbal information elicited from this probe
can be used to judge how well the respondent’s answer fits with the objective of a survey question.
We propose conducting two rounds of approximately 20 cognitive interviews on the draft survey items.
In general, we will attempt to recruit subjects with diverse backgrounds on characteristics such as age and
education. We will also need to make efforts to recruit religious individuals since this group tends to

1

Findings from this study will be used to develop questions that will be tested using quantitative experiments; a separate request will be made for
those experiments.

interpret questions on evolution differently than other groups in the population. Cognitive interview
subjects will be paid $40 to complete a 60 minute interview. We will recruit subjects through
advertisements in local newspapers, the Internet website Craig’s List, and fliers in local churches. The
cognitive interview consent form (Attachment B), cognitive interview protocol (Attachment C),
advertisement (Attachment D), and recruitment screener (Attachment E) are attached to this submission.
The use of cognitive interviewing will help us understand how respondents interpret the proposed
questions. A key issue for us is developing items that are understood by respondents who are not
necessarily familiar with scientific terminology and, more importantly, may not understand the relevant
scientific concepts. The draft cognitive protocol shown in Attachment C addresses the following research
topics.
Research topic 1 (Section A of the Cognitive Protocol): Evolution context
First, we will investigate the extent to which questions testing knowledge of human evolution differ from
responses to questions about the theory of evolution that do not involve specific reference to humans. One
of the themes that emerged from discussions with experts in the sociology of religion was that religious
individuals, particularly those from evangelical or fundamentalist backgrounds, are more likely to reject
aspects of evolution that are related to humans than to other species such as plants or micro-organisms. In
particular, religious individuals are likely to reject the notion of humans being descended from other
species. For example, the existing evolution item asks if the statement “Human beings as we know them
today developed from earlier species of animals” is true. Religious individuals do not necessarily object to
the notion of species changing over time, but may object to the notion of humans evolving from “earlier
species of animals.” This may be understood as being in conflict with the literal Biblical account of
creation, in which God created everything at once “according to its kind.”
We propose testing two versions of the item on evolution. One version is similar to the existing item
asking whether humans descended from earlier species of animals. An alternative version replaces
humans with mice in the question.
Question wording 1: Human beings as we know them today descended from earlier species of animals.
Question wording 2: Mice as we know them today descended from earlier species of animals.

Research topic 2 (Section A of the Cognitive Protocol): Origin of the universe question
There are two key issues with the question regarding origins of the universe. First, there is concern about
the best way to describe the scientific account of the origins of the universe. The existing item describes
the “big bang” account of the origins of the universe as a “huge explosion.” A different way to cast this
account would be to highlight the continuous expanding and cooling of the universe. A second issue with
respect to the question about the origins of the universe is the time frame in which the universe began.
Once again, religious individuals may object to a version that claims the universe was created billions of
years ago.
We propose variations of the question about the origins of the universe that manipulate the description of
the origin of the universe and the timeframe in which the universe was formed. The wording of the
existing NSF item is shown below as question wording 1 and potential alternatives are shown in question
wording 2 and 3.
Question wording 1: The universe began with a huge explosion.

Question wording 2: The universe as a whole has been getting larger for billions of years.
Question wording 3: The universe as a whole has been getting larger ever since it began.

Research topic 3 (Section A of the Cognitive Protocol): Question format
We will also be testing whether the form in which the question is asked influences the responses provided
by survey respondents. We propose manipulating both the form in which the answer is given. One half of
the cognitive test respondents will be shown a True/False version of the questions and the other half will
be shown a forced choice version of the questions. The primary concern with the true/false format is
acquiescence bias. That is, respondents may have a tendency to agree (i.e., choose true) with statements
that they are presented regardless of the content of the statements. Rewording the items into a forced
choice format may help minimize this tendency. Most of the original knowledge of science questions
have been asked in a True/False format. Hence analysis of the cognitive test subject’s verbal explanations
of their answers will provide an understanding of whether the forced choice revisions of the items are
adequate to compare the forced choice questions with the True/False questions.
Research topic 4 (Section B of the Cognitive Protocol): Measuring knowledge of evolution
Another theme that emerged from the discussions with experts was that it is important to measure an
individual’s understanding of evolution and the individual’s acceptance of the theory separately, so that
the relationship between the two can be studied. There were several aspects of evolution that the experts
thought that people should understand. Although many topics of understanding were discussed, there
were a few core concepts that everyone seemed to agree on. These include how species evolve over time,
common ancestry, the inheritance of traits, and the idea that change can occur either slowly or quickly.
Fortunately several resources are available to measure knowledge and acceptance. Items may be adapted
from sources such as the American Association for the Advancement of Science’s Project 2061 website.2
The item below is about common ancestry; the correct response is underlined.
Which of the following statements is TRUE about the evolution of plants and animals?
A. All plants and all animals share a common ancestor with each other.
B. All plants share a common ancestor, but all animals do not share a common ancestor.
C. All animals share a common ancestor, but all plants do not share a common ancestor.
D. No plants share a common ancestor with each other, no animals share a common ancestor with each
other, and no plants share a common ancestor with any animals.
The items may need to be adapted so that they can be more easily administered in the context of a future
survey of the general population where respondents are likely to spend less effort answering the question.
For example, we could simply present the statement “All plants and all animals share a common ancestor
with each other” and ask respondents to indicate whether this is true or false. The cognitive testing
described earlier may turn up interpretive problems with the phrase “common ancestor.” If so, we may
want to rewrite this item to make it easier to understand.
Research topic 5 (Section C of the Cognitive Protocol): Measuring acceptance of evolution

2

See http://assessment.aaas.org/topics/EN#/ for examples.

Distinct from an individual’s knowledge of evolution is his or her acceptance of various aspects of the
theory. One may perfectly understand the theory, but for religious or other reasons choose to reject
different aspects of it. Hence, the acceptance of the theory may partially explain one’s response to items
regarding different aspects of the theory of evolution. This makes measurement of the acceptance of
evolution important.
We propose a couple of different approaches to measuring the acceptance of evolution. First, we propose
taking a direct approach similar to that taken by the Gallup Organization. The Gallup Poll has used this
item: “Do you, personally, believe in the theory of evolution, do you not believe in evolution, or don’t
you have an opinion either way?”
In addition, we propose to develop a multiple item scale to tap the construct of acceptance. This approach
is taken with the multi-item Measure of Acceptance of the Theory of Evolution instrument (MATE).3 The
MATE contains items about the process of evolution, scientific validity of evolutionary theory, evolution
of humans, evidence of evolution, scientific community’s view of evolution, and age of the earth. It was
designed to assess high school biology teachers’ overall acceptance of evolution so it is likely to require
some revision for administration in a future survey of the general population. However, it could serve as
a starting point for a multi-item measure.
A final set of items will be designed to assess the underpinnings of religious objections to the theory of
evolution. For example these items might ask respondents about their beliefs about the implications of
the theory of evolution as shown below in the following item:
C5. If the theory of evolution is true, does this mean that humans are not special compared to
other animals or does it not mean this?
Means that humans are not special compared to other animals
Does NOT mean this
DATA COLLECTION INSTRUMENT
A copy of the questionnaires are provided in Attachment A.
CONFIDENTIALITY
Respondents will be provided the following with all materials.
This information is solicited under the authority of the National Science Foundation Act of 1950, as
amended, and the Confidential Information Protection and Statistical Efficiency Act of 2002. The
information you provide will be used for statistical purposes only. Your responses will be kept
confidential. Your response is voluntary and failure to provide some or all of the requested information
will not in any way adversely affect you.
BURDEN INFORMATION
The estimated respondent burden is 5 hours to screen participants for the study and 40 hours to complete
the cognitive interviews. A total of 45 burden hours are included as part of this request.

3

http://www.nabt.org/websites/institution/File/pdfs/american_biology_teacher/2007/069-06-0332.pdf

INCENTIVE PAYMENTS
Cognitive interview subjects will be paid $40 to complete a 60 minute interview.

CONTACT INFORMATION
The contact person for questions regarding this data collection is:
Robert Bell
Attachments
• Attachment A: Questionnaires used in cognitive testing
• Attachment B: Cognitive Interview Consent Form
• Attachment C: Cognitive Interview Protocol
• Attachment D: Advertisement
• Attachment E: Screener

cc: