Download:
pdf |
pdfREQUEST FOR APPROVAL under the Generic Clearance for NASA Education Performance
Measurement and Evaluation, OMB Control Number 2700‐0159, expiration 04/20/2018
_____________________________________________________________________________________
I. TITLE OF INFORMATION COLLECTION:
NASA Office of Education STEM Challenges Impact Surveys: Student Follow‐Up Instruments
II. TYPE OF COLLECTION:
Attitude/Behavior Scale
Baseline Survey
Cognitive Interview Protocol
Consent Form
Focus Group Protocol
Follow‐up Survey
Instructions
Satisfaction Survey
Usability Protocol
GENERAL OVERVIEW: NASA Office of Education Science, Technology, Engineering, and Mathematics
(STEM) Engagement line of business activities are designed to provide opportunities for participatory
and experiential learning activities that connect learners to NASA‐unique resources. NASA Education’s
STEM Engagement line of business activities are based on best practices in motivation, engagement,
and learning in formal and informal settings and include the following areas:
o Public Education Activities that foster interactions with learners of all ages to spark an interest
in STEM disciplines using NASA‐unique materials and resources. These may be part of a larger
public event and are often shorter in duration than Experiential Learning Opportunities and
STEM Challenges. Public Education Activities often require close coordination with the NASA
Office of Communications.
o Experiential Learning Opportunities that enable learners to acquire knowledge, understand
what they have learned, and apply that knowledge through inquiry‐based and project‐based
activities. NASA opportunities include participatory activities designed to increase involvement,
knowledge, understanding/comprehension, and application of learning in one or more STEM
disciplines using NASA’s resources.
o STEM Challenges that provide creative applications of NASA‐related science, technology,
engineering, mathematics, and cross‐cutting concepts. They challenge existing assumptions and
encourage learners to demonstrate their knowledge of STEM subjects while enhancing
innovation, critical thinking, and problem‐solving skills.
In coordination with data and information gathered from the baseline instrument, this follow‐up
instrument information collection is specific to determining the impact of engineering design and
scientific research STEM Challenge activities on middle school students (grades 5 through 8,
depending on the school system of record in the U.S.)
III. INTRODUCTION AND PURPOSE: STEM Challenge activities are based on best practices in motivation,
engagement, and learning for students and educators in formal and informal settings (e.g., Farland‐
Smith, 2012; Gasiewski, Eagan, Garcia, Hurtado, & Change, 2012; Kim, et al., 2015; Leblebicioglu,
Metin, Yardimci, & Cetin, 2011; Maltese & Tai, 2011. The constructs of interest for these follow‐up
surveys are the engineering design and scientific research processes. In a NASA engineering design
Wills
NASA Office of Education
1
challenge (EDC) activity, the focus is a design task in which students must meet certain criteria through
a series of steps that engineers follow to arrive at a solution to a problem. This engineering problem is
within the context of NASA‐unique content and subject matter experts. Similarly, in an a scientific
research challenge (SRC) activity, students are connected with opportunities to participate in science
data collection by conducting real, hands‐on science according to the scientific method, a body of
techniques for investigating phenomena, acquiring new knowledge in an empirical or measurable
manner, and then correcting and/or integrating previous knowledge subject to specific principles of
scientific reasoning.
Our interest is in understanding why, how, and in what ways students are impacted in the short‐,
intermediate, and long‐term by participation in STEM Challenge activities with an engineering design
or scientific research process focus. Thus, the purpose for pilot testing is to develop valid instruments
that reliably explain the ways in which participants’ attitudes and behaviors are impacted by
participation in these activities. Guided by the most current STEM education and measurement
methodologies, it is the goal of this rigorous instrument development and testing procedure to provide
information that becomes part of the iterative assessment and feedback process for the NASA STEM
Engagement line of business.
Hence, the goals of this cycle of pilot testing are as follows:
o Determine clarity, co mprehensibility, and preliminary psychometric properties (e.g.,
validity, reliability) of these instruments. And, to explore individual item functioning, and to
make any necessary adjustments in preparation for large‐scale testing as the basis for more
sophisticated statistical testing.
o Determine an accurate response burden for these instruments.
Relevant information was gathered from representative respondents on the companion baseline
survey through a truncated usability testing. For reason that the follow up instrument is parallel to
the baseline instrument in items and constructs, we are confident that the follow‐up instrument is
adequate for pilot testing.
I. RESEARCH DESIGN OVERVIEW: NASA Education is using a one‐group pretest‐posttest quasi‐
experimental design. Responses will be used to validate these follow‐up surveys for clarity,
comprehensibility, and to determine psychometric properties with the respondent pool.
Following this pilot phase of testing, indeed NASA Education has tentative research questions and
hypotheses to test regarding the impact of STEM Challenge activities on all participants—students and
teachers alike. Thus, this work is integral to the iterative assessment and feedback process for the
STEM Engagement line of business.
II. TIMELINE: Pilot testing of surveys will take place approximately September 1, 2016 through February
28, 2017, coordinated with the implementation periods of the STEM Challenge activities.
III. SAMPLING STRATEGY: NASA Education employed an estimation procedure to determine the
statistically adjusted number of respondents for the final sample size that meets the minimum criteria
for number of respondents (N ≥ 200) necessary to determining preliminary item characteristics
(Komrey & Bacon, 1992; Reckase, 2000). This estimation procedure accounts for the potential
respondent universe, estimated variance in respondent universe, precision desired, confidence level,
and the prior observed response rate for the category of respondents (Watson, 2001). Watson’s
Wills
NASA Office of Education
2
sample size formula as applied to respondent estimates in Table 1 demonstrates the number of
respondents this pilot effort should reach in order to collect the base sample size of respondents
(2001). In brief, this formula suggests that this pilot effort oversample EDC students by 218
respondents. NASA Education will randomly sample EDC sites to meet the 545 respondent minimum,
but because the number of educators in the EDC activity and the number of participants and educators
in the SRC activity are 200 or less, NASA Education will administer surveys for testing to the census of
participants in those categories.
Table 1. Calculation chart to determine statistically relevant number of respondents
Data
Collection
Source
(N)
Population
Estimate for
FY16
(A)
Sampling
Error +/‐
5% (.05)
(Z)
Confidence
Level 95%/
Alpha 0.05
(P) *Variability
(based on
consistency of
intervention
administration)
50%
2,200
0.0025
3.8416
0.5
327
0.6
545
110
N/A
N/A
N/A
110
N/A
110
200
N/A
N/A
N/A
200
N/A
200
10
N/A
N/A
N/A
10
N/A
10
865
EDC
students
SRC
students
EDC
Educators
SRC
Educators
TOTAL
Base
Sample
Size
Response
Rate
(n) Number
of
Respondents
IV. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:
Data Collection Source
EDC students
SRC students
EDC Educators
SRC Educators
TOTAL
Number of
Respondents
545
110
200
10
Frequency of
Response
1
1
1
1
Total minutes per
Response
15
15
5
5
Total Response
Burden in Hours
136
28
17
1
182
*Burden for Educators, in this instance, is calculated to determine the amount of time spent reading
instructions to student survey respondents.
V. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance
will be maintained in accordance with the Privacy Act of 1974, the e‐Government Act of 2002, the
Federal Records Act, and as applicable, the Freedom of Information Act in order to protect
respondents’ privacy and the confidentiality of the data collected.
VI. PERSONALLY IDENTIFIABLE INFORMATION:
1. Is personally identifiable information (PII) collected? Yes No
2. If yes, will any information that is collected by included in records that are subject to the
Privacy Act of 1974? Yes No
3. If yes, has an up‐to‐date System of Records Notice (SORN) been published?
Yes No
Wills
NASA Office of Education
3
Published in October 2007, the Applicable System of Records Notice is NASA 10EDUA, NASA
Education Program Evaluation System ‐ http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.
APPLICABLE RECORDS:
4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA Education Program
Evaluation System ‐ http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html
5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,
Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed,
whichever is longer.
VII. PARTICIPANT SELECTION APPROACH:
1. Does NASA Education have a respondent sampling plan? Yes No
If yes, please define the universe of potential respondents. If a sampling plan exists,
please describe? The universe of potential respondents includes a statistically
representative sample of EDC student participants, the census of EDC educator participants,
and the census of SRC student and educator participants.
If no, how will NASA Education identify the potential group of respondents and how will
they be selected? Not applicable.
VIII. INSTRUMENT ADMINISTRATION STRATEGY
Describe the type of Consent: Active Passive
1. How will the information be collected:
Web‐based or other forms of Social Media (95%)
Telephone
In‐person
Mail
Other (5%)
If multiple approaches are used for a single instrument, state the projected percent of
responses per approach. The feedback forms will be administered via the web. Because it is
preferable that all follow‐up surveys be administered at the end of an activity, hard copy surveys
will be made available to collect survey responses in the event web access is temporarily
unavailable. In the past, no more than 5% of respondents were asked to complete hard copy
surveys due to internet or computer difficulties.
2. Will interviewers or facilitators be used? Yes No
Note: “Facilitators” refers to Educators who will read and explain student survey instructions.
IX. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:
Consent form
Instrument (attitude & behavior scales, and surveys)
Protocol script (Specify type: Script)
Wills
NASA Office of Education
4
Instructions NOTE: Instructions are included in the instrument
Other (Specify ________________)
X. GIFTS OR PAYMENT: Yes No If you answer yes to this question, please describe and provide a
justification for amount.
XI. ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $420. The cost is
based on an annualized effort of 10 person‐hours at the evaluator’s rate of $42/hour for
administering the survey instruments, collecting and analyzing responses, and editing the survey
instruments for ultimate approval through the methodological testing generic clearance with OMB
Control Number 2700‐0159, exp. 04/30/2018.
XII. CERTIFICATION STATEMENT:
I certify the following to be true:
1. The collection is voluntary.
2. The collection is low burden for respondents and low cost for the Federal Government.
3. The collection is non‐controversial and does raise issues of concern to other federal
agencies.
4. The results will be made available to other federal agencies upon request, while maintaining
confidentiality of the respondents.
5. The collection is targeted to the solicitation of information from respondents who have
experience with the program or may have experience with the program in the future.
Name of Sponsor: Richard Gilmore
Title: Educational Programs Specialist/Evaluation Manager, NASA GRC Office of Education
Email address or Phone number: richard.l.gilmore@nasa.gov
Date: 09/13/16
Wills
NASA Office of Education
5
Bibliography
Farland‐Smith, D. (2012). Personal and Social Interactions Between Young Girls and Scientists: Examining
Critical Aspects for Identity Construction. Journal of Science Teacher Education, 23(1), 1‐18.
Gasiewski, J. A., Eagan, M. K., Garcia, G. A., Hurtado, S., & Change, M. J. (2012). From gatekeeping to
engagement: A multicontextual, mixed method study of student academic engagement in
introductory STEM courses. Research in Higher Education, 53(2), 229‐261.
Kim, C., Kim, D., Yuan, J., Hill, R. B., Doshi, P., & Thai, C. N. (2015). Robotics to promote elementary
education pre‐service teachers' STEM engagement, learning, and teaching. Computers &
Education, 91, 14‐31.
Komrey, J. D., & Bacon, T. P. (1992). Item analysis of acheivement tests based on small numbers of
examinees. Paper presented at the annual meeting of the American Educational Research
Association. San Francisco.
Leblebicioglu, G., Metin, D., Yardimci, E., & Cetin, P. S. (2011). The Effect of Informal and Formal
Interaction between Scientists and Children at a Science Camp on Their Images of Scientists.
Science Education International, 22(3), 158‐174.
Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational
experiences with earned degrees in STEM among US students. Science Education, 95(5), 877‐
907.
nasa.gov. (2016, January). Retrieved from NASA Education Implementation Plan 2015‐2017:
http://www.nasa.gov/sites/default/files/atoms/files/nasa_education_implementation_plan_ve
4_2015‐2017.pdf
Reckase, M. D. (2000). The minimum sample size needed to calibrate items using the three‐parameter
logistic model. Paper presented at the annual meeting of the American Educational Research
Association. New Orleans.
Watson, J. (2001). How to Determine a Sample Size: Tipsheet #60. Retrieved from Penn State
Cooperative Extension: http://www.extension.psu.edu/evaluation/pdf/TS60.pdf
Wills
NASA Office of Education
6
File Type | application/pdf |
File Title | Microsoft Word - SHORT FORM. NASA Education STEM Challenges Impact Surveys-Student Follow-Up Instruments |
Author | lwills2 |
File Modified | 2016-09-16 |
File Created | 2016-09-16 |