Logic Model

Appendix 1: Summer of Innovation Logic Model

Summer of Innovation Conceptual Framework

NASA’s Summer of Innovation Project

In 2010, the National Aeronautics and Space Administration’s (NASA) Office of Education launched the Summer of Innovation (SoI) Pilot, a NASA-infused summer experience for middle school students who underperform, are underrepresented, and underserved in science, technology, engineering, and math (STEM) fields. The SoI Pilot utilized a multi-faceted approach to reach and engage middle school students in STEM learning with NASA content and experiences. The topics addressed ranged broadly and included activities concerning robotics, rocketry, engineering design, meteorology, and space science. Data were collected from various sources during the pilot year to produce lessons learned and best practices regarding the project’s design and implementation and to inform its national evaluation.

Drawing on findings from the SoI Pilot, NASA is modifying its approach for the FY2011 implementation, sharpening its focus on student outcomes of engagement and inspiration. The revised SoI model is a four-year project that targets the same middle school youth, defined as students entering 4th through 9th grade in fall 2011 who are underserved in STEM (i.e., females, minorities, and low-income students), for intensive summer and follow-on school year STEM activities. The project emphasizes the following elements: 1) expansion of community and school-based organizations’ capacity to engage youth in STEM learning activities; 2) strategic infusion of NASA content and resources into the intensive STEM activities; 3) professional development for certified middle school teachers; and 4) development of connections between schools and out-of-school STEM education institutions and networks.

SoI staff has worked with the Abt Associates-Education Development Center (EDC) national evaluation team to develop a project logic model for this year’s national awards (Exhibit 1). The logic model identifies the project’s key “ingredients”, the inputs, linking them to the project’s critical activities and to the project’s short and long-term outcomes, illustrating how the project’s elements are intended to work together in pursuit of the project’s ultimate goal; it provides a pictorial representation of SoI’s theory of change.

SoI’s Theory of Change

SoI seeks to address the current situation that too few girls, minorities, and low-income students are engaged and enthusiastic about STEM topics and too many lack inspiration for pursuing STEM careers, while the country’s economy grows increasingly reliant on a STEM-savvy workforce. It intends to provide funding and support for non-profit awardees and their selected partners, including schools, state and federal agencies, non-profit and for-

Exhibit 1: Logic Model for NASA’s Summer of Innovation FY2011 National Awards

profit youth organizations to engage middle school students in intensive STEM activities during the summer and following school year. The Agency seeks to support the SoI awardees and their partners in integrating NASA’s content and leveraging NASA’s resources to expand their capacity to offer ongoing high-quality STEM learning opportunities so that the awardees and their partners will subsequently provide high-quality STEM programs for middle school youth without additional funding from NASA. Exhibit 1 provides a pictorial representation of SoI’s theory of change for its national awards. Moving from left to right, the model makes explicit the links between the project inputs and activities through SoI’s intended short-term and long-term outcomes. Each of the logic model’s elements is described below.

Inputs and Activities

NASA Inputs and Activities

As described in the FY 2011 Cooperative Agreement Notice (CAN), SoI is a solicitation-driven project, through which NASA seeks to “deepen and broaden the efforts of community and school-based organizations”¹ to engage underserved middle school students in high-quality STEM activities related to NASA’s mission of research and discovery. In addition to funding, the Agency will provide SoI awardees with NASA inquiry-based curricula that address topics including earth and space science, robotics, aeronautics, and rocketry. NASA non-monetary resources also will be made available, such as NASA traveling exhibits, NASA videos and photographs, as well as NASA’s scientists and engineers, its research facilities and visitor centers across the country. NASA will provide awardees with recommended NASA activities, lessons and design challenges appropriate for the students.

To develop the capacity of the awardees and their partners to maximize the impact of the NASA content and resources, NASA educators will provide 40 hours of professional development through direct, hands-on activities and collaborative technology to the certified teachers over the course of the summer and the following school year. The professional development includes engagement in hands-on workshops and the use of collaborative technology to build teachers’ understanding of the NASA content and their confidence in using it, as well as how they can customize it to meet local needs and skill levels. In addition, NASA will also provide basic training for non-certified educators (e.g., youth development staff, college students, museum curators, university faculty) through on-demand video, providing them background information on the NASA content and how to use it effectively. While optimally a good portion of these activities would occur prior to the summer’s student activities, and recur before the start of the school-year activities, NASA recognizes time constraints are such that this approach may not be feasible. Accordingly, the sites will have flexibility as to when these hours will be provided.

In addition, NASA intends to expand the awardees and partners’ capacity to offer high-quality STEM learning opportunities through ongoing guidance and support in the implementation of SoI. Specifically, NASA will provide awardees and their partners with promotional materials and strategies for rapid recruitment of teachers and students, NASA content and resources, and a structure for continuous improvement. In good part, NASA’s reporting requirements for the awardees are intended to facilitate the awardees’ identification of lessons-learned and best practices, and the use of this information to improve their SoI programs. These activities will begin at the SoI kick-off meeting in May 2011, when awardees will develop a thorough understanding of NASA’s expectations and goals; they will culminate in an end of year review conference in the fall, where the awardees will share with NASA and other SoI awardees their lessons learned and refined plans for the future SoI activities.

Awardees/Partners Inputs and Associated Activities

The awardees and partners provide the individual models that make up the SoI project. They use NASA’s inputs and activities to refine these models to integrate NASA content and leverage the NASA resources for planning high-quality, intensive hands-on, inquiry-based activities for students. They facilitate NASA’s forty hours of professional development for certified middle school teachers and augment the training support for the non-certified educators implementing the student activities. These tasks require that awardees and/or their partners have education expertise and understand their community’s context so that the modifications made to the NASA content enhance local teachers’ ability to use the content and maximize their potential to engage and inspire the participating middle school youth. In addition, it necessitates that time is set aside for careful planning of the revised model’s activities.

Awardees and partners also address the practical concerns in implementing SoI. They provide access to schools, a key location for recruiting certified middle school teachers and students, and other venues where the SoI activities will occur, as well as the technology needed to engage students in inquiry-based learning. They customize the SoI promotional materials and recruit participants. They plan for the activities, and the awardees fulfill the necessary management and coordination roles associated with high-quality programs. They also comply with NASA’s reporting requirements, participating in the kick-off meeting and the end-of year review meeting, identifying their lessons learned and best practices, and use this information to further improve their programs. Finally, they also obtain or provide additional funding, which becomes increasingly important in the later years of the award period, when NASA begins to reduce its level of financial support.

All these inputs and activities lead up to the summer and school-year SoI student activities, where the certified teachers and/or non-certified educators provide at least 40 hours of interactive STEM programs during the summer (by September 1, 2011) and an additional 25 hours by March 1 of the following school year.

Outputs

The inputs and activities produce outputs across three levels: for awardees and their partners, certified teachers and non-certified educators, and for students. Up to ten awardees and partner networks will receive SoI national funding; each of these awardees and partner networks will produce at least one (possibly more) teacher, educator, and student activities as part of their program model. These awardees will attend the SoI kick-off meeting in May 2011, comply with NASA’s reporting requirements, and be present at the end of year review meeting in fall 2011.

One hundred fifty certified teachers are to participate in the 40 hours of SoI professional development activities, while the educators who also implement student activities are to engage in additional training activities. These instructors – the 150 certified teachers and the number of educators – then are to lead the student activities for 2,500 middle school students during the summer and/or over the course of the school year. While NASA is targeting underperforming and underserved middle school students (i.e., females, minorities, and low-income students), it will not turn away interested middle school youth who are not members of these sub-groups; accordingly, the final output of interest is the number of targeted students reached by the awardees and their partners.

Immediate Outcomes

If SoI’s theory of change is correct, and the awardees, partners, certified teachers, and educators are able to implement their SoI activities with fidelity to this vision, the inputs and activities will produce a series of expected outcomes for each of these three groups both immediately following the activities as well as in the future.

In the short-term, awardees and partners should be able to continue to implement their SoI program models, each using a minimum of 150 certified teachers to provide intensive STEM learning activities for least 2,500 students, while the amount of NASA funding and support decreases. Critically, the quality of the activities should, at minimum, remain constant and preferably improve through the leveraging of the lessons learned and best practices discussed during the end-of-year review meeting. Associated with this outcome are those related specifically to the partnerships: by collaborating over the summer and again during the school year, the partnerships should be sustained and where needed, deepened, and potentially new relationships initiated. The partnerships should be documented, for example through Letters of Agreement (LOAs) or Memoranda of Understanding (MOUs), demonstrating partners’ commitment to the program. This outcome is most likely to result between schools and informal STEM educators as SoI requires that each awardee collaborate with at least one school, district, or state department of education.

SoI should also develop teachers’ awareness of NASA content and resources and increase their use of the NASA materials with middle school students in their classrooms and potentially, in out-of-school activities they also lead, such as afterschool robotics and astronomy clubs.

The intensive student activities led by teachers and informal educators should affect students’ interest in STEM topics and careers, motivating them to participate in additional STEM activities outside of their schools’ curricula where they will continue to explore NASA’s themes. These additional activities include those offered by NASA through the centers and mission directorates, and other youth organizations including Boys & Girls Clubs, scouting groups, museums, and planetariums.

Long-Term Outcomes

A cycle of achieving the immediate outcomes, and the continued implementation and improvement of the activities, are hypothesized to produce SoI’s long-term goals for the awardees and their partners, the teachers, and the students. The theory posits that by the end of the four years of SoI funding, the awardees and their partners will be able to continue to operate their program models, serving at least the same number of students and teachers with high-quality STEM programming, without NASA funding. Furthermore, they will continue to freshen and improve their program’s curricula, making optimal use of new NASA educational content and resources as they are developed, continuing to customize them to suit the needs of their communities.

Teachers’ participation in recurring professional development activities, and the opportunities to practice using the content and skills they are exposed to, eventually should increase their understanding of the material and heighten their confidence in its instruction. Over time, this greater knowledge and confidence should lead to improved ability to teach the NASA topics.

Ultimately, the awardees, partners, and teacher outcomes should address the situation that prompted SoI’s development. If the theory of change proves true, eventually SoI will produce more students who are interested in STEM, who focus on STEM during their post-secondary education, and meet the country’s growing demand for skilled STEM professionals. In addition, given SoI’s focus of engaging the currently underserved students, SoI will increase the proportion of students traditionally underserved in STEM (i.e., females, minorities, and low-income students), who pursue degrees in STEM and utilize the skills and knowledge that they have gained as members of the STEM workforce.

Assumptions

SoI’s theory of change illustrated in the logic model makes certain assumptions about the context in which the project the inputs and activities generate the outputs. The theory of change assumes that NASA will fully fund SoI for the next four years and that the existing NASA-content is correctly aligned with student grade levels so that the SoI awardees and partners can easily identify those best suited for their students. It also depends on the availability of partners, within the communities that the awardees are reaching, who are capable of fulfilling their strategic roles and filling the gaps in the awardees’ own competencies and relationships. It also assumes that the awardees and their partners accurately represent their capacity to implement SoI activities and that they already possess the ability to deliver STEM content. The key staff members applying for the funding must participate and lead the SoI activities for several years, not simply lend their names to strengthen proposals. In addition, their existing program models must be flexible enough to incorporate the NASA content and use the NASA resources well.

The theory of change also makes assumptions about the project’s participants. It assumes that the reason underserved students are not pursuing STEM degrees and careers is because they are not motivated to seek these goals. It assumes that certified middle school teachers are available and interested to engage students in inquiry-based activities incorporating NASA content. These teachers also must have the capacity of learning how to instruct students in the NASA-content and to use the instructional techniques they require. Finally, the theory assumes that students will be interested in what SoI has to offer, and a sufficient number will select it over other summer opportunities.

External Factors

The SoI model recognizes a set of external factors outside of NASA, the awardees, the teachers, and students that may influence the project’s outcomes. The SoI project team identified two. First, external funding must be available to awardees and grantees, to meet their expenses while NASA reduces its financial support in second, third, and fourth years. Secondly, the project’s results depend on the extent to which schools can develop students’ knowledge and skills in STEM outside of the SoI activities, which are necessary to fulfill the requirements for more advanced STEM courses and succeed in post-secondary STEM degree programs, and well-prepare them for futures in STEM careers that the country’s economy demands.

1 From the SoI Pilot to SoI 2011. Internal NASA Education Memo, December 15, 2010.

Abt Associates Inc. Appendix 1 1-2