The contractor will identify sites through two concurrent processes:
Sites identified as leading users of online learning identified through the literature search.
Polling of TWG members, providers of online learning identified through the literature search, and leading researchers in online learning.
The contractor is currently conducting a meta-analysis that includes a comprehensive search of the literature as well as statistical procedures for estimating the effects of particular characteristics associated with online learning across particular interventions. Although results from the meta-analysis will not be available until September 2007, the contractor anticipates the characteristics most likely to demonstrate an effect on student learning through a review of existing literature reviews. These include:
Significant asynchronous learning component, where students interact with curricular materials and communicate with instructors through email, the Web, and other means that do not require teachers and students to be interacting at exactly the same time;
Pedagogical approaches that are interactive and allow students increased control over the pacing and content of courses;
Instructional quality measures, such as including requirements that teachers be appropriately credentialed or a curricular review process with subject-matter and instructional experts.
Before finalizing the set of case study sites, the contractor will update the selection criteria to reflect specific findings from the meta-analysis.
Due to review and publication requirements, there is typically a two-to-five year lag between the time an intervention is implemented and studied and when it might appear in a peer-reviewed article. Therefore, in addition to the literature review, the contractor will ask TWG members and other recognized experts in online learning to nominate interventions that they find particularly effective or promising. These experts will be asked to identify online implementations with documented effectiveness or practices which they believe have a high-probability of improving student academic outcomes. This nomination process allows the contractor to identify new and innovative approaches that may not be documented in the literature included in the meta-analyses. The contractor will collect additional information about nominated sites in order to evaluate whether the nominations meet our selection criteria.
The contractor will use a phased selection process. First, the contractor will review the Websites of possible nominees to see if the application meets the initial inclusion criteria. Data for this review will come from the literature, Web searches about the online activity, and telephone interviews with potential developers. The protocol developed for initial telephone interviews with potential developers has been included in Appendix F.
Although the inclusion criteria may need to be revised slightly to reflect results from the meta-analysis, the contractor has developed the following guidelines for selecting the ten K-12 case studies.
Curriculum should be comparable to traditional courses in duration and intensity (e.g. eight to twelve weeks of instruction covering multiple sub-topics associated with the course topic).
The learning experience should include a significant asynchronous component, whereby twenty percent or more of instruction is designed to take place asynchronously. The balance may be synchronous online communication (e.g., text chat, Voice-over-IP) and / or with face-to-face interaction.
The learning experience must represent one of the following five categories: (1) a course that is part of a formal online program that offers students degrees; (2) a standalone course addressing an academic subject area, especially advanced courses; (3) a “blended course” with more than twenty percent of instruction occurring face-to-face and focused on an academic subject matter, especially math or science (4) a “blended course” in a multidisciplinary topic areas (e.g., I*EARN, ePals1); and (5) an informal learning activity, including supplemental services. These five categories represent the dominant offerings in the online learning space for K-12 students.
The contractor has developed the following, related guidelines for selecting the ten teacher professional development case studies.
Courses, graduate degree, and mentoring programs should be comparable to traditional offerings in duration and intensity (e.g. courses of eight to twelve weeks of instruction covering multiple sub-topics associated with the course topic; mentoring should be at least one school-year in length).
The learning experience should include significant asynchronous component, whereby twenty percent or more of instruction is designed to take place asynchronously. The balance may be synchronous online communication (e.g., text chat, Voice-over-IP) and / or face-to-face interaction.
The learning experience must represent one of the following five categories(1) a course from a degree program (e.g., an online or blended M.A. program); (2) an individual course offered internally within a school district; (3) and non-district individual course offered to a wide range of users; (4) an established mentoring or coaching program offered by the teachers’ local school district; and (5) an established mentoring or coaching program offered by non-district personnel to users nationwide. These five categories represent typical implementations in the online teacher professional development space.
The contractor will classify interventions that meet the above professional development criteria as either “entirely online” or “blended.” Interventions that are classified as entirely online will not require any face-to-face interaction between students and teachers or among students. All communication, support, and activity will occur online. Blended approaches will require some face-to-face interaction; the intervention may be implemented in a traditional school and classroom where teachers or other faculty provide support and feedback, or the intervention may require teachers to attend short workshops before, during, or after attending an online course.
When choosing among several possible candidates within a given category, the contractor will give priority to interventions that have demonstrated positive effects on student learning. Across cells, the contractor will strive for diversity in terms of geographic locations and type of provider. Providers can be state or local governments, universities, non-profits, or for-profit entities.
Once the Department has approved a set of specific online offerings, the contractor will work with providers of those offerings to identify sites that represent strong and typical implementations. The contractor expects that these sites will include both physical and virtual locations. For physical locations, such as schools, the contractor will screen potential sample schools for student achievement, particularly for subjects and grades associated with the offering when possible. The contractor will eliminate from the sample any schools nominated as exemplary that have below-average achievement compared to other schools in their state serving similar student populations. These schools should be excluded from study because past studies have been criticized for promoting practices in schools that were not accomplishing at least average student achievement outcomes. The two proposed implementation sites for each offering will be submitted to the Department and the TWG for comment and approval. Sites will be notified of their selection only after Department approval of their inclusion in the study.
The data collection plan will address the evaluation questions articulated in Section I:
What conditions influence the quality of online learning?
What implementation practices support effective online learning?
What are emergent, promising practices for effective use of online learning to replace face-to-face K-12 courses?
What are emergent, promising practices for effective use of online learning to enhance face-to-face instruction in K-12 settings?
The plan for this data collection will be submitted to the Department in September 2007. The plan will carefully describe the types and characteristics of online instructional offerings that are of greatest interest based on the project framework, the results of the literature review, and federal and state policy relevance. The initial and revised plan for the online learning case studies will cover the following core areas: key evaluation questions for the data collection; data collection methods; potential survey item indices, if applicable; site visit interview, observation and focus group protocols, if applicable; proposed notification procedures; data collection timeline; and preliminary plan for data analysis.
The data collection plan will cover the site visits to be conducted during Year 2 of the project and will describe the criteria for selecting sites as well as goals associated with the case study visits. We propose three types of site visits: developer site visits, school site visits, and virtual site visits. For developer site visits, the contractor will meet with providers of selected offerings in order to learn more about the functionality associated with the particular offering and collect information associated with the use of the offering and associated outcomes. Before conducting a site visit, researchers will conduct a thorough review of material available on the Web about the product. During developer visits, researchers will learn about the offering through interviews and system demonstrations.
For virtual courses and related offerings, the contractor will seek teachers, tutors, and support staff located at or near the developer site for interviews. In many cases, developers collect information about site use, frequency of interaction between students and teachers, and related information, and the contractor will collect this information where it is available in aggregate form (thus protecting student privacy). Curriculum and support materials, ancillary services, policies, and guidelines associated with the online offering will also be collected when available. The contractor will request log-in information so that researchers can interact with the offering after the site visit.
For school site visits, the contractor will collect data through interviews, focus groups, and observations. Interviews will be conducted with key staff associated with the offering, including principals, technology coordinators, and teachers. Researcher queries will ask for details about the goals and rationales for use, the frequency and context of use of the offering, and the supports and barriers to its use at the site. (For a list of topics and draft items to be covered by the protocol, please see Appendix D for K-12 and teacher professional development protocol items). The contractor will conduct observations of individual or classroom use of the offering. The likely focus of these observations will be on the nature of interaction between students and teachers, both online and in school, as well as on the ease of use of the offering. Where the offering is not part of a whole-school effort, the contractor may also want to identify teachers of appropriate grade levels and subjects who do not use the offering in order to compare curriculum and instruction in the school with and without the offering.
The contractor will conduct student focus groups to collect data from students about their experiences using the online offering. Researchers will ask about strengths and weaknesses of the offering, additional supports or activities they think would better incorporate the online offering with the course of their education, and perceived attitudes and outcomes associated with the offering. The contractor will also collect the following types of documents from school officials: associated curriculum and support materials, ancillary services, policies, and guidelines for use. The contractor will collect information about student academic outcomes when possible.
Two researchers will conduct each school site visit. In most cases, one site visitor will conduct the focus group or interview while the other site visitor concentrates on recording the interview. (Site visitors will switch roles across respondents to avoid excessive fatigue). Regardless of the form of data collection, an internal data quality review team will check data for completeness and plausibility prior to preparation of the data set for analysis.
Permission and privacy concerns in virtual focus groups will be handled in much the same way as for physical site visits. Participating staff in K-12 sites will receive student consent and parent permission forms for distribution to students and parents. The principal or other site administrator will forward permission slips to parents of students who may participate in virtual focus groups. Signed consent forms will be faxed or emailed back to the contractor. Students, particularly those under the age of eighteen who are participating in virtual focus groups, either via chat or videoconference, will be addressed by first names only.
In cases where researchers are given access to online systems containing identifying and other personal information or discourse (e.g., teacher-developed materials, participant profiles, student work, discussions, and chats), the contractor will work with the provider in advance to identify specific areas, groups, or courses that the researchers will access, and obtain advance consent from instructors and/or learners to enter the designated area to collect data. Typically, gaining consent from instructors and students can be accomplished directly through the system (e.g., by posting a message) or though other pre-established online means of communication (e.g., course e-mail list). This is the online analog to researchers walking into a face-to-face course and observing instruction. Researchers will remove identifying information from discussion transcripts and other materials used to document the case. The contractor will treat log-in names and passwords provided by developers confidentially and will request that any accounts given be canceled at the completion of the data collection.
Project-wide training for site visitors, including role play with the interview and focus group protocols, will be conducted at the contractor’s offices.
The use of secondary data sources will enhance the project’s analyses and avoid duplication efforts. As described earlier, a systematic review of the literature will help focus the project on those conditions and practices most likely to influence academic achievement. In addition, by gathering secondary sources about the online implementations, the contractor can focus interview time on issues not covered in existing documents.
The notification process will include a letter to online developers and program administrators from the U.S. Department of Education. The notification letter will include a description of the study and contact information for one of the contractor’s researchers whom the recipient may contact in case there are questions or concerns. The letter will describe the purpose, objectives, and importance of the study. A researcher will follow up with developers and administrators to schedule appropriate times for site visits.
The contractor will select online implementations based on their exemplary practices associated with online learning - an approach that makes participation attractive to developers and implementation sites. In addition, past experience has shown that working with interviewees to schedule site visits during convenient times facilitates data collection activities. The contractor will make clear in communication with potential respondents that they will do whatever possible to minimize burden associated with collection activities.
To improve the quality of data collection instruments and control burden on respondents, all instruments will be pre-tested. Pre-testing of site visit protocols will begin in advance of the next TWG meeting, scheduled for Winter 2007. Several members of the TWG are themselves involved with online development, and by walking through objectives and topics with them, the contractor will gain additional information about what developers and implementers of online implementations are likely to be able to provide. In addition, the contractor will seek to pilot instructor protocols with individuals who are currently teaching or have taught online.
The protocols will standardize data collection efforts across each site, while still providing flexibility for the site visitor to customize questions to individual sites.
Dr. Marianne Bakia is the Co-Project Director for the study. Her mailing address is SRI International, 1100 Wilson Blvd., Arlington, VA 22209. Dr. Bakia can also be reached at 703-247-8571 or via e-mail at marianne.bakia@sri.com.
Dr. Barbara Means is the Co-Project Director for the study. Her mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Means can also be reached at 650-859-4004 or via e-mail at barbara.means@sri.com.
1 * I*EARN and ePals are examples of online applications which blend traditional academic instruction with cross-cultural interaction to engage students in collaborative project work.
| File Type | application/msword |
| Author | Policy User |
| Last Modified By | DoED |
| File Modified | 2007-10-03 |
| File Created | 2007-10-03 |