Meta Analysis Protocol

APPENDIX G: RESEARCH ADEQUACY PROTOCOL FOR

META-ANALYSIS

To be included in the research synthesis, each study has to exhibit Content Relevance as indicated by:

1. Involving learning that takes place over the Internet. The use of the Internet must be a major part of the intervention. A study in which the Internet is only an incidental component of the intervention will be excluded. In operational terms, to qualify as “online learning,” a condition will need to be designed to have at least a quarter of the instruction / learning on the content assessed by the study’s learning measure conducted via the Internet.

2. Describing an intervention study that has been completed. Descriptions of study designs, evaluation plans, or theoretical frameworks will be excluded. The length of the intervention / treatment can vary from a few hours for a workshop / curriculum module to a course or program extending over a quarter, semester, year, or longer.

3. Reporting a learning outcome measured for both experimental and control groups. The outcome needs to be measured in the same way across study conditions (i.e., if the study explicitly indicated that different exams were used for the experimental and control groups, the study will be excluded.) The measure must be objective and direct; self report of learning by learners or by teacher / instructor will not be considered a direct measure. Examples of learning outcome measures include scores on standardized tests; scores on researcher-created assessments; grades / scores on teacher-created assessments (e.g., assignments, midterm / final exams); grades or grade point averages. Examples of learning outcome measures for teachers (in addition to student outcomes) include assessments of content knowledge; analysis of lesson plans or other materials related to the intervention; observation (or logs) of class activities; analysis of portfolios; or supervisor’s rating on job performance. Studies that use only non-learning outcome measures (e.g., attitude, retention, attendance, course completion, level of learner / instructor satisfaction) will be excluded.

4. Specifying the level of the learners. All levels of learners from kindergarteners to adults, whether involved in formal / informal schooling or professional training (e.g. in business or military) will be included, but studies that do not report baseline academic achievement or learning characteristics of the online and control group learners will be excluded.

To be included, studies must also meet basic Quality (method) criteria by:

5. Using a controlled design (experimental or quasi-experimental) to empirically compare learning outcomes for conditions falling into at least one of our three categories. ( Category 1 = online learning compared to off-line/face-to-face learning; Category 2 = combination of online plus off-line / face-to-face learning (i.e., blended learning) compared to off-line / face-to-face learning alone; Category 3 = online learning practice A versus online learning practice B). Excluded are: design studies, exploratory studies, or case studies that do not use a controlled research design. To be included, a quasi-experiment must include a pre-measure of learning or achievement for both the experimental and control groups. At a minimum, pre-existing differences known to be related to the outcome measure must be controlled statistically.

6. Reporting an effect size or sufficient data for effect size calculations.

Other criteria:

7. Published from 1996 to present or, for dissertation, completed in the last two years (so that it can be purchased online through UMI ProQuest Digital Dissertations).

8. Publicly available or archived.

2