M - Study Limitations

Attachment M

Study Limitations

Selection and Recruitment Bias: To judge selection bias, the characteristics of the study population will be compared with the national population of the airport baggage handlers. Specifically, data from three groups below will be compared:

• Establishments that choose not to participate (or do not respond to solicitation)

• Establishments who agree to participate but are not chosen for participation

• Establishments who participate

Within establishments, company musculoskeletal disorder (MSD) related injury and cost data will be compared to characterize differences between individuals who participate versus those individuals who choose not to participate or who decide to remove themselves from the study.

Addressing Threats to Internal Validity: The randomized multiple baseline experimental design for this study will address many but not all threats to internal validity inherent in the quasi experimental before/ after studies that have been commonly used in intervention research. Below is a summary how this study will address common threats.

Selection Threat: A selection threat may occur when the apparent intervention effect could be due to other differences between study groups. A randomized design in this study will be used to ensure that the control and intervention groups are similar, especially for variables (such as establishment size and prior loss history) that can affect the measured outcomes.

History Threat: A “history threat” arises when events which are external to the intervention or prior health history of respondents. The external threat may impact outcome, which typically occurs after the intervention is put into place. To minimize the external threat, a number of factors external to the establishment will be tracked during the study periods, such as changes in state or national legislation (especially those that impact OSH record keeping and the Ohio workers compensation system) and the general business practices in the passenger airline industry. In addition, another type of history threat is the respondents’ work and health history. The main advantage of the proposed study design is that it reduces the potential history threat by controlling the history of MSDs and respondents’ work and health history.

Diffusion Threat: A diffusion threat occurs when the intervention provided to one experimental group “diffuses” to the other. This can commonly occur if the intervention study is conducted within the same company (e.g. placing a control in one line but not another). In this study, this threat may not occur because the nature of the interventions. Engineering controls proposed in the study cannot be diffused to other control or intervention groups. However, rivalry or resentment threat may occur. A rivalry threat can occur when participants in the control group react in some way to receiving the intervention. In this study, this threat will be reduced because the interventions will be placed in entirely different areas (i.e., gates that are not close enough to see the effects of interventions).

Instrumentation/Reporting: An instrumentation threat may arise anytime that the methods for measuring outcomes changes during the course of the study. This threat will be reduced in this study by ensuring that the same protocol for outcome measurements (assessed by questionnaires and workers compensation data analysis). For example, for workers compensation data outcomes, it is possible that overall changes in the workers compensation system could effect case definitions (especially for lost time cases). As mentioned in the section on history threats section, a diary of any workers compensation system changes will be kept. Substantial compensation changes require advance notification to stakeholders. It is likely with this advance notification that compensation outcomes can still be calculated the same way even if changes to the overall system occur.

Regression to the Mean: This threat occurs when the intervention involves groups with high injury rates and the observed decrease in rates is the natural tendency of rates to return to an average level rather than an effect of the intervention tested. The randomized baseline design used in this study will address this issue by ensuring that control and intervention groups are similar in pre-intervention injury rates. Regression to the mean will also be addressed by assessing multiple health outcomes (self-repoted MSD symptoms, sickness absence due to MSDs, MSD injury rate in the workers’ compensation system) for two years.

Testing Threat: A testing threat may occur when the measurement process itself affects outcomes. This can arise especially with any measurements that require establishment and employee participation (such as onsite risk factor assessments, questionnaires). In this study, this threat will be reduced because the intervention and control groups will be treated in similar ways. For example, although interventions will not be placed into the groups at the same time, the same outcomes measures, individual questionnaires, and establishment questionnaires will be collected during onsite visits scheduled (pre-intervention, post implementation, post annual, 2^nd post annual).

Placebo (Hawthorne) Effect Threat: Related to the testing threat, the placebo effect occurs when the participants’ attitudes about whether the control will be successful impact the outcome, rather than the impact alone. Another related effect is the Hawthorne effect where the involvement of external researchers impacts outcomes. As described for the testing threat, the placebo or Hawthorne threat will be reduced in this study because the intervention and control groups will be treated in similar ways, except for the timing of the intervention. Participating establishments and individuals will be fully informed (as required by IRB) as to the rationale and scope of the testing well in advance of the actual data collection. It will be assured that the purpose of the study is not for production or compliance reasons, but rather OSH.

Maturation Threat: A maturation threat may arise when the natural changes (e.g. workforce aging, and work experience changes) of the treatment groups may impact outcomes in addition to or rather than the intervention. In this study, this will be addressed by matching the respondents in the intervention group and the control group by their age, gender, job experience and other related factors that may affect the MSD health outcomes.

Dropout Threat: This threat occurs when a sufficient number of individuals drop from treatment groups to alter the group characteristics that may in turn impact outcomes. Common problems are a “survivor effect” where only relatively healthy participants remain in the study or a positive bias when the remaining participants look more favorably upon the intervention than those that drop out. This threat will be addressed in this study by conducting an “early exit” survey to understand the motivations of participant individuals or establishments who decide to remove themselves from the study. As well, establishment level workers compensation data will still be collected for all non-participating establishments, participating establishments, and drop-outs. In the event that an employee drops out of the study or moves to a non-impacted task, a replacement volunteer employee will be recruited. The same baseline questionnaires described above will be administered to all replacement participating employees.

Recall errors: This error may occur when respondents do not accurately remember their work method, occurrences of the MSD symptoms or related health outcome measures in the preceding year. This error is minimized by the monthly questionnaire survey about their work method and the MSD symptoms.