Supporting Statement Part B- EHS-Net 1-23-2012

Supporting Statement Part B

Supporting Statement (Part B: Statistical Methods) of the

Request for OMB Review and Approval of

Environmental Health Specialists Network (EHS-Net) Program Generic Package

REVISION

January, 2012

Laura Brown, Ph.D.

EHS-Net

Centers for Disease Control and Prevention

National Center for Environmental Health

Emergency and Environmental Health Services

Environmental Health Services Branch

4770 Buford Highway, NE F – 60

Atlanta, GA 30341-3724

lrg0@cdc.gov

770-488-4332

Table of Contents

B. Statistical Methods………………………………………………………………………..	3
1. Respondent Universe and Sampling Method…………………………………………	3
2. Procedures for the Collection of Information…………………………………………	5
3. Methods to Maximize Response Rates and Deal with Nonresponse…………………	7
4. Tests of Procedures or Methods to be Undertaken……………………………………	8
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data………………………………………………………………………	8

2. References………………………………………………………………………………..…..8

B. Statistical Methods

EHS-Net is comprised of retail food establishments in selected geographical areas in California, Minnesota, New York City, New York State, Rhode Island and Tennessee.  While the number of areas included in EHS-Net is small, they are demographically diverse and provide good geographical coverage of the U.S. (northeast, mid-west, south, and west).  When the statistical methods outlined here for ensuring a representative sample in one or more EHS-Net data collections are used, the results of collections covered by this generic OMB package can be used to generalize to the population of retail food establishments in given the EHS-Net site(s).   Furthermore, the geographic and demographic variability across these sites suggests that CDC may be able to use data collected from these studies to draw conclusions about relationships that are likely relevant to establishments in other parts of the U.S.

B.1. Respondent Universe and Sampling Methods

Respondent Universe

The respondent universe is all retail food establishments in the EHS-Net sites. Establishment lists will be obtained from the databases maintained by the EHS-Net sites. CDC will use these lists to generate the sampling frame used to draw the establishment sample for each data collection.

Sampling Methods

Sample Size and Power Calculation. For each data collection, we will conduct a sample size and power calculation to estimate the total number of retail food establishments needed to provide the data collection with sufficient power to detect differences in outcomes of interest. For example, in our most recent study, we planned to test for differences in foodborne illness risk factors between establishments with and without certified kitchen managers. Input values for the power calculation were drawn from previous EHS-Net studies. We wished the study to be able to statistically detect at least a 10% reduction in the odds of the presence of a foodborne illness risk factor. That is, the study needed to have enough power to indicate whether an odds ratio of 0.9, or less, was statistically different from an odds ratio of 1, or no difference between groups. Design parameters were set at 5% type I error, 20% type II error, and 80% study power. The total required number of establishments needed to sustain a study power of 80% and to detect an odds ratio of at least 0.9 was 383. With six EHS-Net sites participating, each site needed to enroll 64 establishments. Since this was the minimum number of establishments required, CDC asked that each site enroll 80 establishments in the study. Although these numbers may differ slightly from data collection to data collection, we expect that 80 will be the maximum number of establishments need in any given data collection.

Sampling Design. Each study will utilize a stratified random sampling design where each EHS-Net site serves as its own mutually exclusive stratum. There are two primary reasons for stratifying by EHS-Net site. The first is that food safety regulations vary by jurisdiction. For example, Tennessee state food safety regulations differ from New York state food safety regulations. These regulations can and do greatly influence restaurant food safety practices and policies. Thus, we felt that EHS-Net site/jurisdiction was a critically important factor for stratification. The second reason for stratifying by EHS-Net site is that EHS-Net sites participate in EHS-Net through a cooperative agreement. The nature of this agreement is such that one site cannot be expected to do a disproportionate amount of work in comparison to other sites (because each site receives relatively equal funding amounts). If we did not stratify by EHS-Net site, it is likely that some sites would have to collect data for each study in far more restaurants than other sites would.

However, stratification on any other variables of interest, such as the number of violations the restaurant received on their last inspection, would require larger sample sizes and place an additional burden on the EHS-Net sites. Thus, practical concerns limit our ability to stratify on other variables of interest. However, we will have data on the number of inspection violations and restaurant size for every restaurant in the EHS-Net population; we can use these data to create sampling weights for these variables, which will support some generalizations of findings (see the Sampling Weights section below for more details).

Establishments will be randomly selected, with equal probability, within their respective EHS-Net site, independent of other sites. This process will give each establishment on the list the same probability of being selected for study participation. We will sample without replacement, meaning that establishments will not have the chance to be selected for the study sample more than once. Sampling with replacement is needed for the use of weighted estimates, which we plan to use. However, the use of sampling without replacement is a common and justifiable practice when the target population is considerably larger than the number of samples needed to be drawn from that population, as is the case with these studies. In other words, we will sample without replacement but will use analytic methods that support weighted estimates.

The total target population of establishments from all EHS-Net sites combined constitutes a highly heterogeneous group. To control for such heterogeneity in the total sample, establishments will be stratified by EHS-Net site so they can be grouped into more homogeneous strata and then sampled within stratum independently. This reduction in heterogeneity of the total sample will lead to reduction in sampling error, which can improve representativeness of the selected sample, and provide weighted estimates (e.g., means) that tend to have less variability than estimates produced from a sample that were drawn using the un-stratified, simple random sampling method.

Sampling weights. Each study will be designed to support the generalization of findings from the sampled establishments to all establishments in the EHS-Net sites. To achieve generalizability, sampling weights will be used to indicate the degree of representativeness that each sampled establishment has in relation to the total target population. Representation in the studies covered by this package is based on two distinguishing establishment characteristics: size as indicated by seating capacity and most recent inspection status (number of critical violations found). These data will be obtained from establishment databases maintained by the EHS-Net sites and provided to CDC in their population establishment lists. CDC will use these data to calculate the appropriate sampling weights for the sampled establishments within each EHS-Net site.

Establishments that refuse to participate will be considered the same as participating establishments, unless data indicate otherwise. We will have basic demographic data (size and inspection status) on the establishment population, and we will collect additional demographic information when possible. These data will be used to ascertain whether there are systematic differences between participating establishments and those that refused to participate so that sampling weights can be adjusted accordingly. In instances where there are systematic differences between refused and participating establishments, refused establishments will be excluded from the total target population. All ineligibles will be excluded from the total target population. Careful considerations will be taken in calculating sampling weights as they can impact the overall population parameter estimates.

Response Rate. The two most recent EHS-Net studies that used methods similar to those proposed in this package yielded response rates around 70%. We expect a similar response rate for future studies. Although this response rate is lower than the 80% response rate mentioned in the OMB guidelines, it is a rate that allows us to feel comfortable with analyses based on the data (Frey & Oishi, 1995). Additionally, we will have access to basic demographic data on the retail food establishments in the population. These data will allow us to determine if there are differences between participating and refusing establishments and to possibly adjust for those differences using statistical techniques.

B.2. Procedures for the Collection of Information

Sampling and Recruitment

As indicated earlier, each EHS-Net site will provide CDC with a list of all establishments in the EHS-Net sites. This list will serve as the sampling frame for the site for each data collection. CDC will use a random number generator in SAS 9.2 to produce a random sample of establishments from this list for each site for each study. As we expect some establishments will refuse to participate and some will be ineligible to participate, we will select more than the needed number of establishments--100 for each site for each study.

For each study, personnel in each site will contact establishments by telephone to recruit their participation in the study. If the manager is willing to participate, the EHS-Net specialist will arrange a mutually convenient time to conduct the data collection.

In instances where an EHS-Net site is unable to recruit the needed number of establishments from the first list of 100 establishments, CDC will randomly select another group of 50 establishments for the site to use to recruit additional respondents. Recruitment will be done via the telephone and a log of each incident of contacts with the establishments will be kept in order to document rates and reasons for refusal and/or ineligibility.

Data Collection

Data will be collected by the EHS-Net environmental health specialists. For the worker interview/survey portion of the studies, the EHS-Net specialists will obtain verbal informed consent and then conduct a face-to-face or telephone semi-structured interview, and/or a written survey with one or more establishment workers. The interviews/surveys will include questions on establishment and worker characteristics and food handling and food safety practices. These activities will take about thirty minutes to complete. To increase cooperation, we will allow establishment management to provide feedback on which workers can be interviewed. A criterion for selection will be that the workers can speak English well enough to complete the interview/survey in English.

For the observation portion of the studies, the EHS-Net specialists will observe the kitchen environment and observe food handling practices. This observation will take about an hour to complete. Thus, data collection will take about an hour and a half per establishment. Although this may seem like a relatively long time for data collection, workers will only be engaged with the data collectors for a relatively short time (about 30 minutes) because the kitchen observation does not actively involve them. We have conducted several studies using methods and data collection durations similar to this one, and have had response rates in the 70% range (Delea et al, 2010; Kirkland et al., 2009; Marcus et al, 2010).

Quality Control Procedures

The data collectors are experienced and knowledgeable in environmental health and food safety and will have received training from CDC on data collection for each study. Data entry will be double-checked by the EHS-Net administrator in each EHS-Net site.

Potential Biases

Managers’ concerns about the food safety of their establishments may result in a lower rate of study participation among establishments with worse food safety practices compared to establishments with better food safety practices. In the past, we have conducted studies using methods similar to those proposed in this package, and these studies have found a wide range of food safety practices, including poor ones (Delea et al, 2010; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Marcus et al, 2010). Nevertheless, in an attempt to assess the impact of this non-response bias on our data, we will compare data on inspection status for restaurants that participate in our studies and restaurants that do not participate in our studies. Although this inspection measure is not a perfect measure of food safety, this analysis may give us a better understanding of how food safety practices may differ between participating and non-participating restaurants.

The observation data collected for these studies may be influenced by reactivity on the part of those observed. In other words, those observed may not respond naturally when they know they are being observed. However, observation data on behavior is considered to be more accurate than self-reported data, particularly when measures are taken to limit the observers’ influence on the observed (Leary, 2004). In our studies, those measures will include the following: 1) observers will attempt to remain relatively unobtrusive during the observation, and 2) when possible, the precise details on which aspects of behavior are being recorded will not be provided to those being observed. Additionally, the observation in this study is about 50 minutes, and research suggests that longer observations allow time for the observed to revert to more natural behavior over the course of the observation (Gall, Borg, & Gall, 1996).

The interview data collected for these studies may be influenced by the social desirability bias- the tendency for people to report greater levels of socially desirable behavior (such as safe food preparation practices) than they actually engage in, or to report their best behavior rather than their typical or worst behavior. Although it is difficult to eliminate this bias altogether, it can be limited by ensuring respondents that the information they report will be anonymous, which we will do (Leary, 2004).

The fact that management will help select the workers to be interviewed may also introduce bias, as management may select workers they believe are knowledgeable about food safety. However, we feel this selection technique is necessary to increase participation.

We will only interview workers that speak English well enough to be interviewed in English. The use of this criterion may introduce bias, as non-English speakers may have different food safety knowledge and practices than English speakers, but the resources are not available to include non-English speaking workers in these studies. Currently, one of our EHS-Net sites is conducting a study in which food safety practice data will be collected from both Spanish-only speaking restaurant managers and workers and English-speaking restaurant managers and workers; the results from this study may give us a better understanding of how data from EHS-Net restaurant studies may be impacted by the restriction of participants to English speakers.

Any presentation of data from EHS-Net studies will acknowledge these potential biases and include a discussion of how they may impact data interpretation.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

For each study, we will engage in several activities designed to maximize response rates. First, all recruiters will receive training on the recruiting process. Second, multiple attempts will be made to contact potential respondents. Specifically, recruiters will make 10 attempts over 5 days to get a participation response from establishments they have not been able to contact, and 5 attempts over 5 days to get a participation response from establishments that have not provided a response (e.g., ‘call back later’). Third, recruiting scripts will emphasize two issues that have been shown to increase response rates—the anonymous nature of the data collection and the importance of the respondents’ participation in the study. The most recent EHS-Net data collections used these techniques and yielded response rates around 70% (Kirkland et al., 2009; Sumner et al, 2011).

We will also attempt to determine if restaurants participating in this study differ systematically from non-participating restaurants. To do this, we will compare participating and non-participating restaurants on two important restaurant characteristics- restaurant size and inspection status. If significant differences are found in these characteristics, any presentation of the data from this study will include a discussion of these differences and how they may impact data interpretation.

B.4. Test of Procedures or Methods to be Undertaken

All data collection materials will be reviewed and evaluated by EHS-Net specialists familiar with collecting data in establishments. Additionally, all data collection materials will be evaluated in pretests with 9 establishments. The pretests will be used to improve the data collection materials.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following CDC personnel will be primarily responsible for the design, including the statistical aspects, of the data collections and will be primarily responsible for data analysis.

Laura Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

Lrg0@cdc.gov

770-488-4332

Brenda Le, M.S.

Statistician

Centers for Disease Control and Prevention, National Center for Environmental Health

Cmo9@cdc.gov

770-488-3756

Kristin Delea, MPH

Health Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

kdelea@cdc.gov

770-488-7065

Personnel in the 6 EHS-Net sites will be responsible for data collection. These sites are listed below.

California Department of Health

Minnesota Department of Health

New York Department of Health

New York City Department of Health and Mental Hygiene

Rhode Island Department of Health

Tennessee Department of Health

References

Gall, M., W. Borg, and J. Gall. 1996. Educational Research. Longman. White Plains, NY.

Green, L., C. Selman, V. Radke, D. Ripley, J. Mack, D. Reimann, T. Stigger, M. Motsinger and L. Bushnell. 2006. Food worker hand washing practices: An observation study. J. Food Protect. 69:2417-2423.

Delea, K., K., Everstine and E. Coleman. 2010. Restaurant leafy green handling practices. Manuscript in preparation.

Frey, H., and S. Oishi. 1995. How to conduct interviews by telephone and in person. SAGE Publications. Thousand Oaks, CA.

Marcus, R., and C., Monteilh. 2010. Restaurant chicken handling practices. Manuscript in preparation.

Kirkland, E., L. Green, C. Stone, D. Reimann, D. Nicholas, R. Mason, R. Frick, S. Coleman, L. Bushnell, H. Blade, V. Radke, C. Selman, and the EHS-Net Working Group. 2009. Tomato handling practices in restaurants. J. Food Protect. 72:1692–1698.

Leary, M. 2004. Introduction to behavioral science research methods. Allyn and Bacon. Boston, MA.

Lee, R., M. Beatty, A. Bogard, M. Esko, R. Anglulo, C. Selman, and the EHS-Net Working Group. 2004. Prevalence of high-risk egg-preparation practices in restaurants that prepare breakfast egg entre´es: An EHS-Net study. J. Food Protect. 67:1444–1450.

Sumner, S., L. Brown, R. Frick, C. Stone, L. Carpenter, L. Bushnell, D. Nicholas, J. Mack, H. Blade, M. Tobin-D’Angelo, K. Everstine, and EHS-Net. 2011. Factors associated with food workers working while experiencing vomiting or diarrhea. J. Food Protect. 74:215-220.

9