The observational study employs an experimental design in which participants are randomly assigned to a treatment group (exposure to food safety messaging) or to a control group (no exposure to food safety messaging). The observational study will be conducted in test kitchen facilities located at North Carolina State University (NCSU); thus, it will be necessary to recruit study participants who live within driving distance of the facility in the Raleigh-Durham area of North Carolina. As described below, convenience sampling will be used with quotas to ensure that study participants reflect the demographic characteristics of the U.S. population. When randomly assigning participants to the treatment and control groups, we will take necessary and reasonable steps to ensure that the demographic characteristics of each group are similar. Because probability-based sampling is not being used, inferences cannot be made to the U.S. population; however, by using an experimental design that is powered to detect change in the desired outcome (the power analysis is described in B.2), we can assess whether the food safety messaging has an impact on the desired outcome.
Respondent Selection Methods
Observation study participants will be recruited from the Raleigh-Durham area of North Carolina using convenience sampling via social media outlets and by sending emails to Expanded Food and Nutrition Education Program (EFNEP) participants to reach low-income consumers. Notices about the study will also be posted in approximately 100 grocery stores and food retailers of various sizes and price points throughout the Raleigh-Durham area of North Carolina. Recruitment materials will direct prospective participants to call or email the study team to be screened for eligibility or to a web link that will host the screening questionnaire (see Appendix E). For participants screened by phone, eligible participants will be invited to participate in the study and an appointment scheduled during the screening call. For participants who complete the web-based screener, eligible participants will be contacted by phone and invited to participate in the study and an appointment scheduled. Appointments will be scheduled during work hours, evenings, and weekends to allow for a broader participant pool.
The respondent universe is English-speaking or Spanish-speaking adults living in the Raleigh-Durham area of North Carolina who have primary responsibility for preparing food in the household and cook meat or poultry at home at least four times a week. Participants must meet specific inclusion and exclusion criteria. The inclusion criteria are as follows:
Age 18 or older
Speak English or Spanish
Have primary responsibility for preparing food in the household
Prepare meals at home at least four times a week
Cook meat or poultry at home in the past 3 months
The exclusion criteria are as follows:
Have ever taken ServSafe training
Have ever been employed as a food worker or manager in a food preparation setting
Are vegetarian or vegan
As part of the screening process, we will also collect information on participant and household demographic characteristics to ensure that the demographics of recruited participants are similar to those of the U.S. population based on Census data. Table B-1 identifies the demographic characteristics and the distribution of these characteristics for the U.S. population. Participants will not be asked specifically about income, but including recruitment from the EFNEP list will help ensure that low-income individuals are included, because there is a maximum income requirement for program eligibility.
Table B-1. Demographic Characteristics for the U.S. Population (2014)
Characteristic |
Response Categories |
Percentage |
Race |
White |
74% |
|
Non-whitea |
26% |
Ethnicity |
Not Hispanic or Latino |
83% |
|
Hispanic or Latino |
17% |
Age |
20–34b |
28% |
|
35–54 |
36% |
|
55+ |
36% |
Education |
Less than high school or high school diploma/GED |
42% |
|
Some college |
29% |
|
Bachelor’s degree |
18% |
|
Graduate or professional degree |
11% |
Household (HH) sizec |
Family HH |
66% |
|
Non-family HH |
34% |
Source: 2010-2014 American Community Survey 5-year estimates, available at https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2014/
aNon-white includes black or African American, American Indian or Alaska Native, Asian, Native Hawaiian and Other Pacific Islander, other races, or 2 or more races.
bAge is reported as ≤ 19; thus, the first age category starts at age 20; however, we will enroll individuals who are 18 or older if they have primary responsibility for preparing meals in the household and meet the other inclusion and exclusion criteria.
cFamily HH includes household with children 18 or younger; married couple family; male householder, no wife; and female householder, no husband. Non-family HH includes household living alone and 65 years or older.
To ensure enough adults with a high school/GED education or less are recruited, we may work with a local market research company in the Raleigh-Durham area to use convenience sampling and the OMB approved screening questionnaire to recruit adults with a high school/GED education or less from their database. The market research company will send a list of eligible individuals to the study team, who will contact eligible participants by phone to invite them to participate in the study and schedule an appointment. The database maintained by the local market research company includes more than 15,000 households and has been developed over time through advertisements, word-of-mouth referrals, outreach to various organizations and industries, and Internet list searches, among other means. The database is constantly refreshed through natural attrition and additions. In addition to contact information, basic socioeconomic characteristics are collected through the registration process and updated periodically, so that the market research firm can easily identify individuals with a high school/GED education or less.
Sample Size and Allocation
The power analysis (described in B.2) indicates a required sample size of 400 participants for each iteration of the observational study, divided equally between a treatment group and a control group, so 200 participants per group. Convenience sampling with set quotas will be used. The expected allocation of the sample is shown in Table B-2. When randomly assigning participants to the treatment and control groups, we will take necessary and reasonable steps to ensure that the demographic characteristics of each group are similar.
Table B-2. Sample Allocation for Each Iteration of the Observational Study
Characteristic |
Response Categories |
Number |
Race |
White |
296 |
|
Non-whitea |
104 |
Ethnicity |
Not Hispanic or Latino |
332 |
|
Hispanic or Latino |
68 |
Age |
20–34b |
112 |
|
35–54 |
144 |
|
55+ |
144 |
Education |
Less than high school or high school diploma/GED |
168 |
|
Some college |
116 |
|
Bachelor’s degree |
72 |
|
Graduate or professional degree |
44 |
Household (HH) sizec |
Family HH |
264 |
|
Non-family HH |
136 |
Source: 2010-2014 American Community Survey 5-year estimates, available at https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2014/
aNon-white includes black or African American, American Indian or Alaska Native, Asian, Native Hawaiian and Other Pacific Islander, other races, or 2 or more races.
bAge is reported as ≤ 19; thus, the first age category starts at age 20; however, we will enroll individuals who are 18 or older if they have primary responsibility for preparing meals in the household and meet the other inclusion and exclusion criteria.
cFamily HH includes household with children 18 or younger; married couple family; male householder, no wife; and female householder, no husband. Non-family HH includes household living alone and 65 years or older.
Note: When randomly assigning participants to the treatment and control groups, we will take steps to ensure that the demographic characteristics of each group are similar.
Response Rate
The expected response rate (show rate) for the observation study is 80% based on the contractor’s experience with similar studies. Thus, the starting sample size for each iteration of the observational study is 500, yielding 400 completed observations (500 x 0.80).
B.2. Procedures for the Collection of Information
This section provides an overview of the study procedures, provides information on the degree of accuracy required for the study, and discusses the estimation procedures. The observational study is not employing statistical methodology for stratification and sample selection. There are no unusual problems requiring specialized sampling procedures. Participants are only being contacted one time, so periodic data collection cycles are not applicable.
Study Procedures
Before the observation and food preparation, the meat/poultry products will be inoculated with a harmless, realistic, and known amount of tracer bacteria or tracer virus, such as bacteriophage MS2. Under video observation, participants will be asked to prepare two recipes: one made with a raw meat product and one made with a ready-to-eat product (recipes and instructions for the first iteration of the study are provided as Appendix L). After receiving the appropriately assigned messaging (the treatment group will receive messaging on food safety specific to the behavior of interest), participants will receive instruction to cook the recipes in the order they would usually do so at home.
The study will be conducted in NCSU test kitchens (located in Raleigh, North Carolina) specifically designed for observation studies. Video recording equipment will be set up to record meal preparation. Trained research staff will conduct the video recording. Following the meal preparation and before clean-up, trained sample collectors will conduct surface swab sampling, and samples will be transported to an NCSU testing laboratory.
Recording of food handling and meal preparation will begin as soon as the participant enters the test kitchen and will end after the participant leaves. Participants’ cleaning and sanitizing of equipment and environment before and after preparation will also be recorded to evaluate intra-meal and inter-meal contamination risks. Following the observation portion of the study, trained sample collectors will take surface swab samples from kitchen surfaces, utensils, food containers, appliance handles, kitchen towels, cutting boards, any devices (e.g., cell phones) that were touched during meal preparation, and the ready-to-eat dish (at least 15 sites in total). The swabs will be delivered to the NCSU testing laboratory and plated to determine presence and concentration of the tracer. The presence of the tracer will indicate that cross-contamination occurred during food preparation. The level of cross-contamination will be compared across the sampling sites to determine the highest risk areas. Kitchen surfaces, appliances, and other potentially contaminated sites will be cleaned and sanitized after each participant to ensure that any bacterial or virus samples collected were from the participants’ behaviors.
Supplementing the observations, post-observation interviews (see Appendix I) will be conducted to provide insight into participants’ views, opinions, and experiences of their preparation practices of these products and to collect information on behaviors that could not be observed (e.g., storage of leftovers or thawing).
Trained coders will use a coding rubric (see Appendix M) to evaluate the video observations based on the four food safety handling behaviors of clean, separate, cook, and chill. This rubric will be used to consistently define when the recommended behavior occurred or when one did not occur when it should have. Additionally, the coders will use notational analysis to assess recorded actions and their frequencies.
Degree of Accuracy Required for the Study
Sample size calculations were conducted to determine the minimum number of participants needed to provide a level of confidence that the experimental component of the observational study is sufficiently powered, meaning that a change of the anticipated size or greater would be interpreted as occurring beyond chance (i.e., statistically significant). By convention, we aim for 80% statistical power and a 95% level of confidence.
The purpose of the observational study is to evaluate the impact of FSIS educational materials on consumers’ demonstrated use of recommended safe food handling practices (clean, separate, cook, and chill). For the initial iteration of the study, the primary outcome of interest is use of a food thermometer to check the doneness of meat and poultry. This is an important but not commonly practiced behavior in American kitchens. Based on recent estimates, it is anticipated that food thermometer use will be observed 5% of the time among the control group participants (Anderson et al., 2004; Phang & Bruhn, 2011; Bruhn, 2014; Mazengia et al., 2015; Scott & Herbold, 2010). Additionally, it is anticipated that the food safety messaging materials will provide medium effects among the treatment group participants. Table B-3 provides potential observed differences between control and treatment groups ranging from 4 to 12 percentage points. It is anticipated that the food safety messaging materials will be sufficient to generate differences in the middle of this range (i.e., the observed difference between the control and treatment groups is 8 percentage points).
Table B-3. Sample Size Requirements for Different Observed Differences between the Control and Treatment Groups
Proper Thermometer Use: Control Group |
Proper Thermometer Use: Treatment Group |
Observed Difference Between Control and Treatment Groups |
Total Sample Size (N) |
5% |
9.0% |
4% |
1,270 |
5% |
11.0% |
6% |
636 |
5% |
13.0% |
8% |
394 |
5% |
15.0% |
10% |
276 |
5% |
17.0% |
12% |
206 |
Accordingly, the proposed sample size of 400 participants (200 per group) takes into consideration several important features of the project: the anticipated base rate for thermometer use and the anticipated distributional characteristics of a dichotomous outcome; what research design is feasible given the logistical constraints of conducting an observational study at one location; and FSIS’s future plans for researching food safety education.
Estimation Procedures
Statistical analysis will be conducted comparing the differences in handling behavior scores between the control and treatment groups for the four food handling behaviors. A comparative analysis will also be conducted on the samples collected from the designated kitchen sites and food samples to determine whether levels of cross-contamination differed between the two groups, as well as to identify the kitchen sites with the highest levels of contamination.
To maximize response rate and address nonresponse, each participant will receive a cash honorarium of $75 and a small gift (food thermometer valued at $5.38) for their participation. In addition, we will send confirmation letters (see Appendix G) with directions and make reminder calls (see Appendix H) to recruited individuals before their scheduled appointment.
On March 3, 2017, NCSU conducted a pilot study with two participants. The purpose of the pilot study was to test all study materials and the time allotted for each observation. Based on the pilot study, we will limit the number of recipes for participants to prepare to two products instead of three, and we will package the inoculated turkey patties so they resemble products available for retail sale. We will also make available additional supplies in the kitchen to make preparation easier for the participants. We also changed introductory text to the observation script to provide more instructional information to participants and added potential points of cross-contamination to the observation rubric as well as the micro-sampling plan. In addition, we revised the post-interview guide to refine several questions and added several questions to collect additional descriptive information.
Sheryl Cates is the RTI Project Director and will manage the study. Dr. Jonathan Blitstein of RTI conducted the power analysis and is providing guidance on statistical aspects of the study. Dr. Benjamin Chapman of NCSU will manage the data collection for the observational study and oversee the analysis. Christopher Bernstein, an FSIS employee, will review the results of the observation study.
References
Anderson, J. B., Shuster, T. A., Hansen, K. E., Levy, A. S., & Volk, A. (2004). A camera’s view of consumer food-handling behaviors. Journal of the American Dietetic Association, 104, 186–191.
Bruhn, C. M. (2014). Chicken preparation in the home: An observational study. Food Protection Trends, 34(5), 318–330.
Mazengia, E., Fisk, C., Liao, G., Huang, H., & Meschke, J. (2015). Direct observational study of the risk of cross-contamination during raw poultry handling: Practices in private homes. Food Protection Trends, 35(1), 8–23.
Phang, H. S., & Bruhn, C. M. (2011). Burger preparation: What consumers say and do in the home. Journal of Food Protection, 74(10), 1708–1716.
Scott, E., & Herbold, N. (2010). An in-home video study and questionnaire survey of food preparation, kitchen sanitation, and hand washing practices. Journal of Environmental Health, 72(10), 8–13.
| File Type | application/msword |
| Author | joconnell |
| Last Modified By | SYSTEM |
| File Modified | 2017-11-09 |
| File Created | 2017-11-09 |