B19. Meal Observation Pilot Protocol_v2

OMB Number: 0584-0530

Expiration Date: XX/XX/XXXX

Appendix B19. Meal Observation Pilot

Appendix B19. Meal Observation Pilot

Overview of the Pilot Study Design

The pilot study team will conduct meal observations in up to nine schools. Data collectors will be trained to collect meal observation data with the paper booklets utilized in prior APEC studies as well as with cameras. In seven of the nine schools, we will conduct dual-method observations of approximately 1,400 meal trays (the same trays will be captured by both methods) to facilitate a comparison of the data (see Exhibit 1-1). In two of the nine schools, we will use only a camera to capture meal tray components and whether the meal is counted as reimbursable.

Exhibit 1-1. Overview of the study design

Pilot study component	Dual-method observations	Camera-only Observations
Primary goal	Compare the accuracy of the two data collection methods: booklets v. cameras.	Assess the feasibility and costs of collecting meal tray and reimbursability data using cameras.
Data collection	Using paper booklets and a camera, conduct approximately 1,400 meal observations of the same trays. Record relevant contextual information about each meal observation.	Using only cameras, capture data on meal tray components and the corresponding reimbursability entry marked by the cashier. Record relevant contextual information about each meal observation.
Analyses	Compare the two methods with regard to accuracy. Describe how well the camera method worked and any challenges that emerged.	Perform a cost analysis of the camera-only method to the paper-only method utilized in APEC-III. Describe how well the camera method worked and any challenges that emerged. Discuss the potential for scaling up.

Both study components - dual-method and camera-only - are critical to the success of the pilot. The dual-method component allows us to compare meal observations of the same trays under the same conditions and determine which produces more accurate results in terms of identification of meal items. However, the dual-method component also introduces additional elements (e.g., staff, equipment, time) that would not be needed if we were testing the camera method alone. For instance, the dual-method observations require three staff present on site, which is more burdensome than the two staff needed for the camera-only observations. Additionally, the dual-method observations do not allow us to also capture a cashier’s entry of reimbursability. Thus, to truly assess the feasibility of collecting linked meal tray and reimbursability data using only cameras, the camera-only observations are essential. Finally, the camera-only method allows the team to demonstrate to FNS the number of meal trays that could be observed and the additional detail on school meals (e.g., micronutrient content) that can be provided when using cameras for data collection and the Remote Food Photography Method to facilitate coding.

To determine accuracy from the dual-method observations, we will compare each method to a “gold standard” review. Staff will compare a 20 percent sample of meal tray observations recorded using each method against the matching photographs of those trays to determine if all meal items were correctly identified by each method.

For the cost analysis, we will look to the camera-only observations and calculate the costs of labor and materials with regard to data collection and data cleaning (i.e., quality control efforts and coding).¹ These costs will be compared to the costs of the same tasks in APEC-III, in which data collectors used only paper booklets to conduct meal observations. Additionally, if the two methods (camera versus paper booklet) are not found to be equally accurate, cost calculations will be adjusted to reflect staff time needed to correct any meal observations with incorrectly identified items.

Finally, data collectors will complete a Site Visit Summary form for each school meal observation, which collects qualitative information on the setting and circumstances of each meal period observed. Data collectors will record information on any special circumstances that arose (e.g., a fire alarm went off during the meal service) and describe the setting and flow of data collection. For instance, they will indicate whether students had to step out of their usual serving line to have their trays photographed. This information will be analyzed to describe the facilitators and challenges of utilizing the camera method and inform our assessment of the potential for scalability.

Data Collection Procedures

Components of the Pre-visit Questionnaire

Section A: Contact and location information

Section B: Meal service information

• Number of meal periods for breakfast and lunch

• Start and end times of meal periods

• Number and type of meal locations (i.e., classroom, cafeteria, other)

• Whether utilize OVS

• Number of cashier/register lines

• Number of students served per meal period

• Provision of breakfast and lunch menus

Section C: Special instructions for the visit (e.g., COVID-19 precautions)

Pre-visit Procedures

Pre-visit questionnaires will be completed to help prepare the schools and data collectors for the data collection visits. The goal is to minimize the disruption and burden on the schools during the visits. The pre-visit questionnaires, including scheduling of the visits, will be completed by the data collectors during their pre-visit telephone call with the school’s principal/cafeteria manager. Data collectors will use this information to prepare for their visit and schedule their meal observations to conduct the maximum number of tray observations.

To further assist with meal tray data collection, data collectors will also ask for the menus for the date of the scheduled observation. These data will be pre-populated into the paper meal observation booklets to help observers identify food items. The menus will also be used to identify foods during analysis.

On-site Procedures

Preparatory Activities. Data collectors will perform the following on-site preparations for the meal observations:

• Request the updated menu from the cafeteria manager, if applicable.

• Ask cafeteria managers for recipes or nutrition facts labels for the various items being served. For convenience, we will ask to photograph these documents instead of trying to obtain copies.

• Request a list of food items in reserve for the day in case menu items run out and cafeteria staff have to provide alternatives.

• Set up and test camera equipment. (See below for more details.)

Dual-method Observations. For the dual-method observations, three staff will be on site during data collection. Two will conduct meal tray observations (see Exhibit 1), and one person will facilitate the observations in the background as well as collect qualitative information using the Site Visit Summary Form.

Exhibit 1. Illustration of on-site data collection using both methods simultaneously

Data Collector #1: Paper Booklet Observer

During each meal observation, one data collector will be tasked with using the paper meal observation booklet to record the food items on every 6th meal tray.^.2 The booklet will gather information specific to the meal period being observed. It will record the menu items offered to students and any reserve menu items used. For individual trays, the booklet will have space to record the tray number and the items included on the tray by item type:

• For breakfast the primary item types are fruit, grain, and liquid milk (1% fat or fat free)

• For lunch the primary item types are fruit, vegetable, meat or meat alternative, grain, and liquid milk (1% fat or fat free).

Data collectors will be positioned so that they can clearly observe the meal tray without obstructing the flow and progress of students within the meal period.

Data Collector #2: Camera Operator

For each meal observation, one data collector will be assigned to set up the “camera station” with the camera equipment and a placemat upon which students will place their trays for a picture. The data collector will consult with the cafeteria manager on the best place to set up the camera station so that it is positioned in a place that allows for a clear photograph but only minimally interferes with the normal flow of the cafeteria or dining area. Exhibits 2 and 3 illustrate how the equipment will be set up to capture images of meal trays.

Exhibit 2. Graphic representation of camera station (side view)

Exhibit 3. Graphic representation of camera station (top view)

As with the observations recorded in the booklets, the camera operator will record images of every 6^th tray so that the data from the camera can be matched with the data of the same trays recorded in the booklets. The photo images will be taken with little to no interruption to the lunch line.

Data Collector #3: Facilitator

The third staff member will play multiple roles to facilitate data collection. First, this person will ensure that no identifying information (e.g., student ID badge) is present on a tray being photographed. Second, they will place a numbered card on every tray as the student approaches the cashier to indicate to the paper booklet observer and the camera operator which trays they need to record. This ensures that data is collected on the same trays using both methods. To ensure that no student feels singled out, we will place a numbered card on every tray, but will use different colors to indicate which tray is an interval of six and should be recorded. Third, they will watch the trays to ensure that the numbered cards and food items remain visible to the data collectors. For instance, ensuring that a student doesn’t place a notebook or other item over the cards or food. When issues occur, the staff member will quietly ask the student to remove the item from the tray, explaining that we want to take a photo of only the food.

The Facilitator will also be responsible for using the Site Visit Summary Form to describe the data collection setting and note any unusual or special circumstances for each meal period observed. Information about each setting will help us to understand the possibilities for scaling-up the camera method of data collection. For instance, the booklet will include questions such as:

• Was there enough room to set up the camera where you wanted to?

• Did students have to go out of the usual serving line to have their tray photographed?

• Did a new line form as students waited to have their trays photographed? What happened to cause this line to form?

Camera-only Observations. The camera-only observations will largely follow the same procedures described above for the dual-method observations, minus Data Collector #1, the Paper Booklet Observer. Two staff will be on site during data collection for the camera-only observations: Data Collectors #2 and #3, the Camera Operator and Facilitator, respectively.

The Camera Operator and Facilitator will perform roughly the same functions noted above, as well as two additional tasks:

1. Purchase a meal in order to weigh and photograph each food item,³ and

2. Instead of identifying which trays need to be recorded, which is irrelevant during camera-only observations because every tray will be photographed, the Facilitator will flag which meal trays were either served to a non-student (e.g., teacher) or which were marked as “not reimbursable” by the cashier.

Prior to start of a meal period, these two staff will purchase a standard meal tray and will separately weigh and photograph each food item. These food images and weights will be linked to the Food and Nutrient Database for Dietary Studies (FNDDS), other nutrient databases, or a custom recipe, which will allow energy and nutrient intake to be calculated as an additional analysis for the camera-only observations.

Visual Markers Will be Used to Identify the Following:

1. Non-students

2. Student, not reimbursable

3. Student, reimbursable

Additionally, the Facilitator will observe the cashier and place a visual marker, such as a card with a picture of an animal or a brightly-colored post-it note, to identify the meals provided to a non-student (e.g., teacher) and any meals marked as “not reimbursable” by the cashier. That marker will be observable in the images captured of each meal tray, and allow the team to code the meal appropriately. So as not to make students feel singled-out, we will place a marker on every tray prior to taking a photograph, and only the data collectors will know what the markers signify. For instance, we may use different colored post-its to indicate meals provided to non-students and whether a cashier marked a tray as reimbursable or not.

Differences Between the Traditional APEC Meal Observation Methodology and the Pilot Study. Meal observation procedures for the pilot study will closely mirror those from previous APEC studies. Data collectors will conduct observations in the standard cafeteria setting, classroom setting, and for pre-packaged “to-go” meals. When observing meals for the prior APEC studies, data collectors observed the transactions for each sampled tray and recorded the following:

• All food items selected;

• Who the meal was served to (student or adult); and

• Whether the cashier recorded the meal as reimbursable.

These three pieces of information were needed to calculate the school’s incidence of meal claiming error. For this pilot study, however, staff conducting the dual-method observations will record the food items only. For the camera-only observations, we will capture the meal tray components and also record who is served the meal and whether the cashier records the meal as reimbursable. Thus, for the camera-only observations, not the dual-method observations, we will calculate meal claiming errors.

Post-visit Procedures

Paper Booklet Data. Before leaving the meal service location, the data collector will perform a quality check to confirm that all needed information is recorded in the meal observation booklet. Upon completion of each meal observation booklet, the data collector will review the booklet to ensure that the information recorded is complete, accurate, and understandable; abbreviations or short-hand text will be corrected. Upon returning to the home office, data collectors will transfer the data from their meal observation booklet into a database for cleaning and analysis.

Camera Data. The Westat study team will review the camera images to ensure that no identifying information was captured (e.g., pictures of staff or students, ID badges, notebooks with names on them, etc.) If an image includes identifying information of any kind, the study team will black out that portion of the image prior to analysis.

Data Storage and Transfer. After data is collected from each school site, staff will complete the following steps to ensure proper storage and transfer of data.

1. Ensure that all electronic items captured have been saved on the project drive, including:

1. Cafeteria menu, recipes, and ingredient lists (as applicable)

2. Photos and weights of standard food items

3. Test photos of meal trays

4. Meal tray observation images

1. Once all data have been transferred to the drive, back up the camera roll images in a separate location on the secure project drive.

2. Once all data have been saved and backed up, delete the camera roll.

3. 

   1. This information is being collected to provide the Food and Nutrition Service with key information on the annual error rates and improper payments for the school meal programs. This is a voluntary collection and FNS will use the information to examine school meal error rates and inform future APEC studies. This collection requests personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0530. The time required to complete this information collection is estimated to average 1.5 hours (90 minutes) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 1320 Braddock Place, 5th Floor, Alexandria, VA 22306 ATTN: PRA (0584-0530). Do not return the completed form to this address.

   After reviewing camera images for PII, photo data will be sent to the subcontractor for analysis via a secure file transfer protocol.

1 We assume that the other tasks in this pilot study - sampling, recruitment, training, and reporting – would not differ significantly between the two methods, and therefore we will not include those expenses in our analyses. Additionally, we will not include analytic costs, because RFPM allows for a more granular level of analysis than can be accomplished using the data available in paper booklets, making it an unequal comparison.

2 Observing every 6^th tray provides for high quality data collection by allowing data collectors using paper booklets the time needed to focus on specific trays and record their information. Thus, we will observe tray 6, 12, 18, and so forth.

3 Depending on the variety of options, the team may purchase multiple meals in order to weigh and photograph all food items.