SSB_08292023_clean

Evaluating Deep Learning Algorithm Assessment of Digital Photographs for Dental Public Health Surveillance

Supporting Statement B

Program Official/Contact

Susan O Griffin, PhD

Economist, Division of Oral Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

P: 404-422-0867

F: 770-488-6080

Sig1@cdc.gov

8/25/2023

TABLE OF CONTENTS

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 3

B1. Respondent Universe and Sampling Methods 3

B2. Procedures for the Collection of Information 3

B3. Methods to Maximize Response Rates and Deal with No Response 4

B4. Tests of Procedures or Methods to be Undertaken 5

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5

REFERENCES 6

ATTACHMENTS

1. Public Health Service Act [42 U.S.C. 247, 301]

2A. Data collection form part A (Paper clinical dental screening from)

2B. Data collection form part B (Sample photos of teeth)

2C. Screenshot of data collection application

2D. Invitation to schools to participate

2E. Consent form

2F. Notice of screening results

3A. 60-Day Federal Register Notice

3B. 60-day Federal Register Notice public comments and agency response

4. CDC Institutional Review Board determination

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods

The Centers for Disease Control and Prevention (CDC) is examining the validity and feasibility of using digital photos taken by non-dental professionals, which in turn would be analyzed by deep learning algorithms, to assess youth’s oral health status in lieu of assessment by dental professionals. The data collected from this request include examiner’s assessment of youth’s oral health and corresponding photos. The examiner calls and photos will be used to both 1) train the deep learning algorithms to assess caries, sealant presence, and fluorosis severity and 2) test the accuracy of the algorithm assessments of the digital photos compared to the gold standard human examination.

Therefore, the purpose of this data collection is not to obtain a representative sample of a certain population but rather to obtain a sufficient number of photos of caries, sealants, and fluorosis to train the algorithm to identify these conditions. Whereas caries and sealants are common (prevalences of 56.8% and 51.7%, respectively, among adolescents aged 12 to 15 years) (Centers for Disease Control and Prevention 2019); moderate to severe fluorosis affects less than 4% of US adolescents (Beltrán-Aguilar et al. 2010). Fluorosis prevalence can increase with the fluoride content of drinking water. Thus, to have enough photos of fluorosis cases, data will need to be collected in areas served by water systems with higher naturally occurring fluoride (fluoride concentration ≥1.0 parts per million).

Because examiners require extensive training and it is expensive to travel examiners across states, CDC will collect data in only one state that has several public water systems with higher naturally occurring fluoride. Among the 21 state oral health programs funded by CDC, Colorado has the largest population served by public water systems with higher naturally occurring fluoride. Indeed, the first studies to examine the association between water fluoride content and fluorosis severity were conducted in some of these counties (Dean 1990).

The convenience sampling frame will consist of middle schools in counties with at least one water system with higher naturally occurring fluoride. Middle schools were selected because most of their students should have erupted permanent molars (teeth at highest risk for caries and typically receiving sealant) and erupted cuspids (the anterior teeth at highest risk for fluorosis). Data from CDC and the National Center for Education Statistics indicate that there are 25 Colorado counties with at least 1 water system with higher naturally occurring fluoride and at least one middle school. The total number of middle schools in these counties is 120 serving 67,897 students.

CDC is funding the Colorado Department of Public Health and Environment to determine which of these 120 schools are likely to have a large number of students served by public water systems with higher naturally occurring fluoride and from these schools, to collect data for 1,000 students. It is estimated the data will be collected from approximately 50 middle schools.

B2. Procedures for the Collection of Information

CDC is funding the Colorado Department of Public Health and Environment to recruit eligible schools and dental examiners, gain consent, arrange logistics, and collect data consisting of de-identified examination data and photos. CDC is funding a national expert in dental public health surveillance to train the examiners. Finally, CDC is funding researchers at Purdue University to develop photo-taking protocols and deep learning algorithms to identify dental conditions. All examiners, the trainer, and researchers who will analyze individual-level data are being reviewed through Purdue’s Institutional Review Board (IRB). CDC is providing dental examination protocols, review of photo taking protocols, and funding. CDC has met with experts in caries, dental sealant materials, fluorosis, and dental public health surveillance to review protocols for data collection.

Data collected for each student will include 1) human assessment of fluorosis severity in the 6 upper anterior teeth and caries/sealant assessment of the occlusal surfaces of the 8 permanent molars (See Attachment 2A) and 2) 9 digital photos of the upper anterior teeth acquired using an onboard smartphone camera and 24 digital photos of the occlusal surfaces of the 8 permanent molar teeth acquired using an intraoral camera (See Attachment 2B). The intraoral camera will be linked to the smartphone via Wi-Fi. Digital photos of the teeth and the completed paper screening form will be automatically saved and stored to the smartphone via a data collection app (See Attachment 2C). After data have been collected for all students in the school, the data stored on the phone will be uploaded to a HIPAA compliant cloud storage that only can be accessed by examiners and CDC designated researchers. Data stored in this cloud will be secured in accordance with HIPAA. Upon completion of this project, the de-identified data will be stored on a CDC secured server. CDC will retain records in accordance with the applicable CDC records control schedule.

B3. Methods to Maximize Response Rates and Deal with No Response

The Colorado Department of Public Health and Environment has administered school-based oral health screening surveys on an ongoing basis since 2003 and has maintained high response rates, ranging from 68% to 84%. The Colorado state oral health program has built long-standing collaborations with key partners such as the state Department of Education (DOE), school districts and schools and will also implement strategies to enhance buy-in and participation of this collection, such as communicating the importance and benefits of the collection to key stakeholders such as DOE, schools and parents and coordinating with schools to ensure the collection dates and logistics have minimum impact on the regular school schedule. In addition, parents/guardians of children will be provided with the screening results for their child (Attachment 2F), whether child needs to visit the dentist for unmet treatment needs, and a list of dentists who accept Medicaid. 

B4. Tests of Procedures or Methods to be Undertaken

Trained and calibrated dental examiners in this project will use the same diagnostic criteria to identify caries, sealants, and fluorosis as that used by the National Health and Nutrition Examination Survey (NHANES). The NHANES oral health examination protocol has been rigorously tested for validity and reliability and has collected data on these three dental conditions since 1999 (Centers for Disease Control and Prevention 2016; Dye et al. 2019). Colorado examiners will undergo three days of training by a reference examiner prior to data collection.

The photo taking protocols for dental examination closely resemble general photography using smartphone cameras, but they are specifically designed to assist dental examiners and non-dental professionals in obtaining high-quality and standardized photos. The fluorosis assessment protocol enables acquisition of photos of the upper anterior teeth, using an onboard smartphone camera, independent of devices and light conditions. The caries/sealant assessment protocol allows for obtaining detailed photos of the occlusal surfaces of the molars, using an intra-oral camera.

The deep learning algorithms will be developed and trained to detect caries, sealant presence, and the severity of fluorosis in a fully automated and interpretable manner. The algorithms will be carefully designed to minimize bias and account for confounding factors often associated with human examiners. For the fluorosis assessment, the learning algorithm will consist of three modules: color correction, tooth segmentation, and classification. It will be trained to assess the severity of fluorosis in the upper anterior teeth. The caries/sealant assessment algorithm will consist of two modules: tooth segmentation and classification, aimed at detecting caries and sealant presence. To ensure robustness and accuracy, the algorithms will be trained on separate datasets for both training and validation purposes. The performance evaluation of the algorithms will involve comparing outcomes with that of expert human examiners, considered as the gold standard.

CDC consulted with experts in caries, dental sealant materials, fluorosis, and dental public health surveillance to review the data collection procedures.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Using digital images taken under field conditions to assess caries, sealants, and fluorosis is a novel approach. Six national experts in the areas of cariology, fluorosis, and dental public health surveillance have reviewed and provided input on the photo taking protocols.

Table: Individuals Consulted on Data Collection/Statistical Analysis

Name	Title	Contact Information	Role
Dr. Margherita Fontana	Clifford Nelson Endowed Professor and Professor of Dentistry Director, Global Initiatives Program in Oral and Craniofacial Health, University of Michigan	Email: mfontan@umich.edu Phone: (734) 647-1225	Caries and sealant material consultant
Dr. Steven Levy	Wright-Bush-Shreves Endowed Professor of Research, University of Iowa	Email: Steven-levy@uiowa.edu Phone: (319) 335-7185	Caries, sealant material, and fluorosis consultant
Dr. John Warren	Graduate Program Director and Professor of Preventive & Community Dentistry, University of Iowa	Email: john-warren@uiowa.edu Phone: (319) 335-7205	Fluorosis and caries consultant
Dr. Jayanth Kumar	California State Dental Director	Email: Jayanth.Kumar@cdph.ca.gov Phone: (916) 324-1715	Caries, fluorosis, and surveillance consultant
Dr. Mark Moss	Division Director of Public Health Dentistry Associate Professor, Department of Foundational Sciences, Eastern Carolina University	Email: mossm17@ecu.edu Phone: (252) 737-7229	Caries, fluorosis, and surveillance consultant
Dr. Eugenio Beltran	Adjunct Professor Epidemiology and Health Promotion, New York University	E-mail:  eba3@nyu.edu Phone: (212) 998-9800	Will train dental examiners to assess caries, fluorosis, and dental sealants

REFERENCES

Beltrán-Aguilar ED, Barker L, Dye BA. 2010. Prevalence and severity of dental fluorosis in the United States, 1999-2004. NCHS Data Brief. (53):1-8.

Centers for Disease Control and Prevention. 2016. National Health and Nutrition Examination Survey (NHANES) Oral Health Examiners Manual. [accessed July 24, 2023] https://wwwn.cdc.gov/nchs/data/nhanes/2015-2016/manuals/2016_Oral_Health_Examiners_Procedures_Manual.pdf.

Centers for Disease Control and Prevention. 2019. Oral Health Surveillance Report: Trends in Dental Caries and Sealants, Tooth Retention, and Edentulism, United States, 1999–2004 to 2011–2016. [accessed May 5, 2023] https://www.cdc.gov/oralhealth/publications/OHSR-2019-index.html.

Dean HT. 1990. Endemic fluorosis and its relation to dental caries. 1938. Nutrition. 6(6):435-445.

Dye BA, Afful J, Thornton-Evans G, Iafolla T. 2019. Overview and quality assurance for the oral health component of the National Health and Nutrition Examination Survey (NHANES), 2011-2014. BMC Oral Health. 19(1):95.