Abstract
Syndromic genetic conditions, in aggregate, affect 8% of the population1. Many syndromes have recognizable facial features2 that are highly informative to clinical geneticists3,4,5. Recent studies show that facial analysis technologies measured up to the capabilities of expert clinicians in syndrome identification6,7,8,9. However, these technologies identified only a few disease phenotypes, limiting their role in clinical settings, where hundreds of diagnoses must be considered. Here we present a facial image analysis framework, DeepGestalt, using computer vision and deep-learning algorithms, that quantifies similarities to hundreds of syndromes. DeepGestalt outperformed clinicians in three initial experiments, two with the goal of distinguishing subjects with a target syndrome from other syndromes, and one of separating different genetic subtypes in Noonan syndrome. On the final experiment reflecting a real clinical setting problem, DeepGestalt achieved 91% top-10 accuracy in identifying the correct syndrome on 502 different images. The model was trained on a dataset of over 17,000 images representing more than 200 syndromes, curated through a community-driven phenotyping platform. DeepGestalt potentially adds considerable value to phenotypic evaluations in clinical genetics, genetic testing, research and precision medicine.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the findings of this study are divided into two groups, published data and restricted data. Published data are available from the reported references and also in Supplementary Table 6. Restricted data are curated from Face2Gene users under a license and cannot be published, to protect patient privacy.
References
Baird, P. A., Anderson, T., Newcombe, H. & Lowry, R. Genetic disorders in children and young adults: a population study. Am. J. Hum. Genet. 42, 677–693 (1988).
Hart, T. & Hart, P. Genetic studies of craniofacial anomalies: clinical implications and applications. Orthod. Craniofac. Res. 12, 212–220 (2009).
Ferry, Q. et al. Diagnostically relevant facial gestalt information from ordinary photos. eLife 3, e02020 (2014).
Basel-Vanagaite, L. et al. Recognition of the Cornelia de Lange syndrome phenotype with facial dysmorphology novel analysis. Clin. Genet. 89, 557–563 (2016).
Rai, M. C. E., Werghi, N., Al Muhairi, H. & Alsafar, H. Using facial images for the diagnosis of genetic syndromes: a survey. In 2015 International Conference on Communications, Signal Processing, and their Applications (ICCSPA) (2015).
Shukla, P., Gupta, T., Saini, A., Singh, P. & Balasubramanian, R. A deep learning frame-work for recognizing developmental disorders. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2017).
Hadj-Rabia, S. et al. Automatic recognition of the XLHED phenotype from facial images. Am. J. Med. Genet. A. 173, 2408–2414 (2017).
Valentine, M. et al. Computer-aided recognition of facial attributes for Fetal Alcohol Spectrum disorders. Pediatrics 140, e20162028 (2017).
Gripp, K. W., Baker, L., Telegrafi, A. & Monaghan, K. G. The role of objective facial analysis using FDNA in making diagnoses following whole exome analysis. Report of two patients with mutations in the BAF complex genes. Am. J. Med. Genet. A. 170, 1754–1762 (2016).
Delgadillo, V., Maria del Mar, O., Gort, L., Coll, M. J. & Pineda, M. Natural history of Sanfilippo syndrome in Spain. Orphanet J. Rare Dis. 8, 189 (2013).
Kole, A. et al. The Voice of 12,000 Patients: experiences and expectations of rare disease patients on diagnosis and care in Europe. Eurordis http://www.eurordis.org/IMG/pdf/voice_12000_patients/EURORDISCARE_FULLBOOKr.pdf (2009).
Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014 1701–1708 (IEEE, 2014).
Huang, G. B., Ramesh, M., Berg, T. & Learned-Miller, E. Labeled faces in the Wild: a database for studying face recognition in unconstrained environments. In Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition (2008).
Yi, D., Lei, Z., Liao, S. & Li, S. Z. Learning face representation from scratch. Preprint at https://arxiv.org/abs/1411.7923 (2014).
Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 815–823 (IEEE,2015).
Li, H., Lin, Z., Shen, X., Brandt, J. & Hua, G. A convolutional neural network cascade for face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 5325–5334 (IEEE, 2015).
Karlinsky, L. & Ullman, S. Using linking features in learning non-parametric part models. Computer Vision–ECCV 2012, 326–339 (2012).
Rohatgi, S. et al. Facial diagnosis of mild and variant CdLS: Insights from a dysmorphologist survey. Am. J. Med. Genet. A. 152, 1641–1653 (2010).
Bird, L. M., Tan, W. H. & Wolf, L. The role of computer-aided facial recognition technology in accelerating the identification of Angelman syndrome. In 35th Annual David W Smith Workshop (2014).
Allanson, J. E. et al. The face of Noonan syndrome: does phenotype predict genotype. Am. J. Med. Genet. A. 152, 1960–1966 (2010).
Gulec, E. Y., Ocak, Z., Candan, S., Ataman, E. & Yarar, C. Novel mutations in PTPN11 gene in two girls with Noonan syndrome phenotype. Int. J. Cardiol. 186, 13–15 (2015).
Zenker, M. et al. SOS1 is the second most common Noonan gene but plays no major role in cardio-facio-cutaneous syndrome. J. Med. Genet. 44, 651–656 (2007).
Rusu, C., Idriceanu, J., Bodescu, I., Anton, M. & Vulpoi, C. Genotype-phenotype correlations in Noonan Syndrome. Acta Endocrinologica 10, 463–476 (2014).
Cavé, H. et al. Mutations in RIT1 cause Noonan syndrome with possible juvenile myelomonocytic leukemia but are not involved in acute lymphoblastic leukemia. Eur. J. Hum. Genet. 24, 1124–1131 (2016).
Kouz, K. et al. Genotype and phenotype in patients with Noonan syndrome and a RIT1 mutation. Genet. Med. 18, 1226–1234 (2016).
Winter, R. M. & Baraitser The London Dysmorphology Database. J. Med. Genet. 24, 509–510 (1987).
Robinson, P. N. & Mundlos, S. The human phenotype ontology. Clin. Genet. 77, 525–534 (2010).
Köhler, S. et al. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. A. J. Hum. Genet. 85, 457–464 (2009).
Zarate, Y. A. et al. Natural history and genotype-phenotype correlations in 72 individuals with SATB2-associated syndrome. Am. J. Med. Genet. A. 176, 925–935 (2018).
Liehr, T. et al. Next generation phenotyping in Emanuel and Pallister Killian Syndrome using computer-aided facial dysmorphology analysis of 2D photos. Clin. Genet. 93, 378–381 (2017).
Hennekam, R. & Biesecker, L. G. Next-generation sequencing demands next-generation phenotyping. Hum. Mutat. 33, 884–886 (2012).
Huang, G., Mattar, M., Lee, H. & Learned-Miller, E. G. Learning to align from scratch. In Advances in Neural Information Processing Systems 2012 764–772 (2012).
Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 2014 3320–3328 (2014).
Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Web-scale training for face identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 2746–2754 (IEEE, 2015).
Zhou, E., Cao, Z. & Yin, Q. Naive-deep face recognition: touching the limit of LFW benchmark or not? Preprint at https://arxiv.org/abs/1501.04690 (2015).
Liu, J., Deng, Y., Bai, T., Wei, Z. & Huang, C. Targeting ultimate accuracy: face recognition via deep embedding. Preprint at https://arxiv.org/abs/1506.07310 (2015).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).
Parkhi, O. M., Vedaldi, A. & Zisserman, A. Deep face recognition. In Proceedings of the British Machine Vision Conference 1, 6 (2015).
Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. International Conference on Machine Learning 2015 448–456 (2015).
Chollet, F. et al. Keras. http://keras.io (2015).
Abadi, M. et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2016).
He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proc. IEEE International Conference on Computer Vision 1026–1034 (IEEE, 2015).
Kingma, D. and Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international Conference on Artificial Intelligence and Statistics 249–256 (2010).
Efron, B. Bootstrap methods: another look at the jackknife. In Breakthroughs in Statistics 569–593 (Springer, New York, 1992).
Acknowledgements
The authors thank the patients and their families, as well as Face2Gene users worldwide who contribute with their knowledge and dedication to the improvement of this and other tools for the ultimate benefit of better healthcare.
Author information
Authors and Affiliations
Contributions
Y.G., Y.H. and O.B. initiated the project. Y.G., Y.H. and D.G. developed the DeepGestalt framework. N.F., L.B.-S., P.M.K., S.B.K., M.Z., L.M.B. and K.W.G. designed and conducted the clinical experiments. O.B. and G.N. finalized the dataset, computed statistics and contributed to the software engineering. Y.G., Y.H., O.B., P.M.K. and K.W.G. contributed to the writing of the manuscript.
Corresponding author
Ethics declarations
Competing interests
Y.G., Y.H., O.B., G.N., N.F. and D.G. are employees of FDNA; L.B.-S., P.M.K. and K.W.G. are advisors of FDNA; L.B.-S., P.M.K., M.Z., L.M.B. and K.W.G. are members of the scientific advisory board of FDNA.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1 and 2 and Supplementary Tables 1–6
Rights and permissions
About this article
Cite this article
Gurovich, Y., Hanani, Y., Bar, O. et al. Identifying facial phenotypes of genetic disorders using deep learning. Nat Med 25, 60–64 (2019). https://doi.org/10.1038/s41591-018-0279-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41591-018-0279-0
This article is cited by
-
Artificial intelligence empowering rare diseases: a bibliometric perspective over the last two decades
Orphanet Journal of Rare Diseases (2024)
-
Machine learning models including patient-reported outcome data in oncology: a systematic literature review and analysis of their reporting quality
Journal of Patient-Reported Outcomes (2024)
-
Assessing the diagnostic utility of the Gaucher Earlier Diagnosis Consensus (GED-C) scoring system using real-world data
Orphanet Journal of Rare Diseases (2024)
-
Computer-based facial recognition as an assisting diagnostic tool to identify children with Noonan syndrome
BMC Pediatrics (2024)
-
Computer-aided diagnostic screen for Congenital Central Hypoventilation Syndrome with facial phenotype
Pediatric Research (2024)