research-article

Open access

Muti-Stage Hierarchical Food Classification

Authors:

Fengqing ZhuAuthors Info & Claims

MADiMa '23: Proceedings of the 8th International Workshop on Multimedia Assisted Dietary Management

Pages 79 - 87

https://doi.org/10.1145/3607828.3617798

Published: 29 October 2023 Publication History

Abstract

Food image classification serves as a fundamental and critical step in image-based dietary assessment, facilitating nutrient intake analysis from captured food images. However, existing works in food classification predominantly focuses on predicting 'food types', which do not contain direct nutritional composition information.This limitation arises from the inherent discrepancies in nutrition databases, which are tasked with associating each 'food item' with its respective information. Therefore, in this work we aim to classify food items to align with nutrition database. To this end, we first introduce VFN-nutrient dataset by annotating each food image in VFN with a food item that includes nutritional composition information. Such annotation of food items, being more discriminative than food types, creates a hierarchical structure within the dataset. However, since the food item annotations are solely based on nutritional composition information, they do not always show visual relations with each other, which poses significant challenges when applying deep learning-based techniques for classification.To address this issue, we then propose a multi-stage hierarchical framework for food item classification by iteratively clustering and merging food items during the training process, which allows the deep model to extract image features that are discriminative across labels. Our method is evaluated on VFN-nutrient dataset and achieve promising results compared with existing work in terms of both food type and food item classification.

References

[1]

Saad Albawi, Tareq Abed Mohammed, and Saad Al-Zawi. 2017. Understanding of a convolutional neural network. Proceedings of International Conference on Engineering and Technology (2017), 1--6.

[2]

Dario Allegra, Sebastiano Battiato, Alessandro Ortis, Salvatore Urso, and Riccardo Polosa. 2020. A review on food recognition technology for health applications. Health psychology research (2020).

[3]

Guangzhou An, Masahiro Akiba, Kazuko Omodaka, Toru Nakazawa, and Hideo Yokota. 2021. Hierarchical deep learning models using transfer learning for disease detection and classification based on small number of medical images. Proceedings of the International Conference on Pattern Recognition Workshop (2021), 571--598.

[4]

Berker Arslan, Sefer Memi, Elena Battini Sönmez, and Okan Zafer Batur. 2022. Fine-Grained Food Classification Methods on the UEC FOOD-100 Database. IEEE Transactions on Artificial Intelligence, Vol. 3, 2 (2022), 238--243.

[5]

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 -- Mining Discriminative Components with Random Forests. Proceedings of European Conference on Computer Vision (2014).

[6]

C. J. Boushey, M. Spoden, F. M. Zhu, E. J. Delp, and D. A. Kerr. 2017. New mobile methods for dietary assessment: review of image-assisted and image-based dietary assessment methods. Proceedings of the Nutrition Society, Vol. 76, 3 (2017), 283--294.

[7]

Steven Coughlin, Mary Whitehead, Jeff Mastromonico, Dale Hardy, and Selina Smith. 2015. Smartphone Applications for Promoting Healthy Diet and Nutrition: A Literature Review. Jacobs journal of food and nutrition (2015).

[8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2009), 248--255.

[9]

Brendan Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science (2007).

[10]

Wenjin Fu, Yue Han, Jiangpeng He, Sriram Baireddy, Mridul Gupta, and Fengqing Zhu. 2023. Conditional synthetic food image generation. Proceedings of the IS&T International Symposium on Electronic Imaging, Vol. 35 (January 2023), 1--6. San Francisco, CA.

[11]

Jixiang Gao, Jingjing Chen, Huazhu Fu, and Yu-Gang Jiang. 2022. Dynamic Mixup for Multi-Label Long-Tailed Food Ingredient Recognition. IEEE Transactions on Multimedia (2022).

Digital Library

[12]

Yanming Guo, Yu Liu, Erwin M. Bakker, Yuanhao Guo, and Michael S. Lew. 2018. CNN-RNN: a large-scale hierarchical image classification framework. Multimedia Tools and Applications (2018).

[13]

Jiangpeng He, Luotao Lin, Heather A. Eicher-Miller, and Fengqing Zhu. 2023 a. Long-Tailed Food Classification. Nutrients, Vol. 15, 12 (2023). https://doi.org/10.3390/nu15122751

[14]

Jiangpeng He, Luotao Lin, Jack Ma, Heather A. Eicher-Miller, and Fengqing Zhu. 2023 b. Long-Tailed Continual Learning For Visual Food Recognition. arXiv preprint arXiv:2307.00183 (2023).

[15]

Jiangpeng He, Runyu Mao, Zeman Shao, Janine L Wright, Deborah A Kerr, Carol J Boushey, and Fengqing Zhu. 2021. An end-to-end food image analysis system. Electronic Imaging, Vol. 2021, 8 (2021), 285--1.

[16]

Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. 2020a. Incremental Learning In Online Scenario. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020), 13926--13935.

[17]

Jiangpeng He, Zeman Shao, Janine Wright, Deborah Kerr, Carol Boushey, and Fengqing Zhu. 2020b. Multi-task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation. Proceedings of IEEE Conference on Multimedia Information Processing and Retrieval (2020), 49--54.

[18]

Jiangpeng He and Fengqing Zhu. 2021. Online Continual Learning For Visual Food Classification. Proceedings of IEEE/CVF International Conference on Computer Vision Workshops (2021), 2337--2346.

[19]

Jiangpeng He and Fengqing Zhu. 2022. Exemplar-Free Online Continual Learning. 2022 IEEE International Conference on Image Processing (2022), 541--545. https://doi.org/10.1109/ICIP46576.2022.9897554

[20]

Jiangpeng He and Fengqing Zhu. 2023. Single-Stage Heavy-Tailed Food Classification. arXiv preprint arXiv:2307.00182 (2023).

[21]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016).

[22]

Long He and Dandan Song. 2022. Hierarchical image classification with a literally toy dataset. Proceedings of International Conference on Mechanisms and Robotics, Vol. 12331 (2022), 1101--1110.

[23]

Shota Horiguchi, Sosuke Amano, Makoto Ogawa, and Kiyoharu Aizawa. 2018. Personalized Classifier for Food Image Recognition. Proceedings of IEEE Transactions on Multimedia, Vol. 20 (2018), 2836--2848.

[24]

Shuqiang Jiang, Weiqing Min, Yongqiang Lyu, and Linhu Liu. 2020. Few-Shot Food Recognition via Multi-View Representation Learning. ACM Trans. Multimedia Comput. Commun. Appl., Vol. 16, 3 (2020).

Digital Library

[25]

Yoshiyuki Kawano and Keiji Yanai. 2014. Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation. Proceedings of European Conference on Computer Vision (2014).

[26]

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. Proceedings of International Conference on Learning Representations (2017).

[27]

Brendan Kolisnik, Isaac Hogan, and Farhana Zulkernine. 2021. Condition-CNN: A hierarchical multi-label fashion image classification model. Expert Systems with Applications, Vol. 182 (2021), 115195.

Digital Library

[28]

Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. University of Toronto (2009). Toronto, Ontario.

[29]

Luotao Lin, Jiangpeng He, Fengqing Zhu, Edward J. Delp, and Heather A. Eicher-Miller. 2023. Integration of USDA Food Classification System and Food Composition Database for Image-Based Dietary Assessment among Individuals Using Insulin. Nutrients, Vol. 15, 14 (2023).

[30]

Chengxu Liu, Yuanzhi Liang, Yao Xue, Xueming Qian, and Jianlong Fu. 2021. Food and Ingredient Joint Learning for Fine-Grained Recognition. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, 6 (2021), 2480--2493.

[31]

Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. Proceedings of International Conference on Learning Representations (2017).

[32]

Runyu Mao, Jiangpeng He, Luotao Lin, Zeman Shao, Heather A. Eicher-Miller, and Fengqing Maggie Zhu. 2021a. Improving Dietary Assessment Via Integrated Hierarchy Food Classification. Proceedings of the IEEE International Workshop on Multimedia Signal Processing (2021), 1--6.

[33]

Runyu Mao, Jiangpeng He, Zeman Shao, Sri Kalyan Yarlagadda, and Fengqing Zhu. 2021b. Visual Aware Hierarchy Based Food Recognition. Proceedings of the International Conference on Pattern Recognition Workshop (2021), 571--598.

Digital Library

[34]

Weiqing Min, Linhu Liu, Zhiling Wang, Zhengdong Luo, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. 2020. ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Network. Proceedings of ACM International Conference on Multimedia (2020).

Digital Library

[35]

Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. 2023. Large Scale Visual Food Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023), 9932--9949.

Digital Library

[36]

Duc Thanh Nguyen, Zhimin Zong, Philip O. Ogunbona, Yasmine Probst, and Wanqing Li. 2014. Food image classification using local appearance and global structural information. Neurocomputing, Vol. 140 (2014), 242--251.

[37]

Xinyue Pan, Jiangpeng He, Andrew Peng, and Fengqing Zhu. 2022. Simulating Personal Food Consumption Patterns Using a Modified Markov Chain. Proceedings of 7th International Workshop on Multimedia Assisted Dietary Management (2022), 61--69. https://doi.org/10.1145/3552484.3555747

Digital Library

[38]

Andrew Peng, Jiangpeng He, and Fengqing Zhu. 2023. Self-Supervised Visual Representation Learning on Food Images. arXiv preprint arXiv:2303.09046 (2023).

[39]

Siddeshwar Raghavan, Jiangpeng He, and Fengqing Zhu. 2023. Online Class-Incremental Learning For Real-World Food Classification. arXiv preprint arXiv:2301.05246 (2023).

[40]

Javier Ródenas, Bhalaji Nagarajan, Marc Bola nos, and Petia Radeva. 2022. Learning Multi-Subset of Classes for Fine-Grained Food Recognition. Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management (2022), 17--26.

Digital Library

[41]

Sabiha Samad, Fahmida Ahmed, Samsun Naher, Muhammad Ashad Kabir, Anik Das, Sumaiya Amin, and Sheikh Mohammed Shariful Islam. 2022. Smartphone apps for tracking food consumption and recommendations: Evaluating artificial intelligence-based functionalities, features and quality of current apps. Intelligent Systems with Applications, Vol. 15 (2022), 200103.

[42]

Ketan Rajshekhar Shahapure and Charles Nicholas. 2020. Cluster Quality Analysis Using Silhouette Score. 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (2020).

[43]

Zeman Shao, Shaobo Fang, Runyu Mao, Jiangpeng He, Janine L. Wright, Deborah A. Kerr, Carol J. Boushey, and Fengqing Zhu. 2021a. Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation. 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP) (2021), 1--6. https://doi.org/10.1109/MMSP53017.2021.9733557

[44]

Zeman Shao, Yue Han, Jiangpeng He, Runyu Mao, Janine Wright, Deborah Kerr, Carol Jo Boushey, and Fengqing Zhu. 2021b. An Integrated System for Mobile Image-Based Dietary Assessment. Proceedings of the 3rd Workshop on AIxFood (2021), 19--23. https://doi.org/10.1145/3475725.3483625

Digital Library

[45]

Zeman Shao, Gautham Vinod, Jiangpeng He, and Fengqing Zhu. 2023. An End-to-end Food Portion Estimation Framework Based on Shape Reconstruction from Monocular Image. arXiv preprint arXiv:2308.01810 (2023).

[46]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of International Conference on Learning Representations (2015).

[47]

Akhilesh Kumar Singh, Shantanu Mittal, Prashant Malhotra, and Yash Vardhan Srivastava. 2020. Clustering Evaluation by Davies-Bouldin Index(DBI) in Cereal data using K-Means. 2020 Fourth International Conference on Computing Methodologies and Communication (2020).

[48]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2015).

[49]

Yanqi Wu, Xue Song, and Jingjing Chen. 2022. Few-Shot Food Recognition with Pre-Trained Model. Proceedings of the 1st International Workshop on Multimedia for Cooking, Eating, and Related APPlications, 45--48.

Digital Library

[50]

Sergey Zagoruyko and Nikos Komodakis. 2016. Wide Residual Networks. Proceedings of the British Machine Vision Conference (2016).

[51]

Yu Zhou, Xiaoni Li, Yucan Zhou, Yu Wang, Qinghua Hu, and Weiping Wang. 2022. Deep collaborative multi-task network: A human decision process inspired model for hierarchical image classification. Pattern Recognition, Vol. 124 (2022), 108449.

Digital Library

[52]

Abdulkadir ?engür, Yaman Akbulut, and Ümit Budak. 2019. Food Image Classification with Deep Features. 2019 International Artificial Intelligence and Data Processing Symposium (2019), 1--6. io

Cited By

Pan XHe JZhu F(2024)FMiFood: Multi-Modal Contrastive Learning for Food Image Classification2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP61759.2024.10743395(1-6)Online publication date: 2-Oct-2024
https://doi.org/10.1109/MMSP61759.2024.10743395
R MKomala Devi K(2024)Food Classification by extracting the important features using VGGNet based Models in Precision Agriculture2024 2nd International Conference on Networking and Communications (ICNWC)10.1109/ICNWC60771.2024.10537499(1-7)Online publication date: 2-Apr-2024
https://doi.org/10.1109/ICNWC60771.2024.10537499
Ataguba GRiyadh MOkaiyeto SDaniel JXiao HOrji R(2024)Ethical Practices for Collecting Ground-Truth Food Datasets: A Systematic Review2024 IEEE Conference on Artificial Intelligence (CAI)10.1109/CAI59869.2024.00105(530-536)Online publication date: 25-Jun-2024
https://doi.org/10.1109/CAI59869.2024.00105
Show More Cited By

Index Terms

Muti-Stage Hierarchical Food Classification
1. Applied computing
  1. Life and medical sciences
    1. Health informatics
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object recognition
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Visual Aware Hierarchy Based Food Recognition
Pattern Recognition. ICPR International Workshops and Challenges
Abstract
Food recognition is one of the most important components in image-based dietary assessment. However, due to the different complexity level of food images and inter-class similarity of food categories, it is challenging for an image-based food ...
A Large-Scale Benchmark for Food Image Segmentation
MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Food image segmentation is a critical and indispensible task for developing health-related applications such as estimating food calories and nutrients. Existing food image segmentation models are underperforming due to two reasons: (1) there is a lack ...
Food vs Non-Food Classification
MADiMa '16: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management

Automatic understanding of food is an important research challenge. Food recognition engines can provide a valid aid for automatically monitoring the patient's diet and food-intake habits directly from images acquired using mobile or wearable cameras. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MADiMa '23: Proceedings of the 8th International Workshop on Multimedia Assisted Dietary Management

October 2023

94 pages

ISBN:9798400702846

DOI:10.1145/3607828

General Chairs:
Stavroula Mougiakakou
University of Bern, Switzerland
,
Keiji Yanai
The University of Electro-Communications, Tokyo, Japan
,
Dario Allegra
University of Catania, Italy

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 16 of 24 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
322
Total Downloads

Downloads (Last 12 months)255
Downloads (Last 6 weeks)21

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pan XHe JZhu F(2024)FMiFood: Multi-Modal Contrastive Learning for Food Image Classification2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP61759.2024.10743395(1-6)Online publication date: 2-Oct-2024
https://doi.org/10.1109/MMSP61759.2024.10743395
R MKomala Devi K(2024)Food Classification by extracting the important features using VGGNet based Models in Precision Agriculture2024 2nd International Conference on Networking and Communications (ICNWC)10.1109/ICNWC60771.2024.10537499(1-7)Online publication date: 2-Apr-2024
https://doi.org/10.1109/ICNWC60771.2024.10537499
Ataguba GRiyadh MOkaiyeto SDaniel JXiao HOrji R(2024)Ethical Practices for Collecting Ground-Truth Food Datasets: A Systematic Review2024 IEEE Conference on Artificial Intelligence (CAI)10.1109/CAI59869.2024.00105(530-536)Online publication date: 25-Jun-2024
https://doi.org/10.1109/CAI59869.2024.00105
Pan XHe JZhu F(2023)Personalized Food Image Classification: Benchmark Datasets and New Baseline2023 57th Asilomar Conference on Signals, Systems, and Computers10.1109/IEEECONF59524.2023.10476964(1095-1099)Online publication date: 29-Oct-2023
https://doi.org/10.1109/IEEECONF59524.2023.10476964

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents