Building Trust in Decision with Conformalized Multi-view Deep Classification
Authors: 
Wei Liu, Yufei Chen, Xiaodong YueAuthors Info & Claims
Pages 7278 - 7287
Abstract
Uncertainty-aware multi-view deep classification methods have markedly improved the reliability of results amidst the challenges posed by noisy multi-view data, primarily by quantifying the uncertainty of predictions. Despite their efficacy, these methods encounter limitations in real-world applications: 1) They are limited to providing a single class prediction per instance, which can lead to inaccuracies when dealing with samples that are difficult to classify due to inconsistencies across multiple views. 2) While these methods offer a quantification of prediction uncertainty, the magnitude of such uncertainty often varies with different datasets, leading to confusion among decision-makers due to the lack of a standardized measure for uncertainty intensity. To address these issues, we introduce Conformalized Multi-view Deep Classification (CMDC), a novel method that generates set-valued rather than single-valued predictions and integrates uncertain predictions as an explicit class category. Through end-to-end training, CMDC minimizes the size of prediction sets while guaranteeing that the set-valued predictions contain the true label with a user-defined probability, building trust in decision-making. The superiority of CMDC is validated through comprehensive theoretical analysis and empirical experiments on various multi-view datasets.
References
[1]
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. 2021. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, Vol. 76 (2021), 243--297.
[2]
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. 2013. Deep canonical correlation analysis. In International conference on machine learning. PMLR, 1247--1255.
[3]
Anastasios N Angelopoulos and Stephen Bates. 2021. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 (2021).
[4]
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. 2020. Uncertainty Sets for Image Classifiers using Conformal Prediction. In International Conference on Learning Representations.
[5]
Nicholas Bien, Pranav Rajpurkar, Robyn L Ball, Jeremy Irvin, Allison Park, Erik Jones, Michael Bereket, Bhavik N Patel, Kristen W Yeom, Katie Shpanskaya, et al. 2018. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS medicine, Vol. 15, 11 (2018), e1002699.
[6]
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. 2020. Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval. In Computer Vision -- ECCV 2020. Springer International Publishing, 677--694.
[7]
Thierry Denøeux, Zoulficar Younes, and Fahed Abdallah. 2010. Representing uncertainty on set-valued variables using belief functions. Artificial Intelligence, Vol. 174, 7--8 (2010), 479--499.
[8]
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. 2022. Training uncertainty-aware classifiers with conformalized deep learning. Advances in Neural Information Processing Systems, Vol. 35 (2022), 22380--22395.
[9]
Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. IEEE, 178--178.
[10]
Li Fei-Fei and Pietro Perona. 2005. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Vol. 2. IEEE, 524--531.
[11]
Wei Fu, Yufei Chen, Wei Liu, Xiaodong Yue, and Chao Ma. 2023. Evidence Reconciled Neural Network for Out-of-Distribution Detection in Medical Images. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 305--315.
[12]
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. PMLR, 1050--1059.
[13]
Yu Geng, Zongbo Han, Changqing Zhang, and Qinghua Hu. 2021. Uncertainty-Aware Multi-View Representation Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 7545--7553.
[14]
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321--1330.
[15]
Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. 2021. Trusted Multi-View Classification. In International Conference on Learning Representations.
[16]
Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. 2023. Trusted Multi-View Classification With Dynamic Evidential Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 2 (2023), 2551--2566.
[17]
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. 2021. Multi-View Representation Learning via Total Correlation Objective. Advances in Neural Information Processing Systems, Vol. 34 (2021).
[18]
Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2020. Subspace inference for Bayesian deep learning. In Uncertainty in Artificial Intelligence. PMLR, 1169--1179.
[19]
Siddhartha Jain, Ge Liu, Jonas Mueller, and David Gifford. 2020. Maximizing overall diversity for improved uncertainty estimates in deep ensembles. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4264--4271.
[20]
Bingbing Jiang, Chenglong Zhang, Yan Zhong, Yi Liu, Yingwei Zhang, Xingyu Wu, and Weiguo Sheng. 2023. Adaptive collaborative fusion for multi-view semi-supervised classification. Information Fusion, Vol. 96 (2023), 37--50.
[21]
AUDUN. Jøsang. 2018. Subjective Logic: A formalism for reasoning under uncertainty. Springer.
[22]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[23]
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. 2011. HMDB: a large video database for human motion recognition. In 2011 International conference on computer vision. IEEE, 2556--2563.
[24]
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, Vol. 30 (2017).
[25]
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2013. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, Vol. 36, 3 (2013), 453--465.
[26]
Xinyan Liang, Pinhan Fu, Qian Guo, Keyin Zheng, and Yuhua Qian. 2024. DC-NAS: Divide-and-Conquer Neural Architecture Search for Multi-Modal Classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 13754--13762.
[27]
Xinyan Liang, Yuhua Qian, Qian Guo, Honghong Cheng, and Jiye Liang. 2021. AF: An association-based fusion method for multi-modal classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44, 12 (2021), 9236--9254.
[28]
Wei Liu, Yufei Chen, Xiaodong Yue, Changqing Zhang, and Shaorong Xie. 2023. Safe Multi-View Deep Classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 8870--8878.
[29]
Wei Liu, Xiaodong Yue, Yufei Chen, and Thierry Denoeux. 2022. Trusted multi-view deep learning with opinion aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 7585--7593.
[30]
Andrey Malinin and Mark Gales. 2018. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, Vol. 31 (2018).
[31]
Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, Vol. 6. International Conference on Representation Learning.
[32]
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. 2020. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, Vol. 33 (2020), 3581--3591.
[33]
Ruslan Salakhutdinov and Andriy Mnih. 2008. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In Proceedings of the 25th international conference on Machine learning. 880--887.
[34]
Murat cSensoy, L Kaplan, and M Kandemir. 2018. Evidential deep learning to quantify classification uncertainty. Advances in Neural Information Processing Systems (2018).
[35]
Glenn Shafer. 1976. A mathematical theory of evidence. Princeton university press.
[36]
Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. 2023. Improving expert predictions with conformal prediction. In International Conference on Machine Learning. PMLR, 32633--32653.
[37]
David Stutz, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. 2021. Learning Optimal Conformal Classifiers. In International Conference on Learning Representations.
[38]
Shiliang Sun, Wenbo Dong, and Qiuyang Liu. 2020. Multi-view representation learning with deep gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).
[39]
Jakub Swiatkowski, Kevin Roth, Bastiaan Veeling, Linh Tran, Joshua Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. 2020. The k-tied normal distribution: A compact parameterization of Gaussian mean field posteriors in Bayesian neural networks. In International Conference on Machine Learning. PMLR, 9289--9299.
[40]
Martijn van Breukelen, Robert PW Duin, David MJ Tax, and JE Den Hartog. 1998. Handwritten digit recognition by combined classifiers. Kybernetika, Vol. 34, 4 (1998), 381--386.
[41]
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200--2011 dataset. California Institute of Technology (2011).
[42]
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. 2015. On deep multi-view representation learning. In International conference on machine learning. PMLR, 1083--1092.
[43]
Hok Shing Wong, Li Wang, Raymond Chan, and Tieyong Zeng. 2021. Deep tensor CCA for multi-view learning. IEEE Transactions on Big Data, Vol. 8, 6 (2021), 1664--1677.
[44]
Cai Xu, Jiajun Si, Ziyu Guan, Wei Zhao, Yue Wu, and Xiyue Gao. 2024. Reliable conflictive multi-view learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 16129--16137.
[45]
Chang Xu, Dacheng Tao, and Chao Xu. 2013. A survey on multi-view learning. arXiv preprint arXiv:1304.5634 (2013).
[46]
Xiaoyu Yang, Jie Lu, and En Yu. 2024. Adapting Multi-modal Large Language Model to Concept Drift in the Long-tailed Open World. arXiv preprint arXiv:2405.13459 (2024).
[47]
Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. 2020. Deep partial multi-view learning. IEEE transactions on pattern analysis and machine intelligence, Vol. 44, 5 (2020), 2402--2415.
[48]
Liyuan Zhang, Wei Liu, Yufei Chen, and Xiaodong Yue. 2022. Reliable multi-view deep patent classification. Mathematics, Vol. 10, 23 (2022), 4545.
[49]
Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, et al. 2024. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947 (2024).
[50]
Hai Zhou, Zhe Xue, Ying Liu, Boang Li, Junping Du, Meiyu Liang, and Yuankai Qi. 2023. CALM: An Enhanced Encoding and Confidence Evaluating Framework for Trustworthy Multi-view Learning. In Proceedings of the 31st ACM International Conference on Multimedia. 3108--3116.
[51]
Xin Zou, Chang Tang, Xiao Zheng, Zhenglai Li, Xiao He, Shan An, and Xinwang Liu. 2023. DPNET: Dynamic Poly-attention Network for Trustworthy Multi-modal Classification. In Proceedings of the 31st ACM International Conference on Multimedia. 3550--3559.
Index Terms
- Building Trust in Decision with Conformalized Multi-view Deep Classification
Recommendations
New Multi-View Classification Method with Uncertain Data
Multi-view classification aims at designing a multi-view learning strategy to train a classifier from multi-view data, which are easily collected in practice. Most of the existing works focus on multi-view classification by assuming the multi-view data ...
Joint long and short span self-attention network for multi-view classification
AbstractMulti-view classification aims to efficiently utilize information from different views to improve classification performance. In recent researches, many effective multi-view learning methods have been proposed to perform multi-view data analysis. ...
Highlights- A novel end-to-end unified multi-view classification framework is proposed.
- A long and short span self-attention layer is constructed.
- An adaptive weight loss fusion strategy is designed.
- The performance of our method ...
A Novel Algorithm to Multi-view TSK Classification Based on the Dirichlet Distribution
Advanced Intelligent Computing Technology and ApplicationsAbstractWith the help of multi-view classification technology, the classification performance can be effectively improved. However, the traditional multi-view TSK classification method has the problem of dimension explosion when superimposing the features ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 20Total Downloads
- Downloads (Last 12 months)20
- Downloads (Last 6 weeks)20
Reflects downloads up to 13 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in