On the Practicality of Deterministic Epistemic Uncertainty

Janis Postels, Mattia Segù, Tao Sun, Luca Daniel Sieber, Luc Van Gool, Fisher Yu, Federico Tombari
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:17870-17909, 2022.

Abstract

A set of novel approaches for estimating epistemic uncertainty in deep neural networks with a single forward pass has recently emerged as a valid alternative to Bayesian Neural Networks. On the premise of informative representations, these deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution (OOD) data while adding negligible computational costs at inference time. However, it remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications - both prerequisites for their practical deployment. To this end, we first provide a taxonomy of DUMs, and evaluate their calibration under continuous distributional shifts. Then, we extend them to semantic segmentation. We find that, while DUMs scale to realistic vision tasks and perform well on OOD detection, the practicality of current methods is undermined by poor calibration under distributional shifts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-postels22a, title = {On the Practicality of Deterministic Epistemic Uncertainty}, author = {Postels, Janis and Seg{\`u}, Mattia and Sun, Tao and Sieber, Luca Daniel and Van Gool, Luc and Yu, Fisher and Tombari, Federico}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {17870--17909}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/postels22a/postels22a.pdf}, url = {https://proceedings.mlr.press/v162/postels22a.html}, abstract = {A set of novel approaches for estimating epistemic uncertainty in deep neural networks with a single forward pass has recently emerged as a valid alternative to Bayesian Neural Networks. On the premise of informative representations, these deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution (OOD) data while adding negligible computational costs at inference time. However, it remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications - both prerequisites for their practical deployment. To this end, we first provide a taxonomy of DUMs, and evaluate their calibration under continuous distributional shifts. Then, we extend them to semantic segmentation. We find that, while DUMs scale to realistic vision tasks and perform well on OOD detection, the practicality of current methods is undermined by poor calibration under distributional shifts.} }
Endnote
%0 Conference Paper %T On the Practicality of Deterministic Epistemic Uncertainty %A Janis Postels %A Mattia Segù %A Tao Sun %A Luca Daniel Sieber %A Luc Van Gool %A Fisher Yu %A Federico Tombari %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-postels22a %I PMLR %P 17870--17909 %U https://proceedings.mlr.press/v162/postels22a.html %V 162 %X A set of novel approaches for estimating epistemic uncertainty in deep neural networks with a single forward pass has recently emerged as a valid alternative to Bayesian Neural Networks. On the premise of informative representations, these deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution (OOD) data while adding negligible computational costs at inference time. However, it remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications - both prerequisites for their practical deployment. To this end, we first provide a taxonomy of DUMs, and evaluate their calibration under continuous distributional shifts. Then, we extend them to semantic segmentation. We find that, while DUMs scale to realistic vision tasks and perform well on OOD detection, the practicality of current methods is undermined by poor calibration under distributional shifts.
APA
Postels, J., Segù, M., Sun, T., Sieber, L.D., Van Gool, L., Yu, F. & Tombari, F.. (2022). On the Practicality of Deterministic Epistemic Uncertainty. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:17870-17909 Available from https://proceedings.mlr.press/v162/postels22a.html.

Related Material