research-article

Pseudo-unknown uncertainty learning for open set object detection

Authors:

Ying ChenAuthors Info & Claims

Volume 303, Issue C

https://doi.org/10.1016/j.knosys.2024.112414

Published: 18 November 2024 Publication History

Abstract

Despite the significant strides made by modern object detectors in the closed-set scenarios, open-set object detection (OSOD) remains a formidable challenge. This is particularly evident in misclassifying objects from unknown categories into pre-existing known classes or ignored background classes. A novel approach called PUDet (Pseudo-unknown Uncertainty Detector) based on Evidential Deep Learning (EDL) is proposed, incorporating two modules: the Class-wise Contrastive Learning Network (CCL) and the Uncertainty-Aware Labeling Network (UAL). For CCL, the module leverages class-wise contrastive learning to encourage intra-class compactness and inter-class separation, thereby reducing the overlap between known and unknown classes. Simultaneously, it establishes compact boundaries for known classes and generates pseudo-unknown candidates to facilitate UAL for better learning pseudo-unknown uncertainty. For UAL, the Weight-Impact EDL (WI-EDL) approach is introduced to enhance uncertainty in edge samples by collecting categorical evidence and weight impact. Subsequently, UAL refines uncertainty via localization quality calibration, facilitating the mining of pseudo-unknown samples from foreground and background proposals to construct compact boundaries between known and unknown categories. In comparison to the state of the arts, the proposed PUDet showcases a substantial improvement, achieving a reduction in Absolute Open-Set Errors by 13%–16% across six OSOD benchmarks.

Highlights

•

Class-wise Contrastive Learning Network constructs compact known class boundaries, obtaining pseudo-unknown candidates.

•

Uncertainty Aware Labeling Network learns the uncertainty of pseudo-unknown samples via weight-impact evidential deep learning.

•

The accurate boundary between known and unknown categories is constructed.

References

[1]

Girshick R.B., Donahue J., Darrell T., Malik J., Rich feature hierarchies for accurate object detection and semantic segmentation, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, IEEE Computer Society, 2014, pp. 580–587,.

Digital Library

[2]

Ren S., He K., Girshick R., Sun J., Faster R-CNN: Towards real-time object detection with region proposal networks, IEEE Trans. Pattern Anal. Mach. Intell. 39 (6) (2017) 1137–1149,.

Digital Library

[3]

Redmon J., Divvala S.K., Girshick R.B., Farhadi A., You only look once: Unified, real-time object detection, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, IEEE Computer Society, 2016, pp. 779–788,.

[4]

Tian Z., Shen C., Chen H., He T., FCOS: fully convolutional one-stage object detection, in: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, IEEE, 2019, pp. 9626–9635,.

[5]

Han J., Ren Y., Ding J., Pan X., Yan K., Xia G.-S., Expanding low-Density Latent Regions for open-set object detection, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, la, USA, June 18-24, 2022, IEEE, 2022, pp. 9581–9590,.

[6]

Scheirer W.J., Rocha A.d., Sapkota A., Boult T.E., Toward open set recognition, IEEE Trans. Pattern Anal. Mach. Intell. 35 (7) (2013) 1757–1772,.

Digital Library

[7]

Dhamija A.R., Günther M., Ventura J., Boult T.E., The overlooked elephant of object detection: Open set, in: IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, IEEE, 2020, pp. 1010–1019,.

[8]

Miller D., Nicholson L., Dayoub F., Sünderhauf N., Dropout sampling for robust object detection in open-set conditions, in: 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, May 21-25, 2018, IEEE, 2018, pp. 1–7,.

Digital Library

[9]

Zhou Z., Yang Y., Wang Y., Xiong R., Open-set object detection using classification-free object proposal and instance-level contrastive learning, IEEE Robot. Autom. Lett. 8 (3) (2023) 1691–1698,.

[10]

Du X., Wang Z., Cai M., Li Y., VOS: learning what you don’t know by virtual outlier synthesis, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022, URL https://openreview.net/forum?id=TW7d65uYu5M.

[11]

Joseph K.J., Khan S.H., Khan F.S., Balasubramanian V.N., Towards open world object detection, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, June 19-25, 2021, Computer Vision Foundation / IEEE, 2021, pp. 5830–5840,. URL https://openaccess.thecvf.com/content/CVPR2021/html/Joseph_Towards_Open_World_Object_Detection_CVPR_2021_paper.html.

[12]

Bendale A., Boult T.E., Towards open set deep networks, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, IEEE Computer Society, 2016, pp. 1563–1572,.

[13]

Neal L., Olson M.L., Fern X.Z., Wong W.-K., Li F., Open set learning with counterfactual images, in: Ferrari V., Hebert M., Sminchisescu C., Weiss Y. (Eds.), Computer Vision - ECCV 2018 - 15th European Conference, in: Lecture Notes in Computer Science, Vol. 11210, Munich, Germany, September 8-14, 2018, Proceedings, Part VI, Springer, 2018, pp. 620–635,.

Digital Library

[14]

Yoshihashi R., Shao W., Kawakami R., You S., Iida M., Naemura T., Classification-reconstruction learning for open-set recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, Computer Vision Foundation / IEEE, 2019, pp. 4016–4025,. URL http://openaccess.thecvf.com/content_CVPR_2019/html/Yoshihashi_Classification-Reconstruction_Learning_for_Open-Set_Recognition_CVPR_2019_paper.html.

[15]

Sun X., Yang Z., Zhang C., Ling K.V., Peng G., Conditional Gaussian distribution learning for open set recognition, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, Computer Vision Foundation / IEEE, 2020, pp. 13477–13486,. URL https://openaccess.thecvf.com/content_CVPR_2020/html/Sun_Conditional_Gaussian_Distribution_Learning_for_Open_Set_Recognition_CVPR_2020_paper.html.

[16]

Yang H.-M., Zhang X.-Y., Yin F., Yang Q., Liu C.-L., Convolutional prototype network for open set recognition, IEEE Trans. Pattern Anal. Mach. Intell. 44 (5) (2022) 2358–2370,.

[17]

Xia Z., Wang P., Dong G., Liu H., Spatial location constraint prototype loss for open set recognition, Comput. Vis. Image Underst. 229 (2023),.

Digital Library

[18]

Miller D., Dayoub F., Milford M., Sünderhauf N., Evaluating merging strategies for sampling-based uncertainty techniques in object detection, in: International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, IEEE, 2019, pp. 2348–2354,.

Digital Library

[19]

Gal Y., Ghahramani Z., Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, in: Balcan M.-F., Weinberger K.Q. (Eds.), Proceedings of the 33nd International Conference on Machine Learning, in: JMLR Workshop and Conference Proceedings, Vol. 48, ICML 2016, New York City, NY, USA, June 19-24, 2016, JMLR.org, 2016, pp. 1050–1059. URL http://proceedings.mlr.press/v48/gal16.html.

[20]

Zheng J., Li W., Hong J., Petersson L., Barnes N., Towards open-set object detection and discovery, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, la, USA, June 19-20, 2022, IEEE, 2022, pp. 3960–3969,.

[21]

Wu Z., Lu Y., Chen X., Wu Z., Kang L., Yu J., UC-OWOD: unknown-classified open world object detection, in: Avidan S., Brostow G.J., Cissé M., Farinella G.M., Hassner T. (Eds.), Computer Vision - ECCV 2022 - 17th European Conference, in: Lecture Notes in Computer Science, Vol. 13670, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X, Springer, 2022, pp. 193–210,.

Digital Library

[22]

Zhao X., Liu X., Shen Y., Qiao Y., Ma Y., Wang D., Revisiting open world object detection, CoRR (2022) arXiv:2201.00471.

[23]

Hendrycks D., Basart S., Mazeika M., Zou A., Kwon J., Mostajabi M., Steinhardt J., Song D., Scaling out-of-distribution detection for real-world settings, in: Chaudhuri K., Jegelka S., Song L., Szepesvári C., Niu G., Sabato S. (Eds.), International Conference on Machine Learning, in: Proceedings of Machine Learning Research, Vol. 162, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, PMLR, 2022, pp. 8759–8773. URL https://proceedings.mlr.press/v162/hendrycks22a.html.

[24]

Sun Y., Guo C., Li Y., ReAct: Out-of-distribution detection with rectified activations, in: Ranzato M., Beygelzimer A., Dauphin Y.N., Liang P., Vaughan J.W. (Eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, 2021, pp. 144–157. URL https://proceedings.neurips.cc/paper/2021/hash/01894d6f048493d2cacde3c579c315a3-Abstract.html.

[25]

Vaze S., Han K., Vedaldi A., Zisserman A., Open-set recognition: A good closed-set classifier is all you need, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022, URL https://openreview.net/forum?id=5hLP5JY9S2d.

[26]

Chan R., Rottmann M., Gottschalk H., Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation, in: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, IEEE, 2021, pp. 5108–5117,.

[27]

Holub A., Perona P., Burl M.C., Entropy-based active learning for object recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2008, Anchorage, AK, USA, 23-28 June, 2008, IEEE Computer Society, 2008, pp. 1–8,.

[28]

Liu W., Wang X., Owens J.D., Li Y., Energy-based out-of-distribution detection, in: Larochelle H., Ranzato M., Hadsell R., Balcan M.-F., Lin H.-T. (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual, 2020, URL https://proceedings.neurips.cc/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html.

[29]

Sensoy M., Kaplan L.M., Kandemir M., Evidential deep learning to quantify classification uncertainty, in: Bengio S., Wallach H.M., Larochelle H., Grauman K., Cesa-Bianchi N., Garnett R. (Eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, MontrÉAl, Canada, 2018, pp. 3183–3193. URL https://proceedings.neurips.cc/paper/2018/hash/a981f2b708044d6fb4a71a1463242520-Abstract.html.

[30]

Amini A., Schwarting W., Soleimany A., Rus D., Deep evidential regression, in: Larochelle H., Ranzato M., Hadsell R., Balcan M.-F., Lin H.-T. (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual, 2020, URL https://proceedings.neurips.cc/paper/2020/hash/aab085461de182608ee9f607f3f7d18f-Abstract.html.

[31]

Chen G., Qiao L., Shi Y., Peng P., Li J., Huang T., Pu S., Tian Y., Learning open set network with discriminative reciprocal points, in: Vedaldi A., Bischof H., Brox T., Frahm J.-M. (Eds.), Computer Vision - ECCV 2020 - 16th European Conference, in: Lecture Notes in Computer Science, Vol. 12348, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, Springer, 2020, pp. 507–522,.

Digital Library

[32]

Chen G., Peng P., Wang X., Tian Y., Adversarial reciprocal points learning for open set recognition, IEEE Trans. Pattern Anal. Mach. Intell. 44 (11) (2022) 8065–8081,.

[33]

Koh P.W., Liang P., Understanding black-box predictions via influence functions, in: Precup D., Teh Y.W. (Eds.), Proceedings of the 34th International Conference on Machine Learning, in: Proceedings of Machine Learning Research, Vol. 70, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, PMLR, 2017, pp. 1885–1894. URL http://proceedings.mlr.press/v70/koh17a.html.

[34]

Zhou D.-W., Ye H.-J., Zhan D.-C., Learning placeholders for open-set recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, June 19-25, 2021, Computer Vision Foundation / IEEE, 2021, pp. 4401–4410,. URL https://openaccess.thecvf.com/content/CVPR2021/html/Zhou_Learning_Placeholders_for_Open-Set_Recognition_CVPR_2021_paper.html.

[35]

Vaze S., Han K., Vedaldi A., Zisserman A., Open-set recognition: A good closed-set classifier is all you need, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022, URL https://openreview.net/forum?id=5hLP5JY9S2d.

[36]

Everingham M., Gool L.V., Williams C.K.I., Winn J.M., Zisserman A., The pascal visual object classes (VOC) challenge, Int. J. Comput. Vis. 88 (2) (2010) 303–338,.

Digital Library

[37]

Lin T.-Y., Maire M., Belongie S.J., Hays J., Perona P., Ramanan D., Dollár P., Zitnick C.L., Microsoft COCO: common objects in context, in: Fleet D.J., Pajdla T., Schiele B., Tuytelaars T. (Eds.), Computer Vision - ECCV 2014 - 13th European Conference, in: Lecture Notes in Computer Science, Vol. 8693, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, Springer, 2014, pp. 740–755,.

[38]

He K., Zhang X., Ren S., Sun J., Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, IEEE Computer Society, 2016, pp. 770–778,.

[39]

Lin T.-Y., Dollár P., Girshick R.B., He K., Hariharan B., Belongie S.J., Feature pyramid networks for object detection, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, IEEE Computer Society, 2017, pp. 936–944,.

[40]

P. Nagarajan, G. Warnell, P. Stone, Deterministic Implementations for Reproducibility in Deep Reinforcement Learning, in: 2nd Reproducibility in Machine Learning Workshop At ICML 2018, Stockholm, Sweden, 2018, URL.

[41]

Gundersen O.E., Shamsaliei S., Isdahl R., Do machine learning platforms provide out-of-the-box reproducibility?, Future Gener. Comput. Syst. 126 (2022) 34–47,.

Digital Library

[42]

Su B., Zhang H., Li J., Zhou Z., Toward generalized few-shot open-set object detection, IEEE Trans. Image Process. 33 (2024) 1389–1402,.

Digital Library

Index Terms

Pseudo-unknown uncertainty learning for open set object detection
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Unsupervised learning and clustering

Index terms have been assigned to the content through auto-classification.

Recommendations

Uncertainty-Aware Deep Open-Set Object Detection
Rough Sets
Abstract
Open-set object detection better simulates the real world compared with close-set object detection. Besides the classes of interest, it also pays attention to unknown objects in the environment. We extend the previous concept of open-set object ...
Enhancing Open-Set Object Detection via Uncertainty-Boxes Identification
Pattern Recognition and Computer Vision
Abstract
Open-set object detection is a challenging task in computer vision, which aims to detect known object categories while simultaneously identifying unknown objects. Inspired by how humans naturally distinguish unseen objects by comparing their ...
VLP Based Open-set Object Detection with Improved RT-DETR
CAIBDA '24: Proceedings of the 2024 4th International Conference on Artificial Intelligence, Big Data and Algorithms

Despite the remarkable accuracy of traditional object detectors, they are unable to detect novel categories. This paper proposes a method for open-set object detection based on generating pseudo-labels using the Vision-Language Pre-trained (VLP) model. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 303, Issue C

Nov 2024

408 pages

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 18 November 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents