Abstract
Learning discriminative features and mining salient visual patterns play an important role in salient object detection (SOD) task. Existing SOD methods suffer from limited receptive field and insufficient cross-level feature mining. To this end, we propose a hypergraph attentional convolutional neural network for SOD task. Specifically, our method consists of (1) an attention based feature fusion module, which efficiently fuses lower layer as well as higher layer features, (2) a hypergraph-based long-range dependency encoder, which enhances the receptive field and global context for detection model, (3) a feature refinement layer, which highlights discriminative features and fuses attentional inputs, and (4) a dual iterative feature propagation decoder, which propagates features and upscale lower level feature maps to higher resolution. Both qualitative and quantitively experiments on public datasets verify the effectiveness of our proposed method. Compared with previous works, our model plays favorably against the state-of-the-arts methods.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Li, Z., Liu, G., Zhang, D., Xu, Y.: Robust single-object image segmentation based on salient transition region. Pattern Recogn. 52, 317–331 (2016)
Zhi, X.-H., Shen, H.-B.: Saliency driven region-edge-based top-down level set evolution reveals the asynchronous focus in image segmentation. Pattern Recogn. 80, 241–255 (2018)
Cai, Q., Liu, H., Qian, Y., Zhou, S., Duan, X., Yang, Y.-H.: Saliency-guided level set model for automatic object segmentation. Pattern Recogn. 93, 147–163 (2019)
Gavrila, D., Philomin, V.: Real-time object detection for “smart” vehicles. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 87–93 (1998)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Qian, Q., Wu, X.-J., Kittler, J., Xu, T.-Y.: Correlation tracking with implicitly extending search region. Vis. Comput. 37(5), 1029–1043 (2021)
Wang, Y., Wei, X., Ding, L., Tang, X., Zhang, H.: A robust visual tracking method via local feature extraction and saliency detection. Vis. Comput. 36(4), 683–700 (2020)
Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., Jagersand, M.: Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Liu, J.-J., Hou, Q., Cheng, M.-M., Feng, J., Jiang, J.: A simple pooling based design for real-time salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Zhao, X., Pang, Y., Zhang, L., Lu, H., Zhang, L.: Suppress and balance: A simple gated network for salient object detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision—ECCV 2020, pp. 35–51. Springer International Publishing, Cham (2020)
Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 212–221 (2017)
Zhang, L., Dai, J., Lu, H., He, Y., Wang, G.: A bi-directional message passing model for salient object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1741–1750 (2018)
Mohammadi, S., Noori, M., Bahri, A., Ghofrani Majelan, S., Havaei, M.: Cagnet: content-aware guidance for salient object detection. Pattern Recogn. 103, 107303 (2020)
Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Feng, M., Lu, H., Ding, E.: Attentive feedback network for boundary-aware salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1623–1632 (2019)
Liang, J., Zhou, J., Tong, L., Bai, X., Wang, B.: Material based salient object detection from hyperspectral images. Pattern Recogn. 76, 476–490 (2018)
Huo, L., Jiao, L., Wang, S., Yang, S.: Object-level saliency detection with color attributes. Pattern Recogn. 49, 162–173 (2016)
Li, G., Yu, Y.: Visual saliency detection based on multiscale deep cnn features. IEEE Trans. Image Process. 25(11), 5012–5024 (2016)
Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multilevel convolutional features for salient object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)
Wang, T., Zhang, L., Wang, S., Lu, H., Yang, G., Ruan, X., Borji, A.: Detect globally, refine locally: A novel approach to saliency detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3127–3135 (2018)
Zeng, Y., Zhuge, Y., Lu, H., Zhang, L., Qian, M., Yu, Y.: Multi-source weak supervision for saliency detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6067–6076 (2019)
Zhang, L., Zhang, J., Lin, Z., Lu, H., He, Y.: Capsal: leveraging captioning to boost semantics for salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6017–6026 (2019)
Chen, S., Tan, X., Wang, B., Lu, H., Hu, X., Fu, Y.: Reverse attention-based residual network for salient object detection. IEEE Trans. Image Process. 29, 3763–3776 (2020)
Zhang, X., Wang, T., Qi, J., Lu, H., Wang, G.: Progressive attention guided recurrent network for salient object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 714–722 (2018)
Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.: A stagewise refinement model for detecting salient objects in images. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4039–4048 (2017)
Wu, Z., Su, L., Huang, Q.: Cascaded partial decoder for fast and accurate salient object detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902–3911 (2019)
Liu, Y., Cheng, M.-M., Zhang, X.-Y., Nie, G.-Y., Wang, M.: Dna: deeply supervised nonlinear aggregation for salient object detection. IEEE Trans. TYCBetics (2021)
Deng, Z., Hu, X., Zhu, L., Xu, X., Qin, J., Han, G., Heng, P.-A.: R3net: recurrent residual refinement network for saliency detection. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, International Joint Conferences on Artificial Intelligence Organization, pp. 684–690 (2018)
Pang, Y., Zhao, X., Zhang, L., Lu, H.: Multi-scale interactive network for salient object detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9410–9419 (2020)
Luo, Z., Mishra, A., Achkar, A., Eichel, J.,Li, S., Jodoin, P.-M.: Non-local deep features for salient object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Liu, N., Han, J., Yang, M.-H.: Picanet: Pixel-wise contextual attention learning for accurate saliency detection. IEEE Trans. Image Process. 29, 6438–6451 (2020)
Li, X., Song, D., Dong, Y.: Hierarchical feature fusion network for salient object detection. IEEE Trans. Image Process. 29, 9165–9175 (2020)
Tu, Z., Ma, Y., Li, C., Tang, J., Luo, B.: Edge-guided non-local fully convolutional network for salient object detection. IEEE Trans. Circ. Syst. Video Technol. 31(2), 582–593 (2021)
Wu, R., Feng, M., Guan, W., Wang, D., Lu, H., Ding, E.: A mutual learning method for salient object detection with intertwined multi-supervision. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8142–8151 (2019)
Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. 35(11), 1683–1699 (2019)
Li, L., Qing, L., Wang, Y., Su, J., Cheng, Y., Peng, Y.: Hf-srgr: a new hybrid feature-driven social relation graph reasoning model. Vis. Comput. 1432–2315 (2021)
Harada, S., Akita, H., Tsubaki, M., Baba, Y., Takigawa, I., Yamanishi, Y., Kashima, H.: Dual graph convolutional neural network for predicting chemical networks. BMC Bioinform. 21(3), 94 (2020)
Sanchez-Gonzalez, A., Heess, N., Springenberg, J.T., Merel, J., Riedmiller, M., Hadsell, R., Battaglia, P.: Graph networks as learnable physics engines for inference and control. In: Dy J., Krause A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 4470–4479 (2018)
Bruna, J., Zaremba, W., Szlam, A., LeCun, Y.: Spectral networks and locally connected networks on graphs. In: Bengio Y., LeCun Y. (eds.) 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14–16 Apr 2014, Conference Track Proceedings (2014)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 Apr 2017, Conference Track Proceedings. https://openreview.net/ (2017)
Li, G., Muller, M., Thabet, A., Ghanem, B.: Deepgcns: can gcns go as deep as cnns?. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
Shen, Y., Li, H., Yi, S., Chen, D., Wang, X.: Person re-identification with deep similarity-guided graph neural network. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., Kalantidis, Y.: Graph-based global reasoning networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 433–442 (2019)
Zhang, Q., Wang, S., Wang, X., Sun, Z., Kwong, S., Jiang, J.: Geometry auxiliary salient object detection for light fields via graph neural networks. IEEE Trans. Image Process. 30, 7578–7592 (2021)
Luo, A., Li, X., Yang, F., Jiao, Z., Cheng, H., Lyu, S.: Cascade graph neural networks for rgb-d salient object detection. In: In 16th European Conference on Computer Vision (ECCV) (2020)
Zhou, D., Huang, J., Schölkopf, B.: Learning with hypergraphs: Clustering, classification, and embedding. In: Scholkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems. MIT Press, Cambridge (2007)
Yan, Y., Qin, J., Chen, J., Liu, L., Zhu, F., Tai, Y., Shao, L.: Learning multigranular hypergraphs for video-based person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Feng, Y., You, H., Zhang, Z., Ji, R., Gao, Y.: Hypergraph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 3558–3565 (2019)
Kim, E.-S., Kang, W.Y., On, K.W., Heo, Y.-J., Zhang, B.-T.: Hypergraph attention networks for multimodal learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14569–14578 (2020)
Bai, S., Zhang, F., Torr, P.H.: Hypergraph convolution and hypergraph attention. Pattern Recogn. 110, 107637 (2021)
Jiang, J., Wei, Y., Feng, Y., Cao, J., Gao, Y.: Dynamic hypergraph neural networks. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, International Joint Conferences on Artificial Intelligence Organization, pp. 2635–2641 (2019)
Liang, Z., Chi, Z., Fu, H., Feng, D.D.: Salient object detection using content sensitive hypergraph representation and partitioning. Pattern Recogn. 45(11), 3886–3901 (2012)
Li, X., Li, Y., Shen, C., Dick, A., Hengel, A.V.D.: Contextual hypergraph modeling for salient object detection. In: 2013 IEEE International Conference on Computer Vision, pp. 3328–3335 (2013)
Zhang, J., Fang, S., Ehinger, K.A., Guo, W., Yang, W., Wei, H.: Probabilistic 1065 hypergraph optimization for salient object detection. In: Sun, Y., Lu, H., Zhang, L., Yang, J., Huang, H. (eds.) Intelligence Science and Big Data Engineering, pp. 368–378. Springer International Publishing, Cham (2017)
Ma, M., Xia, C., Li, J.: Pyramidal feature shrinking for salient object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, pp. 2311–2318 (2021)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. In: Bengio Y., LeCun Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May, Conference Track Proceedings (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Park, J., Woo, S., Lee, J.-Y., Kweon, I.S.: Bam: bottleneck attention module. arXiv preprint arXiv:1807.06514
Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3796–3805 (2017)
Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)
Li, Y., Hou, X., Koch, C., Rehg, J.M., Yuille, A.L.: The secrets of salient object segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 280–287 (2014)
Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5455–5463 (2015)
Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.-H.: Saliency detection via graph-based manifold ranking. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3166–3173 (2013)
Movahedi, V., Elder, J.H.: Design and perceptual validation of performance measures for salient object segmentation. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition—Workshops, pp. 49–56 (2010)
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)
Powers, D.M.W.: Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation, CoRR abs/2010.16061
Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597–1604 (2009)
Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 733–740 (2012)
Fan, D.P., Cheng, M.M., Liu, Y., Li, T., Borji, A.: Structure-measure: a new way to evaluate foreground maps. In: IEEE International Conference on Computer Vision (ICCV), pp. 4558–4567 (2017)
Liu, J.-J., Hou, Q., Cheng, M.-M.: Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Trans. Image Process. 29, 8652–8667 (2020)
Ehrig, M. and Euzenat, J: Relaxed precision and recall for ontology matching. In: Ashpole B., Ehrig M.,, Euzenat J., Stuckenschmidt H. (eds.) Integrating Ontologies ’05, Proceedings of the K-CAP 2005 Workshop on Integrating Ontologies, Banff, Canada, 2 Oct 2005, vol. 156 of CEUR Workshop Proceedings, CEUR-WS.org (2005)
Ji, G.-P., Zhu, L., Zhuge, M., Fu, K.: Fast camouflaged object detection via edge-based reversible re-calibration network. Pattern Recogn. 123, 108414 (2022)
Zhuge, M., Fan, D., Liu, N., Zhang, D., Xu, D., Shao, L.: Salient object detection via integrity learning. arXiv preprint arXiv:2101.07663
Li, G., Yu, Y.: Deep contrast learning for salient object detection. arXiv preprint arXiv:1603.01976
Liu, Y., Zhang, Q., Zhang, D. and Han, J.: Employing deep part-object relationships for salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1232–1241 (2019)
Gupta, A.K., Seal, A., Khanna, P., Krejcar, O., Yazidi, A.: Awks: adaptive, weighted k-means-based superpixels for improved saliency detection. Pattern Anal. Appl. 24(2), 625–639 (2021)
Gupta, A.K., Seal, A., Prasad, M., Khanna, P.: Salient object detection techniques in computer vision—a survey. Entropy 22(10), 1174 (2020)
Gupta, A.K., Seal, A., Khanna, P., Herrera-Viedma, E., Krejcar, O.: Almnet: adjacent layer driven multiscale features for salient object detection. IEEE Trans. Instrum. Meas. 70, 1–14 (2021)
Fan, D., Liu, J., Gao, S., Hou, Q., Borji, A., Chen, M.: Salient objects in clutter: bringing salient object detection to the foreground. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H., Yang, R.: Salient object detection in the deep learning era: an in-depth survey. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3051099
Wu, Z., Su, L., Huang, Q.: Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7263–7272 (2019)
Margolin, R., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2014)
Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., Borji, A.: Enhanced-alignment measure for binary foreground map evaluation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, International Joint Conferences on Artificial Intelligence Organization, pp. 698–704 (2018)
Fu, K., Zhao, Q., Yu-Hua Gu, I., Yang, J.: Deepside: a general deep framework for salient object detection. Neurocomputing 356, 69–82 (2019)
Liu, N., Zhang, N., Wan, K., Shao, L., Han, J.: Visual saliency transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4702–4712 (2021)
Wei, J., Wang, S., Huang, Q.: F3net: fusion, feedback and focus for salient object detection. arXiv preprint arXiv:1911.11445
Cheng, M.-M., Mitra, N.J., Huang, X., Torr, P.H.S., Hu, S.-M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)
Wang, B., Chen, S., Wang, J., Hu, X.: Residual feature pyramid networks for salient object detection. Vis. Comput. 36(9), 1897–1908 (2020)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, Zy., Liu, Jw. Hypergraph attentional convolutional neural network for salient object detection. Vis Comput 39, 2881–2907 (2023). https://doi.org/10.1007/s00371-022-02499-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-022-02499-x