research-article

Co-saliency Detection Based on Hierarchical Consistency

Authors:

Qian LiAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 1392 - 1400

https://doi.org/10.1145/3343031.3351016

Published: 15 October 2019 Publication History

Abstract

As an interesting and emerging topic, co-saliency detection aims at discovering common and salient objects in a group of related images, which is useful to variety of visual media applications. Although a number of approaches have been proposed to address this problem, many of them are designed with the misleading assumption, suboptimal image representation, or heavy supervision cost and thus still suffer from certain limitations, which reduces their capability in the real-world scenarios. To alleviate these limitations, we propose a novel unsupervised co-saliency detection method, which successively explores the hierarchical consistency in the image group including background consistency, high-level and low-level objects consistency in a unified framework. We first design a novel superpixel-wise variational autoencoder (SVAE) network to precisely distinguish the salient objects from the background collection based on the reconstruction errors. Then, we propose a two-stage clustering strategy to explore the multi-level salient objects consistency by using high-level and low-level features separately. Finally, the co-saliency results are refined by applying a CRF based refinement method with the multi-level salient objects consistency. Extensive experiments on three widely datasets show that our method achieves superior or competitive performance compared to the state-of-the-art methods.

References

[1]

Pablo Andrés Arbeláez, Jordi Pont-Tuset, Jonathan T. Barron, Ferran Marqués, and Jitendra Malik. 2014. Multiscale Combinatorial Grouping. In CVPR. 328--335. https://doi.org/10.1109/CVPR.2014.49

[2]

Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen. 2010. iCoseg: Interactive co-segmentation with intelligent scribble guidance. In CVPR. 3169--3176. https://doi.org/10.1109/CVPR.2010.5540080

[3]

Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating Sentences from a Continuous Space. In SIGNLL. pages 10--21.

[4]

Xiaochun Cao, Zhiqiang Tao, Bao Zhang, Huazhu Fu, and Wei Feng. 2014. Selfadaptively weighted co-saliency detection via rank constraint. TIP 23, 9 (2014), 4175--4186. https://doi.org/10.1109/TIP.2014.2332399

[5]

Kai-Yueh Chang, Tyng-Luh Liu, and Shang-Hong Lai. 2011. From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model. In cvpr. 2129--2136. https://doi.org/10.1109/CVPR.2011.5995415

[6]

Hwann-Tzong Chen. 2010. Preattentive co-saliency detection. In ICIP. 1117--1120. https://doi.org/10.1109/ICIP.2010.5650014

[7]

Ming Ming Cheng, Niloy J. Mitra, Xiaolei Huang, and Shi Min Hu. 2014. SalientShape: Group saliency in image collections. Visual Computer 30, 4 (2014), 443--453. https://doi.org/10.1007/s00371-013-0867--4

Digital Library

[8]

Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. 2017. Structure- Measure: A NewWay to Evaluate Foreground Maps. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22--29, 2017. 4558--4567. https://doi.org/10.1109/ICCV.2017.487

[9]

Brendan J Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. science 315, 5814 (2007), 972--976.

[10]

Huazhu Fu, Xiaochun Cao, and Zhuowen Tu. 2013. Cluster-Based Co-Saliency Detection. TIP 22, 10 (2013), 3766--3778. https://doi.org/10.1109/TIP.2013.2260166

Digital Library

[11]

Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In CVPR. pages 2414--2423. https: //doi.org/10.1109/CVPR.2016.265

[12]

Junwei Han, Gong Cheng, Zhenpeng Li, and Dingwen Zhang. 2017. A unified metric learning-based framework for co-saliency detection. IEEE Transactions on Circuits and Systems for Video Technology (2017).

[13]

Junwei Han, Gong Cheng, Zhenpeng Li, and Dingwen Zhang. 2018. A Unified Metric Learning-Based Framework for Co-Saliency Detection. IEEE Trans. Circuits Syst. Video Techn. 28, 10 (2018), 2473--2483. https://doi.org/10.1109/TCSVT.2017. 2706264

Digital Library

[14]

Junwei Han, Rong Quan, Dingwen Zhang, and Feiping Nie. 2018. Robust Object Co-Segmentation Using Background Prior. IEEE Trans. Image Processing 27, 4 (2018), 1639--1651. https://doi.org/10.1109/TIP.2017.2781424

Digital Library

[15]

J Han, D Zhang, X Hu, L Guo, and Ren. 2015. Background Prior Based Salient Object Detection via Deep Reconstruction Residual. TCSVT 25, 8 (2015), 1309-- 1321. https://doi.org/10.1109/TCSVT.2014.2381471

Digital Library

[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. TPAMI 37, 9 (2015), 1904--1916. https://doi.org/10.1109/TPAMI.2015.2389824

Digital Library

[17]

Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, Xiaoning Qian, and Yung-Yu Chuang. 2018. Unsupervised CNN-Based Co-saliency Detection with Graphical Optimization. In ECCV. 502--518. https://doi.org/10.1007/978--3-030-01228--1_30

[18]

David E. Jacobs, Dan B. Goldman, and Eli Shechtman. 2010. Cosaliency: where people look when comparing images. In UIST. 219--228. https://doi.org/10.1145/ 1866029.1866066

[19]

Koteswar Rao Jerripothula, Jianfei Cai, and Junsong Yuan. 2016. Image Cosegmentation via Saliency Co-fusion. IEEE Trans. Multimedia 18, 9 (2016), 1896-- 1909. https://doi.org/10.1109/TMM.2016.2576283

Digital Library

[20]

Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. CoRR abs/1312.6114 (2013). arXiv:1312.6114 http://arxiv.org/abs/1312.6114

[21]

Philipp Krähenbühl and Vladlen Koltun. 2011. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12--14 December 2011, Granada, Spain. 109--117. http://papers.nips.cc/paper/ 4296-efficient-inference-in-fully-connected-crfs-with-gaussian-edge-potentials

[22]

Guanbin Li, Yuan Xie, Liang Lin, and Yizhou Yu. 2017. Instance-Level Salient Object Segmentation. In CVPR. 247--256. https://doi.org/10.1109/CVPR.2017.34

[23]

Hongliang Li, Fanman Meng, and King Ngi Ngan. 2013. Co-Salient Object Detection From Multiple Images. IEEE Trans. Multimedia 15, 8 (2013), 1896--1909. https://doi.org/10.1109/TMM.2013.2271476

Digital Library

[24]

Hongliang Li and King Ngi Ngan. 2011. A Co-Saliency Model of Image Pairs. TIP 20, 12 (2011), 3365--3375. https://doi.org/10.1109/TIP.2011.2156803

Digital Library

[25]

Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence across Scenes and Its Applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 978--994. https://doi.org/10.1109/TPAMI.2010.147

Digital Library

[26]

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2012. Isolation-Based Anomaly Detection. TKDD 6, 1 (2012), 3:1--3:39. https://doi.org/10.1145/2133360.2133363

[27]

Zhi Liu,Wenbin Zou, Lina Li, Liquan Shen, and Olivier Le Meur. 2014. Co-saliency detection based on hierarchical segmentation. SPL 21, 1 (2014), 88--92.

[28]

Huchuan Lu, Xiaohui Li, Lihe Zhang, Xiang Ruan, and Ming-Hsuan Yang. 2016. Dense and Sparse Reconstruction Error Based Saliency Descriptor. TIP 25, 4 (apr 2016), 1592--1603. https://doi.org/10.1109/TIP.2016.2524198

Digital Library

[29]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). arXiv:1409.1556 http://arxiv.org/abs/1409.1556

[30]

Zhiyu Tan, Liang Wan, Wei Feng, and Chi-Man Pun. 2013. Image co-saliency detection by propagating superpixel affinities. In ICASSP. 2114--2118. https: //doi.org/10.1109/ICASSP.2013.6638027

[31]

Lina Wei, Shanshan Zhao, Omar El Farouk Bourahla, Xi Li, and Fei Wu. 2017. Group-wise Deep Co-saliency Detection. In IJCAI. 3041--3047. https://doi.org/ 10.24963/ijcai.2017/424

[32]

John M. Winn, Antonio Criminisi, and Thomas P. Minka. 2005. Object Categorization by Learned Universal Visual Dictionary. In ICCV. 1800--1807. https: //doi.org/10.1109/ICCV.2005.171

[33]

Jianru Xue, Le Wang, Nanning Zheng, and Gang Hua. 2013. Automatic salient object extraction with contextual cue and its applications to recognition and alpha matting. Pattern Recognition 46, 11 (nov 2013), 2874--2889. https://doi.org/ 10.1016/j.patcog.2013.03.028

[34]

Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. 2013. Saliency Detection via Graph-Based Manifold Ranking. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23--28, 2013. 3166--3173. https://doi.org/10.1109/CVPR.2013.407

Digital Library

[35]

Xiwen Yao, Junwei Han, Dingwen Zhang, and Feiping Nie. 2017. Revisiting Co- Saliency Detection: A Novel Approach Based on Two-Stage Multi-View Spectral Rotation Co-clustering. IEEE Trans. Image Processing 26, 7 (2017), 3196--3209. https://doi.org/10.1109/TIP.2017.2694222

Digital Library

[36]

Dingwen Zhang, Junwei Han, Chao Li, Jingdong Wang, and Xuelong Li. 2016. Detection of Co-salient Objects by Looking Deep and Wide. IJCV 120, 2 (2016), 215--232. https://doi.org/10.1007/s11263-016-0907--4

Digital Library

[37]

Dingwen Zhang, Junwei Han, Chao Li, Jingdong Wang, and Xuelong Li. 2016. Detection of Co-salient Objects by Looking Deep and Wide. International Journal of Computer Vision 120, 2 (2016), 215--232. https://doi.org/10.1007/ s11263-016-0907--4

Digital Library

[38]

Dingwen Zhang, Deyu Meng, and Junwei Han. 2017. Co-saliency detection via a self-paced multiple-instance learning framework. TPAMI 39, 5 (2017), 865--878.

Digital Library

[39]

Jianming Zhang, Stan Sclaroff, Zhe Lin, Xiaohui Shen, Brian L. Price, and Radomír Mech. 2016. Unconstrained Salient Object Detection via Proposal Subset Optimization. In CVPR. 5733--5742. https://doi.org/10.1109/CVPR.2016.618

[40]

Lei Zhang and Hua Huang. 2012. Hierarchical Narrative Collage For Digital Photo Album. Computer Graphics Forum 31, 7 (sep 2012), 2173--2181. https: //doi.org/10.1111/j.1467--8659.2012.03210.x

Digital Library

[41]

Xiaoju Zheng, Zheng-Jun Zha, and Liansheng Zhuang. 2018. A Feature-Adaptive Semi-Supervised Framework for Co-saliency Detection. In 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22--26, 2018. 959--966. https://doi.org/10.1145/3240508.3240648

Digital Library

Cited By

Wang HLi BWu SShen SLiu FDing SZhou A(2023)Rethinking the Learning Paradigm for Dynamic Facial Expression Recognition2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01722(17958-17968)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01722
Ge YZhang QXiang TZhang CZhang JBi H(2023)GSNNet: Group semantic-guided neighbor interaction network for co-salient object detectionComputer Vision and Image Understanding10.1016/j.cviu.2022.103611227(103611)Online publication date: Jan-2023
https://doi.org/10.1016/j.cviu.2022.103611
Zhu TZhang DHu YWang TJiang XZhu JLi J(2022)Horizontal-to-Vertical Video ConversionIEEE Transactions on Multimedia10.1109/TMM.2021.309220224(3036-3048)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3092202
Show More Cited By

Index Terms

Co-saliency Detection Based on Hierarchical Consistency
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Interest point and salient region detections

Recommendations

Co-saliency detection via inter and intra saliency propagation

The goal of salient object detection from an image is to extract the regions which capture the attention of the human visual system more than other regions of the image. In this paper a novel method is presented for detecting salient objects from a set ...
Co-saliency Detection with Graph Matching
Survey Paper, Research Commentary and Regular Papers

Recently, co-saliency detection, which aims to automatically discover common and salient objects appeared in several relevant images, has attracted increased interest in the computer vision community. In this article, we present a novel graph-matching ...
Traffic sign detection based on visual co-saliency in complex scenes

Co-saliency detection aims at finding the salient regions from multiple images which capture the focus of human visual system. In this paper, a novel visual co-saliency algorithm is proposed, which adopts three human visual attention cues: contrast, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National High Technology Re-search and Development Program of China
China Postdoctoral Science Foundation
Innovation Fund of State Key Laboratory for Novel Software Technology
National Natural Science Foundation of China

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
229
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang HLi BWu SShen SLiu FDing SZhou A(2023)Rethinking the Learning Paradigm for Dynamic Facial Expression Recognition2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01722(17958-17968)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01722
Ge YZhang QXiang TZhang CZhang JBi H(2023)GSNNet: Group semantic-guided neighbor interaction network for co-salient object detectionComputer Vision and Image Understanding10.1016/j.cviu.2022.103611227(103611)Online publication date: Jan-2023
https://doi.org/10.1016/j.cviu.2022.103611
Zhu TZhang DHu YWang TJiang XZhu JLi J(2022)Horizontal-to-Vertical Video ConversionIEEE Transactions on Multimedia10.1109/TMM.2021.309220224(3036-3048)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3092202
Chen ZLi BXu JWu SDing SZhang W(2022)Towards Practical Certifiable Patch Defense with Vision Transformer2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01472(15127-15137)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.01472
Zhang JLi BXu JWu SDing SZhang LWu C(2022)Towards Efficient Data Free Blackbox Adversarial Attack2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01469(15094-15104)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.01469
Ma TBi MZhang JYuan WZhang ZXie YDing SMa L(2022)Mutually Reinforcing Structure with Proposal Contrastive Consistency for Few-Shot Object DetectionComputer Vision – ECCV 202210.1007/978-3-031-20044-1_23(400-416)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20044-1_23
Gu ZYao TChen YDing SMa L(2022)Hierarchical Contrastive Inconsistency Learning for Deepfake Video DetectionComputer Vision – ECCV 202210.1007/978-3-031-19775-8_35(596-613)Online publication date: 23-Oct-2022
https://doi.org/10.1007/978-3-031-19775-8_35
Chen ZLi BWu SXu JDing SZhang W(2022)Shape Matters: Deformable Patch AttackComputer Vision – ECCV 202210.1007/978-3-031-19772-7_31(529-548)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-19772-7_31
Fan DLi TLin ZJi GZhang DCheng MFu HShen J(2021)Re-thinking Co-Salient Object DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3060412(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2021.3060412
Tang LLi BWu YXiao BDing S(2021)Fast: Feature Aggregation for Detecting Salient Object in Real-TimeICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP39728.2021.9414457(1525-1529)Online publication date: 6-Jun-2021
https://doi.org/10.1109/ICASSP39728.2021.9414457
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents