research-article

Quadruplet Networks for Sketch-Based Image Retrieval

Authors:

Stéphane Dupont,

Saïd MahmoudiAuthors Info & Claims

ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

Pages 184 - 191

https://doi.org/10.1145/3078971.3078985

Published: 06 June 2017 Publication History

Abstract

Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets) to address sketch-based image retrieval (SBIR). We first train our ConvNets on sketch and image object recognition in a large scale benchmark for SBIR (the sketchy database). We then conduct a comprehensive study of ConvNets features for SBIR, using a kNN similarity search paradigm in the ConvNet feature space. In contrast to recent SBIR works, we propose a new architecture the quadruplet networks which enhance ConvNet features for SBIR. This new architecture enables ConvNets to extract more robust global and local features. We evaluate our approach on three large scale datasets. Our quadruplet networks outperform previous state-of-the-art on two of them by a significant margin and gives competitive results on the third. Our system achieves a recall of 42.16% (at k=1) for the sketchy database (more than 5% improvement), a Kendal score of 43.28Τ_b on the TU-Berlin SBIR benchmark (close to 6Τ_b improvement) and a mean average precision (MAP) of 32.16% on Flickr15k (a category level SBIR benchmark).

References

[1]

Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In European conference on computer vision. Springer, 584--599.

[2]

Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In European conference on computer vision. Springer, 584--599.

[3]

Serge Belongie, Jitendra Malik, and Jan Puzicha. 2002. Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence 24, 4 (2002), 509--522.

Digital Library

[4]

Sreyasee Das Bhattacharjee, Junsong Yuan, Weixiang Hong, and Xiang Ruan. 2016. Query Adaptive Instance Search using Object Sketches. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 1306--1315.

Digital Library

[5]

Konstantinos Bozas and Ebroul Izquierdo. 2012. Large scale sketch based image retrieval using patch hashing. In International Symposium on Visual Computing. Springer, 210--219.

[6]

Tu Bui and John Collomosse. 2015. Scalable sketch-based image retrieval using color gradient features. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1--8.

Digital Library

[7]

Tu Bui, Leonardo Ribeiro, Moacir Ponti, and John Collomosse. 2016. Generali- sation and Sharing in Triplet Convnets for Sketch based Visual Search. arXiv preprint arXiv:1611.05301 (2016).

[8]

Xiaochun Cao, Hua Zhang, Si Liu, Xiaojie Guo, and Liang Lin. 2013. Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. In Proceedings of the IEEE International Conference on Computer Vision. 313--320.

Digital Library

[9]

Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop.

[10]

Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1. IEEE, 886--893.

Digital Library

[11]

Aishwarya Deore and BL Gunjal. 2016. Advanced Sketch Based Image Retrieval System Using Object Boundary Selection Algorithm. (2016).

[12]

Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM Trans. Graph. 31, 4 (2012), 44--1.

Digital Library

[13]

Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2010. An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Computers & Graphics 34, 5 (2010), 482--498.

Digital Library

[14]

Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE transactions on visualization and computer graphics 17, 11 (2011), 1624--1636.

Digital Library

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[16]

Rui Hu and John Collomosse. 2013. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Computer Vision and Image Understanding 117, 7 (2013), 790--806.

Digital Library

[17]

Cheng Jin, Zheming Wang, Tianhao Zhang, Qinen Zhu, and Yuejie Zhang. 2015. A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Re- trieval. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval. ACM, 267--274.

Digital Library

[18]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica- tion with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.

Digital Library

[19]

Joseph J LaViola Jr and Robert C Zeleznik. 2007. MathPad 2: a system for the creation and exploration of mathematical sketches. In ACM SIGGRAPH 2007 courses. ACM, 46.

Digital Library

[20]

Qiang Li, Yahong Han, and Jianwu Dang. 2016. Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals. Multimedia Tools and Applications 75, 5 (2016), 2419--2434.

Digital Library

[21]

Yi Li, Yi-Zhe Song, and Shaogang Gong. 2013. Sketch Recognition by Ensemble Matching of Structured Features. In BMVC, Vol. 1. 2.

[22]

David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91--110.

Digital Library

[23]

Jianwei Niu, Jun Ma, Jie Lu, Xuefeng Liu, and Zeyu Zhu. 2017. M-SBIR: An Improved Sketch-Based Image Retrieval Method Using Visual Word Mapping. In International Conference on Multimedia Modeling. Springer, 257--268.

[24]

Tom Y Ouyang and Randall Davis. 2011. ChemInk: a natural real-time recognition system for chemical drawings. In Proceedings of the 16th international conference on Intelligent user interfaces. ACM, 267--276.

Digital Library

[25]

Yonggang Qi, Yi-Zhe Song, Tao Xiang, Honggang Zhang, Timothy Hospedales, Yi Li, and Jun Guo. 2015. Making better use of edges via perceptual grouping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1856--1865.

[26]

Yonggang Qi, Yi-Zhe Song, Honggang Zhang, and Jun Liu. 2016. Sketch-based image retrieval via Siamese convolutional neural network. In Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2460--2464.

[27]

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35, 4 (2016), 119.

Digital Library

[28]

Ravi Kiran Sarvadevabhatla and R Venkatesh Babu. 2015. Freehand sketch recognition using deep features. arXiv preprint arXiv:1502.00254 (2015).

[29]

Ravi Kiran Sarvadevabhatla, Jogendra Kundu, and others. 2016. Enabling My Robot To Play Pictionary: Recurrent Neural Networks For Sketch Recognition. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 247--251.

Digital Library

[30]

Rosália G Schneider and Tinne Tuytelaars. 2014. Sketch classification and classification-driven analysis using fisher vectors. ACM Transactions on Graphics (TOG) 33, 6 (2014), 174.

Digital Library

[31]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.

[32]

Omar Seddati, Stephane Dupont, and Saïd Mahmoudi. 2015. Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on. IEEE, 1--6.

[33]

Omar Seddati, Stephane Dupont, and Saïd Mahmoudi. 2016. DeepSketch 2: Deep convolutional neural networks for partial sketch recognition. In Content-Based Multimedia Indexing (CBMI), 2016 14th International Workshop on. IEEE, 1--6.

[34]

Omar Seddati, Stéphane Dupont, and Saïd Mahmoudi. 2016. DeepSketch2Image: Deep Convolutional Neural Networks for Partial Sketch Recognition and Image Retrieval. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 739--741.

Digital Library

[35]

Eli Shechtman and Michal Irani. 2007. Matching local self-similarities across images and videos. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 1--8.

[36]

Josef Sivic, Andrew Zisserman, and others. 2003. Video google: A text retrieval approach to object matching in videos. In iccv, Vol. 2. 1470--1477.

Digital Library

[37]

Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Transactions of the Society for Computer Simulation 2, 5 (1964), R--3.

[38]

B Szántó, P Pozsegovics, Z Vámossy, and Sz Sergyan. 2011. Sketch4match Content-based image retrieval system using sketches. In Applied Machine Intelligence and Informatics (SAMI), 2011 IEEE 9th International Symposium on. IEEE, 183--188.

[39]

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition. 1--9.

[40]

Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. 2014. Deep learning for content-based image retrieval: A comprehensive study. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 157--166.

Digital Library

[41]

Yuxin Wang, Miao Yu, Qi Jia, and He Guo. 2011. Query by sketch: An asymmetric sketch-vs-image retrieval system. In Image and Signal Processing (CISP), 2011 4th International Congress on, Vol. 3. IEEE, 1368--1372.

[42]

Yongxin Yang and Timothy M Hospedales. 2015. Deep neural networks for sketch recognition. arXiv preprint arXiv: 1501.07873 (2015).

[43]

Kemal Tugrul Yesilbek, Cansu Sen, Serike Cakmak, and T Metin Sezgin. 2015. SVM-based sketch recognition: which hyperparameter interval to try?. In Proceedings of the workshop on Sketch-Based Interfaces and Modeling. Eurographics Association, 117--121.

Digital Library

[44]

Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. 2015. Sketch-a-net that beats humans. arXiv preprint arXiv:1501.07873 (2015).

[45]

Hua Zhang, Si Liu, Changqing Zhang, Wenqi Ren, Rui Wang, and Xiaochun Cao. 2016. Sketchnet: Sketch classification with web images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1105--1113.

[46]

Rong Zhou, Liuli Chen, and Liqing Zhang. 2012. Sketch-based image retrieval on a large scale database. In Proceedings of the 20th ACM international conference on Multimedia. ACM, 973--976.

Digital Library

Cited By

Yang FIsmail NPang YKebande VAl-Dhaqm AKoh T(2024)A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future DirectionsIEEE Access10.1109/ACCESS.2024.335793912(14847-14869)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3357939
Wang SXia YXiang NQian KYang XYou LZhang J(2024)Multi-colour sketch-based image retrieval with an explicable feature embeddingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108757135(108757)Online publication date: Sep-2024
https://doi.org/10.1016/j.engappai.2024.108757
Yang FIsmail NPang YAlsayed A(2024)CAMIR: fine-tuning CLIP and multi-head cross-attention mechanism for multimodal image retrieval with sketch and text featuresInternational Journal of Multimedia Information Retrieval10.1007/s13735-024-00352-614:1Online publication date: 24-Dec-2024
https://doi.org/10.1007/s13735-024-00352-6
Show More Cited By

Index Terms

Quadruplet Networks for Sketch-Based Image Retrieval
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Sketch-based Image Retrieval using Generative Adversarial Networks
MM '17: Proceedings of the 25th ACM international conference on Multimedia

For sketch-based image retrieval (SBIR), we propose a generative adversarial network trained on a large number of sketches and their corresponding real images. To imitate human search process, we attempt to match candidate images with theimaginary image ...
DeepSketch 3

Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets), state-of-the-art in the field of ...
Triplet Networks Feature Masking for Sketch-Based Image Retrieval
Image Analysis and Recognition
Abstract
Freehand sketches are an intuitive tool for communication and suitable for various applications. In this paper, we present an effective approach that combines triplet networks and an attention mechanism for sketch-based image retrieval (SBIR). The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

June 2017

524 pages

ISBN:9781450347013

DOI:10.1145/3078971

General Chairs:
Bogdan Ionescu
University Politehnica of Bucharest, Romania
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Jiashi Feng
National University of Singapore, Singapore
,
Martha Larson
Radboud University & Delft University of Technology, The Netherlands
,
Rainer Lienhart
University of Augsburg, Germany
,
Cees Snoek
University of Amsterdam & Qualcomm Research Netherlands, The Netherlands

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMR '17

Sponsor:

SIGMM

ICMR '17: International Conference on Multimedia Retrieval

June 6 - 9, 2017

Bucharest, Romania

Acceptance Rates

ICMR '17 Paper Acceptance Rate 33 of 95 submissions, 35%;

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
440
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang FIsmail NPang YKebande VAl-Dhaqm AKoh T(2024)A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future DirectionsIEEE Access10.1109/ACCESS.2024.335793912(14847-14869)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3357939
Wang SXia YXiang NQian KYang XYou LZhang J(2024)Multi-colour sketch-based image retrieval with an explicable feature embeddingEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108757135(108757)Online publication date: Sep-2024
https://doi.org/10.1016/j.engappai.2024.108757
Yang FIsmail NPang YAlsayed A(2024)CAMIR: fine-tuning CLIP and multi-head cross-attention mechanism for multimodal image retrieval with sketch and text featuresInternational Journal of Multimedia Information Retrieval10.1007/s13735-024-00352-614:1Online publication date: 24-Dec-2024
https://doi.org/10.1007/s13735-024-00352-6
Xu PHospedales TYin QSong YXiang TWang L(2023)Deep Learning for Free-Hand Sketch: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.314885345:1(285-312)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPAMI.2022.3148853
Zhang ZChen YZhang YFeng RZhang T(2023)Boosting Fine-Grained Sketch-Based Image Retrieval with Self-Supervised LearningICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP49357.2023.10095145(1-5)Online publication date: 4-Jun-2023
https://doi.org/10.1109/ICASSP49357.2023.10095145
Mahmoudi SBelarbi M(2022)Large Scale Multimedia Management: Recent ChallengesInformation10.3390/info1301002813:1(28)Online publication date: 10-Jan-2022
https://doi.org/10.3390/info13010028
Seddati ODupont SMahmoudi SDutoit T(2022)Towards Human Performance on Sketch-Based Image RetrievalProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549582(77-83)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549582
Huang YWang QYang WLiao QMeng H(2022)Hierarchical Deep Multitask Learning With the Attention Mechanism for Similarity LearningIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2021.313731614:4(1729-1742)Online publication date: Dec-2022
https://doi.org/10.1109/TCDS.2021.3137316
Efthymiadis NTolias GChum O(2022)Edge Augmentation for Large-Scale Sketch Recognition without Sketches2022 26th International Conference on Pattern Recognition (ICPR)10.1109/ICPR56361.2022.9956233(3595-3602)Online publication date: 21-Aug-2022
https://doi.org/10.1109/ICPR56361.2022.9956233
Zhao SWang LQian XChen J(2022)Enhancing performance-based generative architectural design with sketch-based image retrieval: a pilot study on designing building facade fenestrationsThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-021-02170-x38:8(2981-2997)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00371-021-02170-x
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten