research-article

Open access

SmartSketcher: sketch-based image retrieval with dynamic semantic re-ranking

Authors:

Tiziano Portenier,

Matthias ZwickerAuthors Info & Claims

SBIM '17: Proceedings of the Symposium on Sketch-Based Interfaces and Modeling

Article No.: 1, Pages 1 - 12

https://doi.org/10.1145/3092907.3092910

Published: 29 July 2017 Publication History

Abstract

We present a sketch-based image retrieval system, designed to answer arbitrary queries that may go beyond searching for predefined object or scene categories. While sketching is fast and intuitive to formulate visual queries, pure sketch-based image retrieval often returns many outliers because it lacks a semantic understanding of the query. Our key idea is to combine sketch-based queries with inter-active, semantic re-ranking of query results. We leverage progress in deep learning and use a feature representation learned for image classification for re-ranking. This allows us to cluster semantically similar images, re-rank based on the clusters, and present more meaningful query results to the user. We report on two large-scale benchmarks and demonstrate that our re-ranking approach leads to significant improvements over the state of the art. Finally, a user study designed to evaluate a practical use case confirms the benefits of our approach.

Supplementary Material

ZIP File (a1-portenier.zip)

Supplemental files.

Download
12.00 MB

References

[1]

S. Belongie, J. Malik, and J. Puzicha. 2002. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 4 (Apr 2002), 509--522.

Digital Library

[2]

J Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (June 1986), 679--698.

Digital Library

[3]

Xiaochun Cao, Hua Zhang, Si Liu, Xiaojie Guo, and Liang Lin. 2013. SYM-FISH: A Symmetry-Aware Flip Invariant Sketch Histogram Shape Descriptor. In The IEEE International Conference on Computer Vision (ICCV).

Digital Library

[4]

Bryan Catanzaro. 2013. kmeans. https://github.com/bryancatanzaro/kmeans. (2013).

[5]

N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Vol. 1. 886--893 vol. 1.

Digital Library

[6]

Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4 (2012), 44:1--44:10.

Digital Library

[7]

Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors. Visualization and Computer Graphics 17, 11 (2011), 1624--1636.

Digital Library

[8]

Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa. 2012. Sketch-based Shape Retrieval. ACM Trans. Graph. 31, 4, Article 31 (July 2012), 10 pages.

Digital Library

[9]

Mathias Eitz, Ronald Richter, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Photosketcher: interactive sketch-based image synthesis. IEEE Computer Graphics and Applications (2011).

[10]

Rui Hu, Mark Barnard, and John Collomosse. 2010. Gradient field descriptor for sketch based retrieval and localization. In Image Processing (ICIP), 2010 17th IEEE International Conference on. IEEE, 1025--1028. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5649331

[11]

Rui Hu and John Collomosse. 2013. A Performance Evaluation of Gradient Field HOG Descriptor for Sketch Based Image Retrieval. Comput. Vis. Image Underst. 117, 7 (July 2013), 790--806.

Digital Library

[12]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).

[13]

Qiang Li, Yahong Han, and Jianwu Dang. 2015. Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals. Multimedia Tools and Applications (2015).

Digital Library

[14]

Yi Li, Tim Hospedales, Yi-Zhe Song, and Shaogang Gong. 2014. Intra-category sketch-based image retrieval by matching deformable part models. In Proceedings of the British Machine Vision Conference. BMVA Press.

[15]

Shuang Liang, Long Zhao, Yichen Wei, and Jinyuan Jia. 2014. Sketch-based retrieval using content-aware hashing. In Advances in Multimedia Information ProcessingâĂŞPCM 2014. Springer, 133--142. http://link.springer.com/chapter/10.1007/978-3-319-13168-9_14

Digital Library

[16]

Sarthak Parui and Anurag Mittal. 2014. Similarity-invariant sketch-based image retrieval in large databases. In Computer VisionâĂŞECCV 2014. Springer, 398--414. http://link.springer.com/chapter/10.1007/978-3-319-10599-4_26

[17]

Xueming Qian, Xianglong Tan, Yuting Zhang, Richang Hong, and Meng Wang. 2016. Enhancing sketch-based image retrieval by re-ranking and relevance feedback. IEEE Transactions on Image Processing 25, 1 (2016), 195--208.

[18]

Jose M. Saavedra. 2014. Sketch based image retrieval using a soft computation of the histogram of edge local orientations (S-HELO). In Image Processing (ICIP), 2014 IEEE International Conference on. IEEE, 2998--3002. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7025606

[19]

Jose M. Saavedra and Benjamin Bustos. 2010. Pattern Recognition: 32nd DAGM Symposium, Darmstadt, Germany, September 22--24, 2010. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, Chapter An Improved Histogram of Edge Local Orientations for Sketch-Based Image Retrieval, 432--441.

[20]

Jose M. Saavedra and Benjamin Bustos. 2014. Sketch-based image retrieval using keyshapes. Multimedia Tools and Applications 73, 3 (2014), 2033--2062. http://link.springer.com/article/10.1007/s11042-013-1689-0

Digital Library

[21]

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (proceedings of SIGGRAPH) (2016).

[22]

Ravi Kiran Sarvadevabhatla and R. Venkatesh Babu. 2015. Freehand Sketch Recognition Using Deep Features. CoRR abs/1502.00254 (2015). http://arxiv.org/abs/1502.00254

[23]

O. Seddati, S. Dupont, and S. Mahmoudi. 2015. DeepSketch: Deep convolutional neural networks for sketch recognition and similarity search. In 2015 13th International Workshop on Content-Based Multimedia Indexing (CBMI). 1--6.

[24]

Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

[25]

Xinghai Sun, Changhu Wang, Avneesh Sud, Chao Xu, and Lei Zhang. 2013. MagicBrush: Image Search by Color Sketch. In Proceedings of the 21st ACM International Conference on Multimedia (MM '13). ACM, New York, NY, USA, 475--476.

Digital Library

[26]

Zhenbang Sun, Changhu Wang, Liqing Zhang, and Lei Zhang. 2012. Query-adaptive Shape Topic Mining for Hand-drawn Sketch Recognition. In Proceedings of the 20th ACM International Conference on Multimedia (MM '12). ACM, New York, NY, USA, 519--528.

Digital Library

[27]

Attila Szabo, Andrea Vedaldi, and Paolo Favaro. 2015. Building the View Graph of a Category by Exploiting Image Realism. In The IEEE International Conference on Computer Vision (ICCV) Workshops.

Digital Library

[28]

Furuya Takahiko and Ohbuchi Ryutarou. 2014. Visual Saliency Weighting and Cross-Domain Manifold Ranking for Sketch-based Image Retrieval. Multi-Media Modeling (2014). http://www.kki.yamanashi.ac.jp/~ohbuchi/online_pubs/MMM_2014_Furuya/MMM2014_Furuya_Web.pdf

[29]

A. Vedaldi and B. Fulkerson. 2008. VLFeat: An Open and Portable Library of Computer Vision Algorithms. (2008).

[30]

Fang Wang, Le Kang, and Yi Li. 2015. Sketch-Based 3D Shape Retrieval Using Convolutional Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]

Cao Yang, Wang Changhu, Zhang Liqing, and Zhang Lei. 2011. Edgel Index for Large-Scale Sketch-based Image Search. CVPR (2011). http://research.microsoft.com/pubs/149199/0630.pdf

[32]

Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen Change Loy. 2016. Sketch Me That Shoe. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]

Rong Zhou, Liuli Chen, and Liqing Zhang. 2012. Sketch-based Image Retrieval on a Large Scale Database. In Proceedings of the 20th ACM International Conference on Multimedia (MM '12). ACM, New York, NY, USA, 973--976.

Digital Library

Cited By

Yang FIsmail NPang YKebande VAl-Dhaqm AKoh T(2024)A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future DirectionsIEEE Access10.1109/ACCESS.2024.335793912(14847-14869)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3357939
Kumar NAhmed RB. Honnakasturi VSowmya Kamath SMayya V(2021)Sketch-Based Image Retrieval Using Convolutional Neural Networks Based on Feature Adaptation and Relevance FeedbackAdvanced Techniques for IoT Applications10.1007/978-981-16-4435-1_12(103-113)Online publication date: 3-Aug-2021
https://doi.org/10.1007/978-981-16-4435-1_12
Wang LQian XZhang XHou X(2020)Sketch-Based Image Retrieval With Multi-Clustering Re-RankingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2019.295987530:12(4929-4943)Online publication date: Dec-2020
https://doi.org/10.1109/TCSVT.2019.2959875
Show More Cited By

Index Terms

SmartSketcher: sketch-based image retrieval with dynamic semantic re-ranking
1. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Image search

Recommendations

DeepSketch 3

Freehand sketches are a simple and powerful tool for communication. They are easily recognized across cultures and suitable for various applications. In this paper, we use deep convolutional neural networks (ConvNets), state-of-the-art in the field of ...
Sketch-based image retrieval using keyshapes

Although sketch based image retrieval (SBIR) is still a young research area, there are many applications capable of exploiting this retrieval paradigm, such as web searching and pattern detection. Moreover, nowadays drawing a simple sketch query turns ...
Sketch-based Image Retrieval using Generative Adversarial Networks
MM '17: Proceedings of the 25th ACM international conference on Multimedia

For sketch-based image retrieval (SBIR), we propose a generative adversarial network trained on a large number of sketches and their corresponding real images. To imitate human search process, we attempt to match candidate images with theimaginary image ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SBIM '17: Proceedings of the Symposium on Sketch-Based Interfaces and Modeling

July 2017

51 pages

ISBN:9781450350792

DOI:10.1145/3092907

Conference Chairs:
Holger Winnemoeller
Adobe Systems, Inc.
,
Lyn Bartram
Simon Fraser University, Canada
,
Editor:
Stephen N. Spencer
University of Washington
,
General Chairs:
Bruce Gooch
Texas A&M University
,
Yotam Gingold
George Mason University

Copyright © 2017 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2017

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Expressive '17

Sponsor:

SIGGRAPH
EUROGRAPHICS

Expressive '17: The joint symposium on Computational Aesthetics and Sketch Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering

July 29 - 30, 2017

California, Los Angeles

Acceptance Rates

Overall Acceptance Rate 20 of 36 submissions, 56%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
533
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)12

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang FIsmail NPang YKebande VAl-Dhaqm AKoh T(2024)A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future DirectionsIEEE Access10.1109/ACCESS.2024.335793912(14847-14869)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3357939
Kumar NAhmed RB. Honnakasturi VSowmya Kamath SMayya V(2021)Sketch-Based Image Retrieval Using Convolutional Neural Networks Based on Feature Adaptation and Relevance FeedbackAdvanced Techniques for IoT Applications10.1007/978-981-16-4435-1_12(103-113)Online publication date: 3-Aug-2021
https://doi.org/10.1007/978-981-16-4435-1_12
Wang LQian XZhang XHou X(2020)Sketch-Based Image Retrieval With Multi-Clustering Re-RankingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2019.295987530:12(4929-4943)Online publication date: Dec-2020
https://doi.org/10.1109/TCSVT.2019.2959875
Wang LQian XZhang YShen JCao X(2019)Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-rankingIEEE Transactions on Cybernetics10.1109/TCYB.2019.2894498(1-13)Online publication date: 2019
https://doi.org/10.1109/TCYB.2019.2894498
Zhang JChen YLi LFu HTai CWyvill BFu H(2018)Context-based sketch classificationProceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering10.1145/3229147.3229154(1-10)Online publication date: 17-Aug-2018
https://dl.acm.org/doi/10.1145/3229147.3229154

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten