research-article

Sketch classification and classification-driven analysis using Fisher vectors

Authors:

Rosália G. Schneider,

Tinne TuytelaarsAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 6

Article No.: 174, Pages 1 - 9

https://doi.org/10.1145/2661229.2661231

Published: 19 November 2014 Publication History

Abstract

We introduce an approach for sketch classification based on Fisher vectors that significantly outperforms existing techniques. For the TU-Berlin sketch benchmark [Eitz et al. 2012a], our recognition rate is close to human performance on the same task. Motivated by these results, we propose a different benchmark for the evaluation of sketch classification algorithms. Our key idea is that the relevant aspect when recognizing a sketch is not the intention of the person who made the drawing, but the information that was effectively expressed. We modify the original benchmark to capture this concept more precisely and, as such, to provide a more adequate tool for the evaluation of sketch classification techniques. Finally, we perform a classification-driven analysis which is able to recover semantic aspects of the individual sketches, such as the quality of the drawing and the importance of each part of the sketch for the recognition.

References

[1]

Barla, P., Thollot, J., and Sillion, F. X. 2005. Geometric clustering for line drawing simplification. In ACM SIGGRAPH 2005 Sketches, ACM, New York, NY, USA, SIGGRAPH '05.

Digital Library

[2]

Cao, X., Zhang, H., Liu, S., Guo, X., and Lin, L. 2013. Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. In IEEE International Conference on Computer Vision (ICCV).

Digital Library

[3]

Csurka, G., Dance, C. R., Fan, L., Willamowski, J., and Bray, C. 2004. Visual categorization with bags of keypoints. In In Workshop on Statistical Learning in Computer Vision, ECCV, 1--22.

[4]

Davis, J., Agrawala, M., Chuang, E., Popović, Z., and Salesin, D. 2003. A sketching interface for articulated figure animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '03, 320--328.

Digital Library

[5]

Donmez, N., and Singh, K. 2012. Concepture: A regular language based framework for recognizing gestures with varying and repetitive patterns. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SBIM '12, 29--37.

Digital Library

[6]

Eitz, M., Hildebrand, K., Boubekeur, T., and Alexa, M. 2011. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Transactions on Visualization and Computer Graphics 17, 11 (Nov.), 1624--1636.

Digital Library

[7]

Eitz, M., Hays, J., and Alexa, M. 2012. How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4, 44:1--44:10.

Digital Library

[8]

Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4 (July), 31:1--31:10.

Digital Library

[9]

Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. 2010. The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88, 2 (June), 303--338.

Digital Library

[10]

Hammond, T., and Davis, R. 2007. Ladder, a sketching language for user interface developers. In ACM SIGGRAPH 2007 Courses, ACM, New York, NY, USA, SIGGRAPH '07.

Digital Library

[11]

Hearst, M. A., Dumais, S., Osman, E., Platt, J., and Scholkopf, B. 1998. Support vector machines. Intelligent Systems and their Applications, IEEE 13, 4, 18--28.

Digital Library

[12]

Hoiem, D., Chodpathumwan, Y., and Dai, Q. 2012. Diagnosing error in object detectors. In Proceedings of the 12th European Conference on Computer Vision - Volume Part III, Springer-Verlag, Berlin, Heidelberg, ECCV'12, 340--353.

Digital Library

[13]

Jaakkola, T., and Haussler, D. 1998. Exploiting generative models in discriminative classifiers. In In Advances in Neural Information Processing Systems 11, MIT Press, 487--493.

Digital Library

[14]

LaViola, Jr., J. J., and Zeleznik, R. C. 2004. Mathpad2: A system for the creation and exploration of mathematical sketches. ACM Trans. Graph. 23, 3 (Aug.), 432--440.

Digital Library

[15]

Lazebnik, S., Schmid, C., and Ponce, J. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2, 2169--2178.

Digital Library

[16]

Li, Y., Song, Y.-Z., and Gong, S. 2013. Sketch recognition by ensemble matching of structured features. In In British Machine Vision Conference (BMVC).

[17]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91--110.

Digital Library

[18]

Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge, J. A. 2009. Sketch-based modeling: A survey. Computers & Graphics 33, 1, 85--103.

Digital Library

[19]

Ouyang, T. Y., and Davis, R. 2011. Chemink: A natural real-time recognition system for chemical drawings. In Proceedings of the 16th International Conference on Intelligent User Interfaces, ACM, New York, NY, USA, IUI '11, 267--276.

Digital Library

[20]

Perronnin, F., Liu, Y., Sanchez, J., and Poirier, H. 2010. Large-scale image retrieval with compressed fisher vectors. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, 3384--3391.

[21]

Rubine, D. 1991. Specifying gestures by example. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '91, 329--337.

Digital Library

[22]

Sánchez, J., Perronnin, F., Mensink, T., and Verbeek, J. 2013. Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision 105, 3, 222--245.

Digital Library

[23]

Schmidt, R., Wyvill, B., Sousa, M. C., and Jorge, J. A. 2006. Shapeshop: Sketch-based solid modeling with blobtrees. In ACM SIGGRAPH 2006 Courses, ACM, New York, NY, USA, SIGGRAPH '06.

Digital Library

[24]

Sezgin, T. M. 2001. Sketch based interfaces: Early processing for sketch understanding. In Proceedings of PUI-2001. NY, ACM Press.

Digital Library

[25]

Shesh, A., and Chen, B. 2008. Efficient and dynamic simplification of line drawings. Comput. Graph. Forum 27, 2, 537--545.

[26]

Sivic, J., and Zisserman, A. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings of the International Conference on Computer Vision, vol. 2, 1470--1477.

Digital Library

[27]

Sutherland, I. E. 1964. Sketch pad a man-machine graphical communication system. In Proceedings of the SHARE Design Automation Workshop, ACM, New York, NY, USA, DAC '64, 6.329--6.346.

Digital Library

[28]

Torralba, A., and Efros, A. A. 2011. Unbiased look at dataset bias. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, Washington, DC, USA, CVPR '11, 1521--1528.

Digital Library

[29]

Vedaldi, A., and Fulkerson, B., 2008. VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/.

Cited By

Zheng YPang KDas AChang DSong YMa Z(2024)CreativeSeg: Semantic Segmentation of Creative SketchesIEEE Transactions on Image Processing10.1109/TIP.2024.337419633(2266-2278)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3374196
Paolillo DTarini M(2024)Automatic and User-Assisted Sphere-Mesh ConstructionIEEE Computer Graphics and Applications10.1109/MCG.2024.342665644:6(105-117)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/MCG.2024.3426656
Zhou YWang JYang JNi PLu GFang HLi ZYu HHuang K(2024)Cross-Modal Pixel-and-Stroke representation aligning networks for free-hand sketch recognitionExpert Systems with Applications10.1016/j.eswa.2023.122505240(122505)Online publication date: Apr-2024
https://doi.org/10.1016/j.eswa.2023.122505
Show More Cited By

Index Terms

Sketch classification and classification-driven analysis using Fisher vectors
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding

Recommendations

Context-based sketch classification
Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering

We present a novel context-based sketch classification framework using relations extracted from scene images. Most of existing methods perform sketch classification by considering individually sketched objects and often fail to identify their correct ...
Uncertainty in Communication with a Sketch

Sketches are one of the main tools for the communication of design ideas during the conceptual phase of the design process. In design communication, one of the major problems is the uncertainty associated with imprecisely defined sketches. There is a need ...
Automatic understanding of sketch maps using context-aware classification

We present the first comprehensive system for offline classifying sketch map objects.We created a database of labeled sketch maps for training and evaluation purposes.Context-awareness improves the classification of sketch map objects greatly. Sketching ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 6

November 2014

704 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2661229

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 November 2014

Published in TOG Volume 33, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

134
Total Citations
View Citations
931
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)5

Reflects downloads up to 09 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zheng YPang KDas AChang DSong YMa Z(2024)CreativeSeg: Semantic Segmentation of Creative SketchesIEEE Transactions on Image Processing10.1109/TIP.2024.337419633(2266-2278)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3374196
Paolillo DTarini M(2024)Automatic and User-Assisted Sphere-Mesh ConstructionIEEE Computer Graphics and Applications10.1109/MCG.2024.342665644:6(105-117)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/MCG.2024.3426656
Zhou YWang JYang JNi PLu GFang HLi ZYu HHuang K(2024)Cross-Modal Pixel-and-Stroke representation aligning networks for free-hand sketch recognitionExpert Systems with Applications10.1016/j.eswa.2023.122505240(122505)Online publication date: Apr-2024
https://doi.org/10.1016/j.eswa.2023.122505
Yang LPang KZhang HSong Y(2024)Annotation-Free Human Sketch Quality AssessmentInternational Journal of Computer Vision10.1007/s11263-024-02001-1132:8(2743-2764)Online publication date: 17-Feb-2024
https://doi.org/10.1007/s11263-024-02001-1
Alzahrani MUsman MJarraya SAnwar SHelmy T(2024)Deep models for multi-view 3D object recognition: a reviewArtificial Intelligence Review10.1007/s10462-024-10941-w57:12Online publication date: 12-Oct-2024
https://doi.org/10.1007/s10462-024-10941-w
Zhang SWang LCui ZWang S(2024)A sketch recognition method based on bi-modal model using cooperative learning paradigmNeural Computing and Applications10.1007/s00521-024-09836-236:23(14275-14290)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00521-024-09836-2
Bandyopadhyay HChowdhury PSain AKoley SXiang TBhunia ASong Y(2024)Do Generalised Classifiers Really Work on Human Drawn Sketches?Computer Vision – ECCV 202410.1007/978-3-031-72992-8_13(217-235)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72992-8_13
Tiwari ABiswas SLladós J(2024)SketchGPT: Autoregressive Modeling for Sketch Generation and RecognitionDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70549-6_25(421-438)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1007/978-3-031-70549-6_25
Ali SAslam NKim DAbbas ATufail SAzhar B(2023)Context awareness based Sketch-DeepNet architecture for hand-drawn sketches classification and recognition in AIoTPeerJ Computer Science10.7717/peerj-cs.11869(e1186)Online publication date: 27-Apr-2023
https://doi.org/10.7717/peerj-cs.1186
Chowdhury SSany MAhamed MDas SBadal FDas PTasneem ZHasan MIslam MAli MAbhi SIslam MSarker S(2023)A State-of-the-Art Computer Vision Adopting Non-Euclidean Deep-Learning ModelsInternational Journal of Intelligent Systems10.1155/2023/86746412023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/8674641
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents