research-article

Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition

Authors:

Liang ZhangAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 1055 - 1065

https://doi.org/10.1145/3581783.3612115

Published: 27 October 2023 Publication History

Abstract

With the recent surge in the use of touchscreen devices, free-hand sketching has emerged as a promising modality for human-computer interaction. While previous research has focused on tasks such as recognition, retrieval, and generation of familiar everyday objects, this study aims to create a Sketch Input Method Editor (SketchIME) specifically designed for a professional Command, Control, Communications, Computer, and Intelligence (C4I) system. Within this system, sketches are utilized as low-fidelity prototypes for recommending standardized symbols in the creation of comprehensive situation maps. This paper also presents a systematic dataset comprising 374 specialized sketch types, and proposes a simultaneous recognition and segmentation architecture with multilevel supervision between recognition and segmentation to improve performance and enhance interpretability. By incorporating few-shot domain adaptation and class-incremental learning, the network's ability to adapt to new users and extend to new task-specific classes is significantly enhanced. Results from experiments conducted on both the proposed dataset and the SPG dataset illustrate the superior performance of the proposed architecture. Our dataset and code are publicly available at https://github.com/GuangmingZhu/SketchIME.

References

[1]

Andreas Bulling, Raimund Dachselt, Andrew Duchowski, Robert Jacob, Sophie Stellmach, and Veronica Sundstedt. 2012. Gaze interaction in the post-WIMP world. In CHI. 1221--1224.

[2]

Jungwoo Choi, Heeryon Cho, Jinjoo Song, and Sang Min Yoon. 2019a. Sketchhelper: Real-time stroke guidance for freehand sketch retrieval. IEEE TMM, Vol. 21, 8 (2019), 2083--2092.

[3]

Jungwoo Choi, Heeryon Cho, Jinjoo Song, and Sang Min Yoon. 2019b. SketchHelper: Real-Time Stroke Guidance for Freehand Sketch Retrieval. IEEE TMM, Vol. 21, 8 (2019), 2083--2092.

[4]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv preprint arXiv:1412.3555 (2014).

Digital Library

[5]

Ayan Das, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. 2021. Cloud2curve: Generation and vectorization of parametric sketches. In CVPR. 7088--7097.

[6]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929 (2021).

[7]

Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM TOG, Vol. 31, 4 (2012), 1--10.

Digital Library

[8]

Danilo Gasques, Janet G Johnson, Tommy Sharkey, and Nadir Weibel. 2019. What you sketch is what you get: Quick and easy augmented reality prototyping with pintar. In CHI. 1--6.

[9]

David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In ICLR.

[10]

Jun-Yan He, Xiao Wu, Yu-Gang Jiang, Bo Zhao, and Qiang Peng. 2017. Sketch recognition with deep visual-sequential fusion model. In ACM MM. 448--456.

[11]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.

[12]

Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. 2022. Constrained Few-shot Class-incremental Learning. In CVPR. 9057--9067.

[13]

Rui Hu and John Collomosse. 2013. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. CVIU, Vol. 117, 7 (2013), 790--806.

Digital Library

[14]

Forrest Huang, John F Canny, and Jeffrey Nichols. 2019. Swire: Sketch-based user interface retrieval. In CHI. 1--10.

Digital Library

[15]

Qi Jia, Meiyu Yu, Xin Fan, and Haojie Li. 2017. Sequential dual deep learning with shape and texture features for sketch recognition. arXiv preprint arXiv:1708.02716 (2017).

[16]

Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019. Deepgcns: Can gcns go as deep as cnns?. In CVPR. 9267--9276.

[17]

Ke Li, Kaiyue Pang, Jifei Song, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Honggang Zhang. 2018b. Universal sketch perceptual grouping. In ECCV. 582--597.

[18]

Lei Li, Hongbo Fu, and Chiew-Lan Tai. 2018a. Fast sketch segmentation and labeling with deep learning. IEEE CG&A, Vol. 39, 2 (2018), 38--51.

[19]

Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. 2018. Conditional adversarial domain adaptation. In NeurIPS. 1647--1657.

[20]

Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).

[21]

Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE TKDE, Vol. 22, 10 (2009), 1345--1359.

[22]

Ameya Prabhu, Vishal Batchu, Sri Aurobindo Munagala, Rohit Gajawada, and Anoop Namboodiri. 2018. Distribution-aware binarization of neural networks for sketch recognition. In WACV. 830--838.

[23]

Alan Preciado-Grijalva and Venkata Santosh Sai Ramireddy Muthireddy. 2021. Evaluation of Deep Neural Network Domain Adaptation Techniques for Image Recognition. arXiv preprint arXiv:2109.13420 (2021).

[24]

Yonggang Qi and Zheng-Hua Tan. 2019. SketchSegNet: An end-to-end learning of RNN for multi-class sketch semantic segmentation. IEEE Access, Vol. 7 (2019), 102717--102726.

[25]

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM TOG, Vol. 35, 4 (2016), 1--12.

Digital Library

[26]

Ravi Kiran Sarvadevabhatla, Isht Dwivedi, Abhijat Biswas, Sahil Manocha, and Venkatesh Babu R. 2017a. SketchParse: Towards Rich Descriptions for Poorly Drawn Sketches Using Multi-Task Hierarchical Deep Networks. In ACMMM. 10--18.

[27]

Ravi Kiran Sarvadevabhatla and Jogendra Kundu. 2016. Enabling my robot to play pictionary: Recurrent neural networks for sketch recognition. In ACM MM. 247--251.

[28]

Ravi Kiran Sarvadevabhatla, Sudharshan Suresh, and R. Venkatesh Babu. 2017b. Object Category Understanding via Eye Fixations on Freehand Sketches. IEEE TIP, Vol. 26, 5 (2017), 2508--2518.

[29]

Sarah Suleri, Vinoth Pandian Sermuga Pandian, Svetlana Shishkovets, and Matthias Jarke. 2019. Eve: A sketch-based software prototyping workbench. In CHI. 1--6.

[30]

Baochen Sun, Jiashi Feng, and Kate Saenko. 2017. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, Gabriela Csurka (Ed.). Springer, 153--171.

[31]

Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In ECCV. Springer, 443--450.

[32]

Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. 2020. Few-shot class-incremental learning. In CVPR. 12183--12192.

[33]

Caglar Tirkaz, Berrin Yanikoglu, and T. Metin Sezgin. 2012. Sketched symbol recognition with auto-completion. PR, Vol. 45, 11 (2012), 3926--3937.

[34]

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014).

[35]

Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. In CVPR. 1875--1883.

[36]

Fei Wang, Shujin Lin, Hefeng Wu, Hanhui Li, Ruomei Wang, Xiaonan Luo, and Xiangjian He. 2019a. Spfusionnet: Sketch segmentation using multi-modal data fusion. In ICME. 1654--1659.

[37]

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2019b. Dynamic graph cnn for learning on point clouds. ACM TOG, Vol. 38, 5 (2019), 1--12.

Digital Library

[38]

Xingyuan Wu, Yonggang Qi, Jun Liu, and Jie Yang. 2018. Sketchsegnet: A rnn model for labeling sketch strokes. In MLSP. 1--6.

[39]

Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. 2023. Deep learning for free-hand sketch: A survey. IEEE TPAMI, Vol. 45, 1 (2023), 285--312.

[40]

Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, Zhanyu Ma, and Jun Guo. 2018. Sketchmate: Deep hashing for million-scale human sketch retrieval. In CVPR. 8090--8098.

[41]

Peng Xu, Chaitanya K Joshi, and Xavier Bresson. 2022. Multigraph transformer for free-hand sketch recognition. IEEE TNNLS, Vol. 33, 10 (2022), 5150--5161.

[42]

Lumin Yang, Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, and Youyi Zheng. 2021. Sketchgnn: Semantic sketch segmentation with graph neural networks. ACM TOG, Vol. 40, 3 (2021), 1--13.

Digital Library

[43]

Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. 2017. Sketch-a-net: A deep neural network that beats humans. IJCV, Vol. 122, 3 (2017), 411--425.

Digital Library

[44]

Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. 2015. Sketch-a-Net that Beats Humans. In BMVC. 1--12.

[45]

Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. 2021. Few-shot incremental learning with continually evolved classifiers. In CVPR. 12455--12464.

[46]

An Zhao, Mingyu Ding, Zhiwu Lu, Tao Xiang, Yulei Niu, Jiechao Guan, and Ji-Rong Wen. 2021. Domain-adaptive few-shot learning. In WACV. 1390--1399.

[47]

Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. 2022a. Forward compatible few-shot class-incremental learning. In CVPR. 9046--9056.

[48]

Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. 2022b. Few-shot class-incremental learning by sampling multi-phase tasks. IEEE TPAMI (2022). https://doi.org/10.1109/TPAMI.2022.3200865

Digital Library

[49]

Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha. 2021. Self-promoted prototype refinement for few-shot class-incremental learning. In CVPR. 6801--6810.

Cited By

Liu JWang SZhu GZhang LLi NGao E(2023)Content-Conditioned Generation of Stylized Free-Hand Sketches2023 International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD)10.1109/ICSMD60522.2023.10490999(1-6)Online publication date: 2-Nov-2023
https://doi.org/10.1109/ICSMD60522.2023.10490999

Index Terms

Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques

Recommendations

How do humans sketch objects?

Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze ...
‘CADSketchNet’ - An Annotated Sketch dataset for 3D CAD Model Retrieval with Deep Neural Networks▪
Highlights
- The goal of this paper is to create a sketch dataset that is suitable for developing deep learning-based solutions to the problem of search and retrieval in 3D CAD models.
- A sketch dataset of query images, called ‘CADSketchNet’ has ...
Graphical abstract

Display Omitted

Abstract
Ongoing advancements in the fields of 3D modelling and digital archiving have led to an outburst in the amount of data stored digitally. Consequently, several retrieval systems have been developed depending on the type of data stored in these ...
Hand-drawn face sketch recognition by humans and a PCA-based algorithm for forensic applications
Special issue on recent advances in biometrics

Because face sketches represent the original faces in a very concise yet recognizable form, they play an important role in criminal investigations, human visual perception, and face biometrics. In this paper, we compared the performances of humans and a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Chinese Defense Advance Research Program
National Natural Science Foundation of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
109
Total Downloads

Downloads (Last 12 months)109
Downloads (Last 6 weeks)4

Reflects downloads up to 19 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu JWang SZhu GZhang LLi NGao E(2023)Content-Conditioned Generation of Stylized Free-Hand Sketches2023 International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD)10.1109/ICSMD60522.2023.10490999(1-6)Online publication date: 2-Nov-2023
https://doi.org/10.1109/ICSMD60522.2023.10490999

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents