Abstract
Mouse and keyboard interfaces handle traditional text-based queries, and standard search engines provide for effective text-based search. However, everyday documents are filled with not only text, but photos, cartoons, diagrams, and sketches. These images can often be easier to recall than the surrounding text. In an effort to make human computer interaction handle more forms of human-human interaction, sketching has recently become an important means of interacting with computer systems. We propose extending the traditional monomodal model of text-based search to include the capabilities of sketch-based search. Our goal is to create a sketch-based search that can find documents from a single query sketch. We imagine an important use for this technology would be to allow users to search a computerized laboratory notebook for a previously drawn sketch. Because such as sketch will have initially been drawn only a single time, it is important that the search-by-sketch system (1) recognize a wide range of shapes that are not necessarily geometric nor drawn in the same way each time, (2) recognize a query example from only one initial training example, and (3) learn from successful queries to improve accuracy over time. We present here such an algorithm. To test the algorithm, we implemented a proof-of-concept-system: MARQS, a system that uses sketches to query existing media albums. Preliminary results show that the system yielded an average search rank of 1.51, indicating that the correct sketch is presented as either the top or second search result on average.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Gross MD, Do EY-L (1996) Ambiguous intentions: a paper-like interface for creative design. In: UIST ’96: proceedings of the 9th annual ACM symposium on user interface software and technology. ACM, New York, pp 183–192
Gross MD, Do EYL (1996) Demonstrating the electronic cocktail napkin: a paper-like interface for early design. In: CHI ’96: conference companion on human factors in computing systems. ACM, New York, pp 5–6
Hammond T, Davis R (2005) Ladder, a sketching language for user interface developers. Comput Graph 29(4):518–532
Long AC, Landay JA, Rowe LA (2001) “Those look similar!” issues in automating gesture design advice. In: PUI ’01: proceedings of the 2001 workshop on perceptive user interfaces. ACM, New York, pp 1–5
Morrel-Samuels P (1990) Clarifying the distinction between lexical and gestural commands. Int J Man-Mach Stud 32(5):581–590
Chris Long JA, Landay JA, Rowe LA, Michiels J (2000) Visual similarity of pen gestures. In: CHI ’00: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 360–367
Rubine D (1991) Specifying gestures by example. In: SIGGRAPH ’91: proceedings of the 18th annual conference on computer graphics and interactive techniques. ACM, New York, pp 329–337
Fonseca MJ, Pimentel C, Jorge JA (2002) Cali: an online scribble recognizer for calligraphic interfaces, In: AAAI spring symposium on sketch understanding, pp 51–58
Kara LB, Stahovich TF (2004) An image-based trainable symbol recognizer for sketch-based interfaces. In: AAAI fall symposium series 2004: making pen-based interaction intelligent and natural, pp 99–105
Sun Z, Jiang E, Sun J (2005) Adaptive online multi-stroke sketch recognition based on hidden Markov model. In: Lecture notes in artificial intelligences, vol 3784. Springer, Berlin, pp 948–957
Sutherland IE (1988) Sketchpad a man-machine graphical communication system. In: 25 years of DAC: papers on twenty-five years of electronic design automation. ACM, New York, pp 507–524
Sezgin TM, Stahovich T, Davis R (2001) Sketch based interfaces: early processing for sketch understanding. In: PUI ’01: proceedings of the 2001 workshop on perceptive user interfaces. ACM, New York, pp 1–8
Yu B, Cai S (2003) A domain-independent system for sketch recognition. In: GRAPHITE ’03: proceedings of the 1st international conference on computer graphics and interactive techniques in Australasia and South East Asia. ACM, New York, pp 141–146
Paulson B, Hammond T (2008) Paleosketch: Accurate primitive sketch recognition and beautification. In: IUI ’08: proceedings of the 2008 international conference on intelligent user interfaces. pp 1–8
Kato T, Kurita T, Otsu N, Hirata K (1992) A sketch retrieval method for full color image databases—query by visual example. In: 11th IAPA international conference on pattern recognition, pp 530–533
Gennari LM, Kara LB, Stahovich TF (2004) Combining geometry and domain knowledge to interpret hand-drawn diagrams. In: AAAI fall symposium series 2004: making pen-based interaction intelligent and natural, pp 547–562
Leung WH, Chen T (2002) Retrieval of sketches based on spatial relation between strokes. In: Proceedings of the 2002 international conference on image processing, pp I-908–I-911
Saund E, Fleet D, Larner D, Mahoney J (2003) Perceptually-supported image editing of text and graphics. In: UIST ’03: proceedings of the 16th annual ACM symposium on user interface software and technology. ACM, New York, pp 183–192
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Paulson, B., Hammond, T. MARQS: retrieving sketches learned from a single example using a dual-classifier. J Multimodal User Interfaces 2, 3–11 (2008). https://doi.org/10.1007/s12193-008-0006-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-008-0006-0