research-article

MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities

Authors:

Vinoth Pandian Sermuga Pandian,

Christian Beecks,

Matthias JarkeAuthors Info & Claims

OzCHI '20: Proceedings of the 32nd Australian Conference on Human-Computer Interaction

Pages 403 - 412

https://doi.org/10.1145/3441000.3441030

Published: 15 February 2021 Publication History

Abstract

Transforming lo-fi UI sketches to higher-fidelities is an expensive, time-consuming process that requires significant rework. In this paper, we systematically research utilizing AI to assist the transformation of lo-fi sketches to higher fidelities. To provide this assistance, we introduce MetaMorph, an AI tool to detect the constituent UI elements of lo-fi sketches. To train MetaMorph, we collected the UISketch dataset that contains 6,785 hand-drawn sketches of 21 UI elements, 201 hand-drawn lo-fi sketches, and 125,000 synthetically generated lo-fi sketches. MetaMorph provides 63.5% mAP for hand-drawn lo-fi sketches and 82.9% mAP for synthetic lo-fi sketches. Results from ASQ indicate that designers experience an above-average satisfaction level towards ease of task completion (4.9), time taken (5.3), and supporting information (5.3) upon utilizing AI assistance for transforming lo-fi sketches. Their qualitative feedback indicates that they perceive utilizing AI as a novel and useful approach to transform lo-fi sketches into higher fidelities.

Supplementary Material

p403-pandian-supplement (p403-pandian-supplement.pdf)

Presentation

Download
2.52 MB

References

[1]

Kumar Ashwin. 2018. Automated front-end development using deep learning. https://blog.insightdatascience.com/automated-front-end-development-using-deep-learning-3169dd086e82.

[2]

Kumar Ashwin. 2018. sketch-code. https://github.com/ashnkumar/sketch-code.

[3]

Ranan B. Banerji and George W. Ernst. 1972. Limitations in Pattern Recognition and Problem Solving. In Proceedings of the ACM Annual Conference - Volume 1 (Boston, Massachusetts, USA) (ACM ’72). ACM, New York, NY, USA, 28–38. https://doi.org/10.1145/800193.805819

[4]

Wilkins Benjamin. 2017. Sketching Interfaces - Generating code from low fidelity wireframes. https://airbnb.design/sketching-interfaces/.

[5]

Plimmer Beryl and Apperley Mark. 2003. Software to sketch interface designs. In Human-Computer Interaction - INTERACT’03. IOS Press, 73–80. https://www.researchgate.net/publication/228891482_Software_to_sketch_interface_designs.

[6]

Anabela Caetano, Neri Goulart, Manuel Fonseca, and Joaquim Jorge. 2002. JavaSketchIt: Issues in Sketching the Look of User Interfaces. In AAAI Spring Symposium on Sketch Understanding. AAAI Press, Menlo Park, 9–14. https://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.3486&rank=1

[7]

Adrien Coyette, Stéphane Faulkner, Manuel Kolp, Quentin Limbourg, and Jean Vanderdonckt. 2004. SketchiXML: Towards a Multi-agent Design Tool for Sketching User Interfaces Based on USIXML. In Proceedings of the 3rd Annual Conference on Task Models and Diagrams (Prague, Czech Republic) (TAMODIA ’04). ACM, New York, NY, USA, 75–82. https://doi.org/10.1145/1045446.1045461

Digital Library

[8]

Adrien Coyette and Jean Vanderdonckt. 2005. A sketching tool for designing anyuser, anyplatform, anywhere user interfaces. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 3585 LNCS (2005), 550–564. https://doi.org/10.1007/11555261_45

[9]

COCO Dataset. 2017. COCO - Common Objects in Context. (Accessed on 09/16/2019).

[10]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology(UIST ’17).

Digital Library

[11]

Douglas K. Van Duyne, James Landay, and Jason I. Hong. 2002. The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

[12]

Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects?ACM Trans. Graph. 31, 4, Article 44 (July 2012), 10 pages. https://doi.org/10.1145/2185520.2185540

Digital Library

[13]

Daniel Engelberg and Ahmed Seffa. 2002. A Framework for Rapid Mid-Fidelity Prototyping of Web Sites. In Proceedings of the IFIP 17th World Computer Congress - TC13 Stream on Usability: Gaining a Competitive Edge. Kluwer, B.V., Deventer, The Netherlands, The Netherlands, 203–215. http://dl.acm.org/citation.cfm?id=646869.709393

Digital Library

[14]

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88, 2 (1 June 2010), 303–338. https://doi.org/10.1007/s11263-009-0275-4

Digital Library

[15]

Manuel J Fonseca, César Pimentel, and Joaquim A Jorge. 2002. CALI: An online scribble recognizer for calligraphic interfaces. https://www.aaai.org/Papers/Symposia/Spring/2002/SS-02-08/SS02-08-008.pdf

[16]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.

Digital Library

[17]

Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019. Swire: Sketch-based User Interface Retrieval(CHI ’19). ACM, New York, NY, USA, Article 104, 10 pages. https://doi.org/10.1145/3290605.3300334

Digital Library

[18]

Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. 2016. Speed/accuracy trade-offs for modern convolutional object detectors. CoRR abs/1611.10012(2016). arxiv:1611.10012http://arxiv.org/abs/1611.10012

[19]

A. Lancaster. 2004. Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces. IEEE Transactions on Professional Communication 47, 4 (Dec. 2004), 335–336. https://doi.org/10.1109/tpc.2004.837973

[20]

James A. Landay. 1996. SILK: Sketching Interfaces Like Krazy. In Conference Companion on Human Factors in Computing Systems (Vancouver, British Columbia, Canada) (CHI ’96). ACM, New York, NY, USA, 398–399. https://doi.org/10.1145/257089.257396

[21]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521 (27 May 2015), 436. https://doi.org/10.1038/nature14539

[22]

James R. Lewis. 1991. Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. ACM SIGCHI Bulletin 23, 1 (Jan. 1991), 78–81. https://doi.org/10.1145/122672.122692

Digital Library

[23]

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal Loss for Dense Object Detection. CoRR abs/1708.02002(2017). arxiv:1708.02002http://arxiv.org/abs/1708.02002

[24]

Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8693 Lncs. 740–755. https://doi.org/10.1007/978-3-319-10602-1_48

[25]

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single Shot MultiBox Detector. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling(Eds.). Springer International Publishing, Cham, 21–37. https://link.springer.com/chapter/10.1007/978-3-319-46448-0_2.

[26]

Google LLC. 2019. Using TFRecords and tf.Example | TensorFlow Core | TensorFlow. https://www.tensorflow.org/tutorials/load_data/tf_records.

[27]

Microsoft. 2018. Sketch2Code - Microsoft/ailab. https://github.com/Microsoft/ailab/tree/master/Sketch2Code.

[28]

Medina Perez and Luis Jorge. 2016. The UsiSketch Software Architecture. Romanian Journal of Human-Computer Interaction 9, 4(2016), 305–333. https://dial.uclouvain.be/pr/boreal/object/boreal:187342

[29]

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. CoRR abs/1506.01497(2015). arxiv:1506.01497http://arxiv.org/abs/1506.01497

[30]

Dean Rubine. 1991. Specifying Gestures by Example. SIGGRAPH Comput. Graph. 25, 4 (July 1991), 329–337. https://doi.org/10.1145/127719.122753

Digital Library

[31]

Jim Rudd, Ken Stern, and Scott Isensee. 1996. Low vs. high-fidelity prototyping debate. https://doi.org/10.1145/223500.223514

[32]

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Trans. Graph. 35, 4, Article 119 (July 2016), 12 pages. https://doi.org/10.1145/2897824.2925954

Digital Library

[33]

Vinícius C. V. B. Segura, Simone D. J. Barbosa, and Fabiana Pedreira Simões. 2012. UISKEI: A Sketch-based Prototyping Tool for Defining and Evaluating User Interface Behavior. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Capri Island, Italy) (Avi ’12). ACM, New York, NY, USA, 18–25. https://doi.org/10.1145/2254556.2254564

Digital Library

[34]

Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2020. Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches (poster - forthcoming). In Proceedings of the 25th International Conference on Intelligent User Interfaces: Companion (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 2.

Digital Library

[35]

C. Spearman. 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology 15, 1 (1904), 72–101. http://www.jstor.org/stable/1412159

[36]

Sarah Suleri, Vinoth Pandian Sermuga Pandian, Svetlana Shishkovets, and Matthias Jarke. 2019. Eve: A Sketch-based Software Prototyping Workbench. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI EA ’19). ACM, New York, NY, USA, Article Lbw1410, 6 pages. https://doi.org/10.1145/3290607.3312994

Digital Library

[37]

Christian Szegedy, Alexander Toshev, and Dumitru Erhan. 2013. Deep Neural Networks for Object Detection. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 (Lake Tahoe, Nevada) (Nips’13). Curran Associates Inc., Usa, 2553–2561. http://dl.acm.org/citation.cfm?id=2999792.2999897

[38]

Beltramelli Tony. 2017. Teaching Machines to Understand User Interfaces. https://hackernoon.com/teaching-machines-to-understand-user-interfaces-5a0cdeb4d579.

[39]

Miriam Walker, Leila Takayama, and James A. Landay. 2002. High-Fidelity or Low-Fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 46, 5(2002), 661–665. https://doi.org/10.1177/154193120204600513 arXiv:https://doi.org/10.1177/154193120204600513

Cited By

Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Stige ÅZamani EMikalef PZhu Y(2023)Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agendaInformation Technology & People10.1108/ITP-07-2022-051937:6(2324-2352)Online publication date: 29-Aug-2023
https://doi.org/10.1108/ITP-07-2022-0519

Recommendations

UISketch: A Large-Scale Dataset of UI Element Sketches
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We ...
Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches
IUI '20 Companion: Companion Proceedings of the 25th International Conference on Intelligent User Interfaces

User Interface design is an iterative process that progresses through low-, medium-, and high-fidelity prototypes. A few research projects use deep learning to automate this process by transforming low fidelity (lo-fi) sketches into front-end code. ...
SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

User Interface (UI) prototyping is an iterative process where designers initially sketch UIs before transforming them into interactive digital designs. Recent research applies Deep Neural Networks (DNNs) to identify the constituent UI elements of these ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

OzCHI '20: Proceedings of the 32nd Australian Conference on Human-Computer Interaction

December 2020

764 pages

ISBN:9781450389754

DOI:10.1145/3441000

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

OzCHI '20

OzCHI '20: 32nd Australian Conference on Human-Computer-Interaction

December 2 - 4, 2020

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 362 of 729 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
246
Total Downloads

Downloads (Last 12 months)78
Downloads (Last 6 weeks)6

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shi YGao TJiao XCao N(2023)Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature ReviewProceedings of the ACM on Human-Computer Interaction10.1145/36102177:CSCW2(1-35)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610217
Stige ÅZamani EMikalef PZhu Y(2023)Artificial intelligence (AI) for user experience (UX) design: a systematic literature review and future research agendaInformation Technology & People10.1108/ITP-07-2022-051937:6(2324-2352)Online publication date: 29-Aug-2023
https://doi.org/10.1108/ITP-07-2022-0519

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents