research-article

SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image

Authors:

Hongbo FuAuthors Info & Claims

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

Pages 387 - 399

https://doi.org/10.1145/3126594.3126611

Published: 20 October 2017 Publication History

Abstract

The creation of 3D contents still remains one of the most crucial problems for the emerging applications such as 3D printing and Augmented Reality. In Augmented Reality, how to create virtual contents that seamlessly overlay with the real environment is a key problem for human-computer interaction and many subsequent applications. In this paper, we present a sketch-based interactive tool, which we term emph{SweepCanvas}, for rapid exploratory 3D modeling on top of an RGB-D image. Our aim is to offer end-users a simple yet efficient way to quickly create 3D models on an image. We develop a novel sketch-based modeling interface, which takes a pair of user strokes as input and instantly generates a curved 3D surface by sweeping one stroke along the other. A key enabler of our system is an optimization procedure that extracts pairs of spatial planes from the context to position and sweep the strokes. We demonstrate the effectiveness and power of our modeling system on various RGB-D data sets and validate the use cases via a pilot study.

Supplementary Material

suppl.mov (uistf1973-file3.mp4)

Supplemental video

Download
59.31 MB

References

[1]

Bae, S.-H., Balakrishnan, R., and Singh, K. Ilovesketch: As-natural-as-possible sketching system for creating 3d curve models. In UIST '08 (2008), 151--160.

Digital Library

[2]

Bae, S.-H., Balakrishnan, R., and Singh, K. Everybodylovessketch: 3d sketching for a broader audience. In UIST '09 (2009), 59--68.

Digital Library

[3]

Canny, J. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, 6 (1986), 679--698.

Digital Library

[4]

Casarrubias-Vargas, H., Petrilli-Barceló, A., and Bayro-Corrochano, E. Fast edge detection in rgb-d images. In Iberoamerican Congress on Pattern Recognition, Springer (2014), 868--875.

[5]

Chaudhuri, S., and Koltun, V. Data-driven suggestions for creativity support in 3d modeling. ACM Transactions on Graphics (TOG) 29, 6 (2010), 183.

Digital Library

[6]

Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 3-sweep: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6 (Nov. 2013), 195:1--195:10.

Digital Library

[7]

Chen, X., Kang, S. B., Xu, Y.-Q., Dorsey, J., and Shum, H.-Y. Sketching reality: Realistic interpretation of architectural designs. ACM Trans. Graph. 27, 2 (May 2008), 11:1--11:15.

Digital Library

[8]

Cheng, Y. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence 17, 8 (1995), 790--799.

Digital Library

[9]

Choi, S., Zhou, Q.-Y., Miller, S., and Koltun, V. A large dataset of object scans. arXiv:1602.02481 (2016).

[10]

Cooper, M. Line Drawing Interpretation. Springer-Verlag London, 2008.

[11]

De Paoli, C., and Singh, K. Secondskin: Sketch-based construction of layered 3d models. ACM Trans. Graph. 34, 4 (July 2015), 126:1--126:10.

Digital Library

[12]

Dollár, P., and Zitnick, C. L. Fast edge detection using structured forests. IEEE transactions on pattern analysis and machine intelligence 37, 8 (2015), 1558--1570.

[13]

Dorsey, J., Xu, S., Smedresman, G., Rushmeier, H., and McMillan, L. The mental canvas: A tool for conceptual architectural design and analysis. In PG '07 (2007).

Digital Library

[14]

Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4 (July 2012), 31:1--31:10.

Digital Library

[15]

Engel, J., Koltun, V., and Cremers, D. Direct sparse odometry. In arXiv:1607.02565 (July 2016).

[16]

Favreau, J.-D., Lafarge, F., and Bousseau, A. Line drawing interpretation in a multi-view context. In CVPR (2015).

[17]

Gannon, M., Grossman, T., and Fitzmaurice, G. Tactum: a skin-centric approach to digital design and fabrication. In CHI '15 (2015), 1779--1788.

Digital Library

[18]

Gingold, Y., Igarashi, T., and Zorin, D. Structured annotations for 2D-to-3D modeling. ACM Transactions on Graphics (TOG) 28, 5 (2009), 148.

Digital Library

[19]

Google, Inc. Tilt brush. 2015.

[20]

Holz, D., Holzer, S., Rusu, R. B., and Behnke, S. Real-time plane segmentation using rgb-d cameras. In Robot Soccer World Cup, Springer (2011), 306--317.

Digital Library

[21]

Huo, K., Vinayak, and Ramani, K. Window-shaping: 3d design ideation in mixed reality. In SUI '16 (2016), 189--189.

Digital Library

[22]

Igarashi, T., and Hughes, J. F. A suggestive interface for 3d drawing. In UIST '01 (2001), 173--181.

Digital Library

[23]

Igarashi, T., Matsuoka, S., and Tanaka, H. Teddy: A sketching interface for 3d freeform design. In SIGGRAPH '99 (1999), 409--416.

Digital Library

[24]

Isack, H., and Boykov, Y. Energy-based geometric multi-model fitting. Int. J. Comput. Vision 97, 2 (Apr. 2012), 123--147.

Digital Library

[25]

Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In UIST '11 (2011), 559--568.

Digital Library

[26]

Janoch, A., Karayev, S., Jia, Y., Barron, J. T., Fritz, M., Saenko, K., and Darrell, T. A category-level 3d object dataset: Putting the kinect to work. In Consumer Depth Cameras for Computer Vision. Springer, 2013, 141--165.

[27]

Jones, W., and Sagoo, N. Architects' Sketchbooks. Thames and Hudson, 2011.

[28]

Kallio, K. 3D6B Editor: Projective 3D Sketching with Line-Based Rendering. In Eurographics Workshop on Sketch-Based Interfaces and Modeling (2005).

[29]

Kim, Y., and Bae, S.-H. Sketchingwithhands: 3d sketching handheld products with first-person hand posture. In UIST '16 (2016), 797--808.

Digital Library

[30]

Kolmogorov, V., and Zabih, R. What energy functions can be minimized via graph cuts? In ECCV '02 (2002), 65--81.

Digital Library

[31]

Lau, M., Saul, G., Mitani, J., and Igarashi, T. Modeling-in-context: User design of complementary objects with a single photo. In Proc. SBIM (2010), 17--24.

Digital Library

[32]

Lipp, M., Wonka, P., and Müller, P. Pushpull++. ACM Transactions on Graphics (TOG) 33, 4 (2014), 130.

Digital Library

[33]

Lipson, H., and Shpitalni, M. Optimization-based reconstruction of a 3d object from a single freehand line drawing. In ACM SIGGRAPH 2007 courses, ACM (2007), 45.

Digital Library

[34]

Liu, F., Shen, C., Lin, G., and Reid, I. Learning depth from single monocular images using deep convolutional neural fields. IEEE transactions on pattern analysis and machine intelligence 38, 10 (2016), 2024--2039.

Digital Library

[35]

Monszpart, A., Mellado, N., Brostow, G. J., and Mitra, N. J. Rapter: Rebuilding man-made scenes with regular arrangements of planes. ACM Trans. Graph. 34, 4 (July 2015), 103:1--103:12.

Digital Library

[36]

Nathan Silberman, Derek Hoiem, P. K., and Fergus, R. Indoor segmentation and support inference from rgbd images. In ECCV (2012).

Digital Library

[37]

Nealen, A., Igarashi, T., Sorkine, O., and Alexa, M. FiberMesh: Designing freeform surfaces with 3D curves. ACM Transactions on Graphics 26, 3 (2007), article no. 41.

Digital Library

[38]

Nuernberger, B., Ofek, E., Benko, H., and Wilson, A. D. Snaptoreality: Aligning augmented reality to the real world. In CHI '16 (2016), 1233--1244.

Digital Library

[39]

Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge, J. A. Sketch-based modeling: A survey. Computers & Graphics 33, 1 (2009), 85--103.

Digital Library

[40]

Owada, S., Nielsen, F., Nakazawa, K., and Igarashi, T. A sketching interface for modeling the internal structures of 3d shapes. In ACM SIGGRAPH 2007 courses, ACM (2007), 38.

Digital Library

[41]

öztireli, A. C., Uyumaz, U., Popa, T., Sheffer, A., and Gross, M. 3d modeling with a symmetric sketch. EG (August 2011).

Digital Library

[42]

Paczkowski, P., Dorsey, J., Rushmeier, H., and Kim, M. H. Paper3d: Bringing casual 3d modeling to a multi-touch interface. In UIST '14 (2014), 23--32.

Digital Library

[43]

Paczkowski, P., Kim, M. H., Morvan, Y., Dorsey, J., Rushmeier, H., and O'Sullivan, C. Insitu: Sketching architectural designs in context. ACM TOG (SIGGRAPH Asia) 30, 6 (2011), 182:1--10.

Digital Library

[44]

Sachs, E., Roberts, A., and Stoops, D. 3draww: A tool for designing 3d shapes. IEEE Comput. Graph. Appl. 11, 6 (Nov. 1991), 18--26.

Digital Library

[45]

Sadri, B., and Singh, K. Flow-complex-based shape reconstruction from 3d curves. ACM Trans. Graph. 33, 2 (Apr. 2014), 20:1--20:15.

Digital Library

[46]

Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. Analytic drawing of 3d scaffolds. In ACM TOG (SIGGRAPH Asia), vol. 28 (2009), 149.

Digital Library

[47]

Schnabel, R., Wahl, R., and Klein, R. Efficient ransac for point-cloud shape detection. Computer Graphics Forum 26, 2 (June 2007), 214--226.

[48]

Shao, C., Bousseau, A., Sheffer, A., and Singh, K. Crossshade: Shading concept sketches using cross-section curves. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings) 31, 4 (2012).

Digital Library

[49]

Shao, T., Li, W., Zhou, K., Xu, W., Guo, B., and Mitra, N. J. Interpreting concept sketches. ACM TOG (SIGGRAPH) 32, 4 (2013).

Digital Library

[50]

Shao, T., Monszpart, A., Zheng, Y., Koo, B., Xu, W., Zhou, K., and Mitra, N. J. Imagining the unseen: Stability-based cuboid arrangements for scene understanding. ACM Transactions on Graphics 33, 6 (2014), 209:1--209:11.

Digital Library

[51]

Shao, T., Xu, W., Zhou, K., Wang, J., Li, D., and Guo, B. An interactive approach to semantic modeling of indoor scenes with an rgbd camera. ACM Transactions on Graphics (TOG) 31, 6 (2012), 136.

Digital Library

[52]

Shen, C.-H., Fu, H., Chen, K., and Hu, S.-M. Structure recovery by part assembly. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2012 31, 6 (2012), 180:1--180:11.

Digital Library

[53]

Shesh, A., and Chen, B. Smartpaper: An interactive and user friendly sketching system. In Computer Graphics Forum, vol. 23, Wiley Online Library (2004), 301--310.

[54]

Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. Geosemantic snapping for sketch-based modeling. Computer Graphics Forum 32, 2 (2013), 245--253.

[55]

Snavely, N., Seitz, S. M., and Szeliski, R. Photo tourism: Exploring photo collections in 3d. In SIGGRAPH Conference Proceedings, ACM Press (New York, NY, USA, 2006), 835--846.

Digital Library

[56]

Song, S., and Xiao, J. Sliding shapes for 3d object detection in depth images. In European Conference on Computer Vision, Springer (2014), 634--651.

[57]

Thormählen, T., and Seidel, H.-P. 3d-modeling by ortho-image generation from image sequences. In ACM Transactions on Graphics (TOG), vol. 27, ACM (2008), 86.

Digital Library

[58]

Trimble, Inc. SketchUp. 2017.

[59]

Xie, X., Xu, K., Mitra, N. J., Cohen-Or, D., Gong, W., Su, Q., and Chen, B. Sketch-to-design: Context-based part assembly. Comput. Graph. Forum 32, 8 (2013), 233--245.

[60]

Xu, B., Chang, W., Sheffer, A., Bousseau, A., McCrae, J., and Singh, K. True2form: 3d curve networks from 2d sketches via selective regularization. ACM TOG (SIGGRAPH) 33, 4 (2014).

Digital Library

[61]

Xu, K., Chen, K., Fu, H., Sun, W.-L., and Hu, S.-M. Sketch2scene: Sketch-based co-retrieval and co-placement of 3d models. ACM Transactions on Graphics 32, 4 (2013), 123:1--123:15.

Digital Library

[62]

Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. Sketch: An interface for sketching 3d scenes. In SIGGRAPH '96 (1996), 163--170.

Digital Library

[63]

Zhang, Y., Xu, W., Tong, Y., and Zhou, K. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics (TOG) 34, 5 (2015), 159.

Digital Library

[64]

Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4 (July 2012), 99:1--99:11.

Digital Library

[65]

Zheng, Y., Liu, H., Dorsey, J., and Mitra, M. Smart canvas: Context-inferred interpretation of sketches for preparatory design studies. Computer Graphics Forum (Proc. Eurographics) 35, 2 (2016).

Digital Library

[66]

Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (TOG) 33, 4 (2014), 156.

Digital Library

Cited By

Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Shen YShen YCheng JJiang CFan MWang Z(2024)Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642096(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642096
Yu EBlackburn-Matzen KNguyen CWang OHabib Kazi RBousseau A(2023)VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware CanvasesACM Transactions on Graphics10.1145/359241342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592413
Show More Cited By

Index Terms

SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image

Recommendations

NaturaSketch: Modeling from Images and Natural Sketches

Sketching on paper is a quick and easy way to communicate ideas. However, many sketch-based systems require people to draw in contrived ways instead of sketching freely as they would on paper. NaturaSketch affords a more natural interface through ...
Technical Section: Sketch-based modeling: A survey

User interfaces in modeling have traditionally followed the WIMP (Window, Icon, Menu, Pointer) paradigm. Though functional and very powerful, they can also be cumbersome and daunting to a novice user, and creating a complex model requires considerable ...
Progressive sketching with instant previewing
Highlights
- We propose progressive sketching with instant previewing for sketch-based modeling.
Graphical abstract

Display Omitted

Abstract
Sketch-based modeling uses freeform strokes as basic modeling metaphor and the modeling process typically involves two phases: sketching and sculpting. This paper presents a method to enhance the sketching phase to have the features of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '17: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology

October 2017

870 pages

ISBN:9781450349819

DOI:10.1145/3126594

General Chair:
Krzysztof Gajos
Harvard University
,
Program Chairs:
Jennifer Mankoff
Carnegie Mellon University
,
Chris Harrison
Carnegie Mellon University

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research grants council of Hong Kong Special Administrative Region, China
China Young 1000 Talents Program
Shenzhen Innovation Program
National Natural Science Foundation of China

Conference

UIST '17

Sponsor:

UIST '17: The 30th Annual ACM Symposium on User Interface Software and Technology

October 22 - 25, 2017

QC, Québec City, Canada

Acceptance Rates

UIST '17 Paper Acceptance Rate 73 of 324 submissions, 23%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
848
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)6

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Shen YShen YCheng JJiang CFan MWang Z(2024)Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AIProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642096(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642096
Yu EBlackburn-Matzen KNguyen CWang OHabib Kazi RBousseau A(2023)VideoDoodles: Hand-Drawn Animations on Videos with Scene-Aware CanvasesACM Transactions on Graphics10.1145/359241342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592413
Hirzle TMüller FDraxler FSchmitz MKnierim PHornbæk K(2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581072
Wang ZNguyen CAsente PDorsey J(2023)PointShopAR: Supporting Environmental Design Prototyping Using Point Cloud in Augmented RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580776(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580776
Alghofaili RNguyen CKrs VCarr NMĕch RYu L(2023)WARPY: Sketching Environment-Aware 3D Curves in Mobile Augmented Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00052(367-377)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00052
Huang KWang CLiu RChen G(2022)A Fast and Accurate Spatial Target Snapping Method for 3D Scene Modeling and Mapping in Mobile Augmented RealityISPRS International Journal of Geo-Information10.3390/ijgi1101006911:1(69)Online publication date: 17-Jan-2022
https://doi.org/10.3390/ijgi11010069
Tai N(2022)On-Site Architectural Drawing with Hand-held Mobile InstructionsInternational Journal of Information and Education Technology10.18178/ijiet.2022.12.1.157912:1(1-6)Online publication date: 2022
https://doi.org/10.18178/ijiet.2022.12.1.1579
Liu FDeng XSong JLai YLiu YWang HMa CQin SWang H(2022)SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch CompositionACM Transactions on Interactive Intelligent Systems10.1145/354395612:3(1-26)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543956
Kaimoto HMonteiro KFaridan MLi JFarajian SKakehi YNakagaki KSuzuki R(2022)Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UIProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545626(1-12)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545626
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten