research-article

Paparazzi: surface editing by way of multi-view image processing

Authors:

Hsueh-Ti Derek Liu,

Alec JacobsonAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 6

Article No.: 221, Pages 1 - 11

https://doi.org/10.1145/3272127.3275047

Published: 04 December 2018 Publication History

Abstract

The image processing pipeline boasts a wide variety of complex filters and effects. Translating an individual effect to operate on 3D surface geometry inevitably results in a bespoke algorithm. Instead, we propose a general-purpose back-end optimization that allows users to edit an input 3D surface by simply selecting an off-the-shelf image processing filter. We achieve this by constructing a differentiable triangle mesh renderer, with which we can back propagate changes in the image domain to the 3D mesh vertex positions. The given image processing technique is applied to the entire shape via stochastic snapshots of the shape: hence, we call our method Paparazzi. We provide simple yet important design considerations to construct the Paparazzi renderer and optimization algorithms. The power of this rendering-based surface editing is demonstrated via the variety of image processing filters we apply. Each application uses an off-the-shelf implementation of an image processing method without requiring modification to the core Paparazzi algorithm.

Supplementary Material

MP4 File (a221-liu.mp4)

Download
68.97 MB

References

[1]

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274--2282.

Digital Library

[2]

Marco Attene. 2010. A lightweight approach to repairing digitized polygon meshes. The visual computer 26, 11 (2010), 1393--1406.

Digital Library

[3]

Marco Attene. 2016. As-exact-as-possible repair of unprintable STL files. arXiv preprint arXiv:1605.07829 (2016).

[4]

Gavin Barill, Neil Dickson, Ryan Schmidt, David I.W. Levin, and Alec Jacobson. 2018. Fast Winding Numbers for Soups and Clouds. ACM Transactions on Graphics (2018).

Digital Library

[5]

Jonathan T Barron and Jitendra Malik. 2015. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence 37, 8 (2015), 1670--1687.

[6]

Léon Bottou, Frank E Curtis, and Jorge Nocedal. 2016. Optimization methods for large-scale machine learning. arXiv preprint arXiv:1606.04838 (2016).

[7]

Tyson Brochu and Robert Bridson. 2009. Robust topological operations for dynamic explicit surfaces. SIAM Journal on Scientific Computing 31, 4 (2009), 2472--2493.

Digital Library

[8]

Gyeongmin Choe, Jaesik Park, Yu-Wing Tai, and In So Kweon. 2017. Refining Geometry from Depth Sensors using IR Shading Images. Inter. Journal of Computer Vision (2017).

[9]

Amaël Delaunoy and Emmanuel Prados. 2011. Gradient flows for optimizing triangular mesh-based surfaces: Applications to 3d reconstruction problems dealing with visibility. International journal of computer vision 95, 2 (2011), 100--123.

Digital Library

[10]

Timothy Dozat. 2016. Incorporating nesterov momentum into adam. (2016).

[11]

Jérémie Dumas, An Lu, Sylvain Lefebvre, Jun Wu, and Christian Dick. 2015. By-example synthesis of structurally sound patterns. ACM Transactions on Graphics (TOG) 34, 4 (2015), 137.

Digital Library

[12]

Jihad El-Sana, Elvir Azanli, and Amitabh Varshney. 1999. Skip strips: maintaining triangle strips for view-dependent rendering. In Proceedings of the conference on Visualization'99: celebrating ten years. IEEE Computer Society Press, 131--138.

Digital Library

[13]

Pau Gargallo, Emmanuel Prados, and Peter Sturm. 2007. Minimizing the reprojection error in surface reconstruction from images. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1--8.

[14]

Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. 2414--2423.

[15]

Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. 2018. Unsupervised Training for 3D Morphable Model Regression. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]

Yotam Gingold and Denis Zorin. 2008. Shading-based surface editing. ACM Transactions on Graphics (TOG) 27, 3 (2008), 95.

Digital Library

[17]

Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Transactions on Graphics (TOG) 32, 6 (2013), 162.

Digital Library

[18]

Henri Gouraud. 1971. Continuous shading of curved surfaces. IEEE transactions on computers 100, 6 (1971), 623--629.

Digital Library

[19]

Xianfeng Gu, Steven J Gortler, and Hugues Hoppe. 2002. Geometry images. ACM Transactions on Graphics (TOG) 21, 3 (2002), 355--361.

Digital Library

[20]

Kaiming He and Jian Sun. 2015. Fast Guided Filter. CoRR abs/1505.00996 (2015). http://arxiv.org/abs/1505.00996

[21]

Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided image filtering. In European conference on computer vision. Springer, 1--14.

Digital Library

[22]

Lei He and Scott Schaefer. 2013. Mesh denoising via L 0 minimization. ACM Transactions on Graphics (TOG) 32, 4 (2013), 64.

Digital Library

[23]

Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. 2001. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 327--340.

Digital Library

[24]

Hugues Hoppe. 1997. View-dependent refinement of progressive meshes. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 189--198.

Digital Library

[25]

Alec Jacobson, Daniele Panozzo, et al. 2016. libigl: A simple C++ geometry processing library. (2016). http://libigl.github.io/libigl/.

[26]

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Neural 3D Mesh Renderer. CoRR abs/1711.07566 (2017). arXiv:1711.07566 http://arxiv.org/abs/1711.07566

[27]

Bertrand Kerautret, Xavier Granier, and Achille Braquelaire. 2005. Intuitive shape modeling by shading design. In International Symposium on Smart Graphics. Springer, 163--174.

Digital Library

[28]

Felix Knöppel, Keenan Crane, Ulrich Pinkall, and Peter Schröder. 2015. Stripe Patterns on Surfaces. ACM Trans. Graph. (2015).

Digital Library

[29]

Leif Kobbelt and Mario Botsch. 2004. A survey of point-based techniques in computer graphics. Computers & Graphics 28, 6 (2004), 801--814.

Digital Library

[30]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.

Digital Library

[31]

Y-K Lai, S-M Hu, DX Gu, and Ralph R Martin. 2005. Geometric texture synthesis and transfer via geometry images. In Proc. SPM.

Digital Library

[32]

Eric Landreneau and Scott Schaefer. 2010. Scales and Scale-like Structures. Comput. Graph. Forum (2010).

[33]

Peter Lindstrom and Greg Turk. 2000. Image-driven simplification. ACM Transactions on Graphics (ToG) 19, 3 (2000), 204--241.

Digital Library

[34]

Baoxia Liu and James T Todd. 2004. Perceptual biases in the interpretation of 3D shape from shading. Vision research (2004).

[35]

Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2280--2288.

[36]

Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer, 154--169.

[37]

David Luebke and Carl Erikson. 1997. View-dependent simplification of arbitrary polygonal environments. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 199--208.

Digital Library

[38]

David Luebke and Benjamin Hallen. 2001. Perceptually driven simplification for interactive rendering. In Rendering Techniques 2001. Springer, 223--234.

Digital Library

[39]

Stephen R. Marschner and Donald P. Greenberg. 1997. Inverse Lighting for Photography. In Color and Imaging Conference.

[40]

Roy Or-El, Rom Hershkovitz, Aaron Wetzler, Guy Rosman, Alfred M. Bruckstein, and Ron Kimmel. 2016. Real-Time Depth Refinement for Specular Objects. In Proc. CVPR.

[41]

Celal Ozturk, Emrah Hancer, and Dervis Karaboga. 2014. Color image quantization: a short review and an application with artificial bee colony algorithm. Informatica 25, 3 (2014), 485--503.

Digital Library

[42]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).

[43]

Bui Tuong Phong. 1975. Illumination for computer generated pictures. Commun. ACM 18, 6 (1975), 311--317.

Digital Library

[44]

Emmanuel Prados and Olivier Faugeras. 2006. Shape from shading. Handbook of mathematical models in computer vision (2006), 375--388.

[45]

Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-processing Framework for Inverse Rendering. In Proc. SIGGRAPH.

Digital Library

[46]

Nadia Robertini, Dan Casas, Edilson Aguiar, and Christian Theobalt. 2017. Multi-view Performance Capture of Surface Details. Int. J. Comput. Vision (2017).

Digital Library

[47]

Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016).

[48]

Ryan Schmidt and Karan Singh. 2010. Meshmixer: an interface for rapid mesh composition. In ACM SIGGRAPH 2010 Talks. ACM, 6.

Digital Library

[49]

Christian Schüller, Daniele Panozzo, and Olga Sorkine-Hornung. 2014. Appearance-mimicking Surfaces. ACM Trans. Graph. (2014).

Digital Library

[50]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[51]

Kenshi Takayama, Ryan Schmidt, Karan Singh, Takeo Igarashi, Tamy Boubekeur, and Olga Sorkine. 2011. Geobrush: Interactive mesh geometry cloning. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 613--622.

[52]

Elif Tosun, Yotam I Gingold, Jason Reisman, and Denis Zorin. 2007. Shape optimization using reflection lines. In Proc. SGP.

Digital Library

[53]

Greg Turk. 1991. Generating textures on arbitrary surfaces using reaction-diffusion. Siggraph (1991).

Digital Library

[54]

Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. 2014. scikit-image: image processing in Python. PeerJ 2 (2014), e453.

[55]

CWAM Van Overveld. 1996. Painting gradients: Free-form surface design using shading patterns. In Graphics Interface, Vol. 96. 151--158.

Digital Library

[56]

Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. Siggraph (2001).

Digital Library

[57]

Martin Weier, Michael Stengel, Thorsten Roth, Piotr Didyk, Elmar Eisemann, Martin Eisemann, Steve Grogorick, André Hinkenjann, Ernst Kruijff, Marcus Magnor, et al. 2017. Perception-driven Accelerated Rendering. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 611--643.

Digital Library

[58]

Nathaniel Williams, David Luebke, Jonathan D Cohen, Michael Kelley, and Brenden Schubert. 2003. Perceptually guided simplification of lit, textured meshes. In Proceedings of the 2003 symposium on Interactive 3D graphics. ACM, 113--121.

Digital Library

[59]

Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical engineering 19, 1 (1980), 191139.

[60]

Chenglei Wu, Kiran Varanasi, Yebin Liu, Hans-Peter Seidel, and Christian Theobalt. 2011. Shading-based Dynamic Shape Refinement from Multi-view Video Under General Illumination. In Proc. ICCV.

Digital Library

[61]

Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, and Christian Theobalt. 2014. Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 6 (2014), 200.

Digital Library

[62]

Julie C Xia and Amitabh Varshney. 1996. Dynamic view-dependent simplification for polygonal models. In Proceedings of the 7th conference on Visualization'96. IEEE Computer Society Press, 327-ff.

Digital Library

[63]

Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L 0 gradient minimization. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 174.

Digital Library

[64]

Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. 1999. Shape-from-shading: a survey. IEEE transactions on pattern analysis and machine intelligence 21, 8 (1999), 690--706.

Digital Library

[65]

Wangyu Zhang, Bailin Deng, Juyong Zhang, Sofien Bouaziz, and Ligang Liu. 2015. Guided mesh normal filtering. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 23--34.

Digital Library

[66]

Shuang Zhao. 2014. Modeling and rendering fabrics at micron-resolution. Cornell University.

Cited By

Furukawa RKawasaki HSagawa R(2025)Incremental shape integration with inter‐frame shape consistency using neural SDF for a 3D endoscopic systemHealthcare Technology Letters10.1049/htl2.7000112:1Online publication date: 30-Jan-2025
https://doi.org/10.1049/htl2.70001
Chen JZhu FHan YRen D(2025)Thermo-mesh transformer network for generalizable three-dimensional temperature prediction with mechanism-based representationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2025.110274145(110274)Online publication date: Apr-2025
https://doi.org/10.1016/j.engappai.2025.110274
Lipp LHahn DEcormier-Nocca PRist FWimmer M(2024)View-Independent Adjoint Light Tracing for Lighting Design OptimizationACM Transactions on Graphics10.1145/366218043:3(1-16)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1145/3662180
Show More Cited By

Index Terms

Paparazzi: surface editing by way of multi-view image processing
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
      1. Mesh models

Recommendations

Sharp features on multiresolution subdivision surfaces
Pacific graphics 2001

In this paper we describe a method for creating sharp features and trim regions on multiresolution subdivision surfaces along a set of user-defined curves. Operations such as engraving, embossing, and trimming are important in many surface modeling ...
Towards flattenable mesh surfaces

In many industries, products are constructed by assembled surface patches in @?^3, where each patch is expected to have an isometric map to a corresponding region in @?^2. The widely investigated developable surfaces in differential geometry show this ...
An introduction to image-based rendering
Integrated image and graphics technologies

In this chapter, we review the techniques for image-based rendering. Unlike traditional 3D computer graphics in which 3D geometry of the scene is known, image-based rendering (IBR) techniques render novel views directly from input images. IBR techniques ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 6

December 2018

1401 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3272127

Editor:
Takeo Igarashi
The University of Tokyo, Japan

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2018

Published in TOG Volume 37, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Adobe Systems Inc
NSERC DAS
Autodesk Inc
NSERC Discovery
Canada Research Chairs Program
Fields Institute
Connaught Funds

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
342
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)3

Reflects downloads up to 15 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Furukawa RKawasaki HSagawa R(2025)Incremental shape integration with inter‐frame shape consistency using neural SDF for a 3D endoscopic systemHealthcare Technology Letters10.1049/htl2.7000112:1Online publication date: 30-Jan-2025
https://doi.org/10.1049/htl2.70001
Chen JZhu FHan YRen D(2025)Thermo-mesh transformer network for generalizable three-dimensional temperature prediction with mechanism-based representationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2025.110274145(110274)Online publication date: Apr-2025
https://doi.org/10.1016/j.engappai.2025.110274
Lipp LHahn DEcormier-Nocca PRist FWimmer M(2024)View-Independent Adjoint Light Tracing for Lighting Design OptimizationACM Transactions on Graphics10.1145/366218043:3(1-16)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1145/3662180
Wang CJiang RChai MHe MChen DLiao J(2024)NeRF-Art: Text-Driven Neural Radiance Fields StylizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328340030:8(4983-4996)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3283400
Zhang GTao Y(2024)Construction of Artistic Intelligence System based on Multi-View Image Generation Algorithm2024 International Conference on Inventive Computation Technologies (ICICT)10.1109/ICICT60155.2024.10545003(1287-1293)Online publication date: 24-Apr-2024
https://doi.org/10.1109/ICICT60155.2024.10545003
Furukawa RSagawa ROka SKawasaki H(2024)NeRF-based multi-frame 3D integration for 3D endoscopy using active stereo2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)10.1109/EMBC53108.2024.10782699(1-5)Online publication date: 15-Jul-2024
https://doi.org/10.1109/EMBC53108.2024.10782699
Hamdi AMelas-Kyriazi LMai JQian GLiu RVondrick CGhanem BVedaldi A(2024)GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01873(19812-19822)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01873
Furukawa RChen ESagawa ROka SKawasaki H(2024)Calibration‐free structured‐light‐based 3D scanning system in laparoscope for robotic surgeryHealthcare Technology Letters10.1049/htl2.12083Online publication date: 8-Mar-2024
https://doi.org/10.1049/htl2.12083
Xie JCheng LWang GHu MYu ZDu MNing X(2024)Fusing differentiable rendering and language–image contrastive learning for superior zero-shot point cloud classificationDisplays10.1016/j.displa.2024.10277384(102773)Online publication date: Sep-2024
https://doi.org/10.1016/j.displa.2024.102773
Nishihara KFurukawa RSagawa RKawasaki H(2024)Multiple Active Stereo Systems Calibration Method Based on Neural SDF Using DSSS for Wide Area 3D ReconstructionComputer Vision – ACCV 202410.1007/978-981-96-0969-7_10(158-174)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1007/978-981-96-0969-7_10
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents