Cited By
View all- Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
Joint Sampling and Optimisation for Inverse Rendering
Article No.: 29, Pages 1 - 10
Abstract
When dealing with difficult inverse problems such as inverse rendering, using Monte Carlo estimated gradients to optimise parameters can slow down convergence due to variance. Averaging many gradient samples in each iteration reduces this variance trivially. However, for problems that require thousands of optimisation iterations, the computational cost of this approach rises quickly.
We derive a theoretical framework for interleaving sampling and optimisation. We update and reuse past samples with low-variance finite-difference estimators that describe the change in the estimated gradients between each iteration. By combining proportional and finite-difference samples, we continuously reduce the variance of our novel gradient meta-estimators throughout the optimisation process. We investigate how our estimator interlinks with Adam and derive a stable combination.
We implement our method for inverse path tracing and demonstrate how our estimator speeds up convergence on difficult optimisation tasks.
Supplemental Material
PDF File
Supplementary
- Download
- 423.34 KB
References
[1]
Sai Praveen Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph. 39, 6, Article 245 (nov 2020), 18 pages. https://doi.org/10.1145/3414685.3417833
[2]
Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal Reservoir Resampling for Real-Time Ray Tracing with Dynamic Direct Lighting. ACM Trans. Graph. 39, 4, Article 148 (aug 2020), 17 pages. https://doi.org/10.1145/3386569.3392481
[3]
Wesley Chang, Venkataram Sivaram, Derek Nowrouzezahrai, Toshiya Hachisuka, Ravi Ramamoorthi, and Tzu-Mao Li. 2023. Parameter-Space ReSTIR for Differentiable and Inverse Rendering. In ACM SIGGRAPH 2023 Conference Proceedings (Los Angeles, CA, USA) (SIGGRAPH ’23). Association for Computing Machinery, New York, NY, USA, Article 18, 10 pages. https://doi.org/10.1145/3588432.3591512
[4]
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. 2014. SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.). Vol. 27. Curran Associates, Inc.https://proceedings.neurips.cc/paper_files/paper/2014/file/ede7e2b6d13a41ddf9f4bdef84fdc737-Paper.pdf
[5]
Xi Deng, Fujun Luan, Bruce Walter, Kavita Bala, and Steve Marschner. 2022. Reconstructing Translucent Objects Using Differentiable Rendering. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 38, 10 pages. https://doi.org/10.1145/3528233.3530714
[6]
E. C. Fieller and H. O. Hartley. 1954. Sampling with Control Variables. Biometrika 41, 3/4 (1954), 494–501. http://www.jstor.org/stable/2332729
[7]
Robert M. Gower, Mark Schmidt, Francis Bach, and Peter Richtárik. 2020. Variance-Reduced Methods for Machine Learning. Proc. IEEE 108, 11 (Nov 2020), 1968–1983. https://doi.org/10.1109/JPROC.2020.3028013
[8]
Alex Graves. 2014. Generating Sequences With Recurrent Neural Networks.
[9]
Homan Igehy. 1999. Tracing Ray Differentials. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., USA, 179–186. https://doi.org/10.1145/311535.311555
[10]
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022. Mitsuba 3 renderer. https://mitsuba-renderer.org.
[11]
Rie Johnson and Tong Zhang. 2013. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction. In Advances in Neural Information Processing Systems, C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (Eds.). Vol. 26. Curran Associates, Inc.https://proceedings.neurips.cc/paper_files/paper/2013/file/ac1dd209cbcc5e5d1c6e28598e8cbbe8-Paper.pdf
[12]
James T. Kajiya. 1986. The Rendering Equation. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’86). Association for Computing Machinery, New York, NY, USA, 143–150. https://doi.org/10.1145/15922.15902
[13]
R. E. Kalman. 1960. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering 82, 1 (Mar 1960), 35–45. https://doi.org/10.1115/1.3662552
[14]
Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-Domain Path Tracing. ACM Trans. Graph. 34, 4, Article 123 (jul 2015), 13 pages. https://doi.org/10.1145/2766997
[15]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization.
[16]
Jaakko Lehtinen, Tero Karras, Samuli Laine, Miika Aittala, Frédo Durand, and Timo Aila. 2013. Gradient-Domain Metropolis Light Transport. ACM Trans. Graph. 32, 4, Article 95 (jul 2013), 12 pages. https://doi.org/10.1145/2461912.2461943
[17]
Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. 37, 6, Article 222 (dec 2018), 11 pages. https://doi.org/10.1145/3272127.3275109
[18]
Marco Manzi, Markus Kettunen, Frédo Durand, Matthias Zwicker, and Jaakko Lehtinen. 2016. Temporal Gradient-Domain Path Tracing. ACM Trans. Graph. 35, 6, Article 246 (dec 2016), 9 pages. https://doi.org/10.1145/2980179.2980256
[19]
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. 2020. Monte Carlo Gradient Estimation in Machine Learning. J. Mach. Learn. Res. 21, 1, Article 132 (jan 2020), 62 pages.
[20]
Eric Moulines and Francis Bach. 2011. Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning. In Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.). Vol. 24. Curran Associates, Inc.https://proceedings.neurips.cc/paper_files/paper/2011/file/40008b9a5380fcacce3976bf7c08af5b-Paper.pdf
[21]
Yurii Evgen’evich Nesterov. 1983. A method of solving a convex programming problem with convergence rate O(). In Doklady Akademii Nauk, Vol. 269. Russian Academy of Sciences, 543–547.
[22]
Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. ACM Trans. Graph. 40, 6, Article 248 (dec 2021), 13 pages. https://doi.org/10.1145/3478513.3480501
[23]
Baptiste Nicolet, Fabrice Rousselle, Jan Novak, Alexander Keller, Wenzel Jakob, and Thomas Müller. 2023. Recursive Control Variates for Inverse Rendering. ACM Trans. Graph. 42, 4, Article 62 (jul 2023), 13 pages. https://doi.org/10.1145/3592139
[24]
Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM Trans. Graph. 39, 4, Article 146 (aug 2020), 15 pages. https://doi.org/10.1145/3386569.3392406
[25]
Art B Owen. 2013. Monte Carlo theory, methods and examples. (2013).
[26]
Athanasios Papoulis and S Unnikrishna Pillai. 1984. Probability, Random Variables, and Stochastic Processes. McGraw-Hill, 145–149.
[27]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.) (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[28]
Stanislav Pidhorskyi, Timur Bagautdinov, Shugao Ma, Jason Saragih, Gabriel Schwartz, Yaser Sheikh, and Tomas Simon. 2022. Depth of Field Aware Differentiable Rendering. ACM Trans. Graph. 41, 6, Article 190 (nov 2022), 18 pages. https://doi.org/10.1145/3550454.3555521
[29]
B. T. Polyak and A. B. Juditsky. 1992. Acceleration of Stochastic Approximation by Averaging. SIAM Journal on Control and Optimization 30, 4 (Jul 1992), 838–855. https://doi.org/10.1137/0330046
[30]
Fabrice Rousselle, Wojciech Jarosz, and Jan Novák. 2016. Image-Space Control Variates for Rendering. ACM Trans. Graph. 35, 6, Article 169 (dec 2016), 12 pages. https://doi.org/10.1145/2980179.2982443
[31]
Bimal K Sinha, Joachim Hartung, and Guido Knapp. 2011. Statistical Meta-Analysis with Applications. John Wiley & Sons. https://doi.org/10.1002/9780470386347
[32]
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. 2018. Don’t Decay the Learning Rate, Increase the Batch Size.
[33]
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the Importance of Initialization and Momentum in Deep Learning. In Proceedings of the 30th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 28), Sanjoy Dasgupta and David McAllester (Eds.). PMLR, Atlanta, Georgia, USA, 1139–1147. https://proceedings.mlr.press/v28/sutskever13.html
[34]
Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths Using Constant Memory and Linear Time. ACM Trans. Graph. 40, 4, Article 108 (jul 2021), 14 pages. https://doi.org/10.1145/3450626.3459804
[35]
Kai Yan, Christoph Lassner, Brian Budge, Zhao Dong, and Shuang Zhao. 2022. Efficient Estimation of Boundary Integrals for Path-Space Differentiable Rendering. ACM Trans. Graph. 41, 4, Article 123 (jul 2022), 13 pages. https://doi.org/10.1145/3528223.3530080
[36]
Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo Estimators for Differential Light Transport. ACM Trans. Graph. 40, 4, Article 78 (jul 2021), 16 pages. https://doi.org/10.1145/3450626.3459807
[37]
Cheng Zhang, Zhao Dong, Michael Doggett, and Shuang Zhao. 2021. Antithetic Sampling for Monte Carlo Differentiable Rendering. ACM Trans. Graph. 40, 4, Article 77 (jul 2021), 12 pages. https://doi.org/10.1145/3450626.3459783
[38]
Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. 39, 4, Article 143 (aug 2020), 19 pages. https://doi.org/10.1145/3386569.3392383
[39]
Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. 38, 6, Article 227 (nov 2019), 16 pages. https://doi.org/10.1145/3355089.3356522
Index Terms
- Joint Sampling and Optimisation for Inverse Rendering
Recommendations
Amortizing Samples in Physics-Based Inverse Rendering Using ReSTIR
Recently, great progress has been made in physics-based differentiable rendering. Existing differentiable rendering techniques typically focus on static scenes, but during inverse rendering---a key application for differentiable rendering---the scene is ...
Parameter-space ReSTIR for Differentiable and Inverse Rendering
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference ProceedingsDifferentiable rendering is frequently used in gradient descent-based inverse rendering pipelines to solve for scene parameters – such as reflectance or lighting properties – from target image inputs. Efficient computation of accurate, low variance ...
Reparameterizing discontinuous integrands for differentiable rendering
Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
1113 pages
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SA '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 178 of 869 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 407Total Downloads
- Downloads (Last 12 months)343
- Downloads (Last 6 weeks)37
Reflects downloads up to 01 Feb 2025
Other Metrics
Citations
Cited By
View all- Fan ZShi PGuo MFu RGuo YGuo J(2024)Conditional Mixture Path Guiding for Differentiable RenderingACM Transactions on Graphics10.1145/365813343:4(1-11)Online publication date: 19-Jul-2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in