Abstract
Modern 3D computer graphics technologies are able to reproduce the dynamics and appearance of real-world environments and phenomena, building on theoretical models in applied mathematics, statistics and physics. These methods are applied in architectural design and visualization, biological imaging and visual effects. Differentiable methods, instead, aim to determine how graphics outputs (that is, the real-world dynamics or appearance) change when the environment changes. We survey this growing body of work and propose a holistic and unified differentiable visual computing pipeline. Differentiable visual computing can be leveraged to efficiently solve otherwise intractable problems in physical inference, optimal control, object detection and scene understanding, computational design, manufacturing, autonomous vehicles and robotics. Any application that can benefit from an understanding of the underlying dynamics of the real world stands to benefit substantially from a differentiable graphics treatment. We draw parallels between the well-established computer graphics pipeline and a unified differentiable graphics pipeline, targeting consumers, practitioners and researchers. The breadth of fields that these pipelines draw upon—and are of interest to—includes the physical sciences, data sciences, vision and graphics, machine learning, and adjacent mathematical and computing communities.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).
Ma, P. et al. DiffAqua: a differentiable computational design pipeline for soft underwater swimmers with shape interpolation. ACM Trans. Graph. 40, 132 (2021).
Liu, S., Li, T., Chen, W. & Li, H. Soft Rasterizer: a differentiable renderer for image-based 3D reasoning. In Proc. 2019 IEEE/CVF International Conference on Computer Vision 7708–7717 (IEEE, 2019).
Murthy, J. K. et al. gradSim: differentiable simulation for system identification and visuomotor control. In International Conference on Learning Representations (2020).
Rempe, D. et al. Contact and human dynamics from monocular video. In Proc. European Conference on Computer Vision (ECCV) (eds Vedaldi, A. et al.) 71–87 (Springer, 2020).
Peng, S. et al. Animatable neural radiance fields for modeling dynamic human bodies. In Proc. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 14294–14303 (IEEE, 2021).
Spielberg, A. et al. Learning-in-the-loop optimization: end-to-end control and co-design of soft robots through learned deep latent representations. In Proc. Advances in Neural Information Processing Systems Vol. 32 (eds Wallach, H. et al.) 8284–8294 (Curran Associates, 2019).
Kazhdan, M., Bolitho, M. & Hoppe, H. Poisson surface reconstruction. In Proc. Fourth Eurographics Symposium on Geometry Processing Vol. 7 (eds Fellner, D. W. et al.) 61–70 (ACM, 2006).
Sigmund, O. & Maute, K. Topology optimization approaches. Struct. Multidiscipl. Optim. 48, 1031–1055 (2013).
Prévost, R., Whiting, E., Lefebvre, S. & Sorkine-Hornung, O. Make it stand: balancing shapes for 3D fabrication. ACM Trans. Graph. 32, 81 (2013).
Bächer, M., Whiting, E., Bickel, B. & Sorkine-Hornung, O. Spin-it: optimizing moment of inertia for spinnable objects. ACM Trans. Graph. 33, 96 (2014).
Banović, M. et al. Algorithmic differentiation of the Open CASCADE technology CAD kernel and its coupling with an adjoint CFD solver. Optim. Methods Softw. 33, 813–828 (2018).
Keeter, M. J. Massively parallel rendering of complex closed-form implicit surfaces. ACM Trans. Graph. 39, 141 (2020).
Cascaval, D. et al. Differentiable 3D CAD programs for bidirectional editing. Comput. Graph. Forum 41, 309–323 (2022).
Liu, L., Gu, J., Zaw Lin, K., Chua, T.-S. & Theobalt, C. Neural sparse voxel fields. Adv. Neural Inf. Process. Syst. 33, 15651–15663 (2020).
Fridovich-Keil, S. et al. Plenoxels: radiance fields without neural networks. In Proc. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 5491–5500 (IEEE Computer Society, 2022).
Lombardi, S. et al. Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph. 38, 65 (2019).
Mildenhall, B. et al. NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 99–106 (2022).
Nicolet, B., Jacobson, A. & Jakob, W. Large steps in inverse rendering of geometry. ACM Trans. Graph 40, 248 (2021).
Groueix, T., Fisher, M., Kim, V. G., Russell, B. & Aubry, M. AtlasNet: a papier-mâché approach to learning 3D surface generation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2018).
Cho, M. et al. Differentiable spline approximations. Adv. Neural Inf. Process. Syst. 34, 20270–20282 (2021).
Chen, A., Xu, Z., Geiger, A., Yu, J. & Su, H. TensoRF: tensorial radiance fields. In Proc. 17th European Conference on Computer Vision (ECCV 2022): Part XXXII (eds Avidan, S. et al.) 333–350 (Springer, 2022).
Müller, T., Evans, A., Schied, C. & Keller, A. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41, 102 (2022).
Sosnovik, I. & Oseledets, I. Neural networks for topology optimization. Russ. J. Numer. Anal. Math. Model. 34, 215–223 (2019).
Zehnder, J., Li, Y., Coros, S. & Thomaszewski, B. NTopo: mesh-free topology optimization using implicit neural representations. In Proc. Advances in Neural Information Processing Systems Vol. 34 (eds Ranzato, M. et al.) 10368–10381 (Curran Associates, 2021).
Bhafikatti, S. Finite Element Analysis (New Age International, 2005).
Vaxman, A., Ben-Chen, M. & Gotsman, C. A multi-resolution approach to heat kernels on discrete surfaces. ACM Trans. Graph 29, 121 (2010).
Sharp, N., Attaiki, S., Crane, K. & Ovsjanikov, M. Diffusionnet: Discretization agnostic learning on surfaces. ACM Trans. Graph. 41, 27 (2022).
Liu, H.-T. D., Kim, V. G., Chaudhuri, S., Aigerman, N. & Jacobson, A. Neural subdivision. ACM Trans. Graph. 39, 124 (2020).
Qi, C. R., Su, H., Mo, K. & Guibas, L. J. PointNet: deep learning on point sets for 3D classification and segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 652–660 (IEEE, 2017).
Qi, C. R., Yi, L., Su, H. & Guibas, L. J. PointNet++: deep hierarchical feature learning on point sets in a metric space. Proc. Advances in Neural Information Processing Systems Vol. 30 (eds. Guyon, I.) (Curran Associates, 2017).
Spielberg, A., Amini, A., Chin, L., Matusik, W. & Rus, D. Co-learning of task and sensor placement for soft robotics. IEEE Robot. Autom. Lett. 6, 1208–1215 (2021).
Wang, Y. et al. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. 38, 146 (2019).
Park, J. J., Florence, P., Straub, J., Newcombe, R. & Lovegrove, S. DeepSDF: learning continuous signed distance functions for shape representation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 165–174 (IEEE, 2019).
Monti, F. et al. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 5115–5124 (IEEE, 2017).
Chang, A. X. et al. ShapeNet: an information-rich 3D model repository. Preprint at https://arxiv.org/abs/1512.03012 (2015).
Yang, G. et al. Pointflow: 3D point cloud generation with continuous normalizing flows. In Proc. IEEE/CVF International Conference on Computer Vision 4541–4550 (IEEE, 2019).
Yang, G., Belongie, S., Hariharan, B. & Koltun, V. Geometry processing with neural fields. In Proc. Advances in Neural Information Processing Systems Vol. 34 (eds Ranzato, M. et al.) 22483–22497 (Curran Associates, 2021).
Wang, Y. & Solomon, J. M. Deep closest point: learning representations for point cloud registration. In Proc. IEEE/CVF International Conference on Computer Vision 3523–3532 (IEEE, 2019).
Teran, J., Sifakis, E., Irving, G. & Fedkiw, R. Robust quasistatic finite elements and flesh simulation. In Proc. 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (eds Hansmann, W. et al.) 181–190 (ACM, 2005).
Bächer, M., Knoop, E. & Schumacher, C. Design and control of soft robots using differentiable simulation. Curr. Robot. Rep 2, 211–221 (2021).
Hu, Y. et al. ChainQueen: a real-time differentiable physical simulator for soft robotics. In Proc. 2019 International Conference on Robotics and Automation (ICRA) 6265–6271 (IEEE, 2019).
Du, T. et al. DiffPD: differentiable projective dynamics. ACM Trans. Graph. 41, 13 (2021).
Qiao, Y., Liang, J., Koltun, V. & Lin, M. Differentiable simulation of soft multi-body systems. In Proc. Advances in Neural Information Processing Systems Vol. 34 (eds Ranzato, M. et al.) 5115–5124 (Curran Associates, 2021).
Dubied, M., Michelis, M. Y., Spielberg, A. & Katzschmann, R. K. Sim-to-real for soft robots using differentiable FEM: recipes for meshing, damping and actuation. IEEE Robot. Autom. Lett. 7, 5015–5022 (2022).
de Avila Belbute-Peres, F., Smith, K., Allen, K., Tenenbaum, J. & Kolter, J. Z. End-to-end differentiable physics for learning and control. Adv. Neural Inf. Process. Syst. 31, 7178–7189 (2018).
Chen, R. T., Rubanova, Y., Bettencourt, J. & Duvenaud, D. K. Neural ordinary differential equations. In Proc. Advances in Neural Information Processing Systems Vol. 34 (eds Bengio, S. et al.) 6572–6583 (Curran Associates, 2018).
Long, Z., Lu, Y., Ma, X. & Dong, B. PDE-Net: learning PDEs from data. In Proc. International Conference on Machine Learning (eds Dy, J. & Krause, A.) 3208–3216 (PMLR, 2018).
Sanchez-Gonzalez, A. et al. Learning to simulate complex physics with graph networks. In Proc. International Conference on Machine Learning (eds Daumé III, H. & Singh, A.) 8459–8468 (PMLR, 2020).
Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. & Battaglia, P. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations (2020).
Degrave, J., Hermans, M. & Dambre, J. et al. A differentiable physics engine for deep learning in robotics. Front. Neurorobot. 13, 6 (2019).
Freeman, C. D. et al. Brax—a differentiable physics engine for large scale rigid body simulation. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021).
Werling, K., Omens, D., Lee, J., Exarchos, I. & Liu, C. K. Fast and feature-complete differentiable physics for articulated rigid bodies with contact. In Proc. Conference of Robotics: Science and Systems (RSS) (2021); https://doi.org/10.15607/RSS.2021.XVII.034
Spielberg, A., Du, T., Hu, Y., Rus, D. & Matusik, W. Advanced soft robot modeling in ChainQueen. Robotica 41, 74–104 (2021).
Qiao, Y.-L., Liang, J., Koltun, V. & Lin, M. C. Scalable differentiable physics for learning and control. In Proc. 37th International Conference on Machine Learning (eds Daumé III, H. & Singh, A.) 7847–7856 (PMLR, 2020).
Raissi, M., Yazdani, A. & Karniadakis, G. E. Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations. Science 367, 1026–1030 (2020).
Wiewel, S., Becher, M. & Thuerey, N. Latent space physics: towards learning the temporal evolution of fluid flow. Comput. Graph. Forum 38, 71–82 (2019).
Hu, Y. et al. QuanTaichi: a compiler for quantized simulations. ACM Trans. Graph. 40, 182 (2021).
Li, W. & Todorov, E. Iterative linear quadratic regulator design for nonlinear biological movement systems. In Proc. 1st International Conference on Informatics in Control, Automation and Robotics 222–229 (IEEE, 2004).
Posa, M., Cantu, C. & Tedrake, R. A direct method for trajectory optimization of rigid bodies through contact. Int. J. Robot. Res. 33, 69–81 (2014).
Giftthaler, M., Neunert, M., Stäuble, M. & Buchli, J. The control toolbox—an open-source C++ library for robotics, optimal and model predictive control. In Proc. 2018 IEEE International Conference on Simulation, Modeling and Programming for Autonomous Robots (SIMPAR) (eds Kurniawati, H. et al.) 123–129 (IEEE, 2018).
Tedrake, R. & the Drake Development Team. Drake: a planning, control and analysis toolbox for nonlinear dynamical systems (MIT, 2016); http://drake.mit.edu
Agrawal, A. et al. Differentiable convex optimization layers. In Proc. Advances in Neural Information Processing Systems Vol. 32 (eds Wallach, H. et al.) 9562–9574 (Curran Associates, 2019).
Amos, B., Jimenez, I., Sacks, J., Boots, B. & Kolter, J. Z. Differentiable MPC for end-to-end planning and control. In Proc. 32nd International Conference on Neural Information Processing Systems Vol. 34 (eds Bengio, S. et al.) 8299–8310 (Curran Associates, 2018).
Pfrommer, S., Halm, M. & Posa, M. ContactNets: learning discontinuous contact dynamics with smooth, implicit representations. In Proc. Conference on Robot Learning (eds Faust, A. et al.) 2279–2291 (PMLR, 2021).
Xu, J. et al. Accelerated policy learning with parallel differentiable simulation. In Proc. International Conference on Learning Representations (2022).
Ajay, A. et al. Combining physical simulators and object-based networks for control. In Proc. 2019 International Conference on Robotics and Automation (ICRA) 3217–3223 (IEEE, 2019).
Hwangbo, J. et al. Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, eaau5872 (2019).
Ilyas, A. et al. A closer look at deep policy gradients. In International Conference on Learning Representations (2020).
Metz, L., Freeman, C. D., Schoenholz, S. S. & Kachman, T. Gradients are not all you need. Preprint at https://arxiv.org/abs/2111.05803 (2021).
Suh, H. J., Simchowitz, M., Zhang, K. & Tedrake, R. Do differentiable simulators give better policy gradients? In Proc. International Conference on Machine Learning 20668–20696 (PMLR, 2022).
Pharr, M., Jakob, W. & Humphreys, G. Physically Based Rendering: from Theory to Implementation (Morgan Kaufman, 2016).
Dudchik, Y. I. & Kolchevsky, N. A microcapillary lens for X-rays. Nucl. Instrum. Methods Phys. Res. A 421, 361–364 (1999).
Li, T.-M., Aittala, M., Durand, F. & Lehtinen, J. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 222 (2018).
Loubet, G., Holzschuch, N. & Jakob, W. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 228 (2019).
Zhang, C., Miller, B., Yan, K., Gkioulekas, I. & Zhao, S. Path-space differentiable rendering. ACM Trans. Graph. 39, 143 (2020).
Bangaru, S. P. et al. Systematically differentiating parametric discontinuities. ACM Trans. Graph. 40, 107 (2021).
Kato, H., Ushiku, Y. & Harada, T. Neural 3D mesh renderer. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 3907–3916 (IEEE, 2018).
Hu, Y., Li, T.-M., Anderson, L., Ragan-Kelley, J. & Durand, F. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Trans. Graph. 38, 201 (2019).
Hu, Y. et al. DiffTaichi: differentiable programming for physical simulation. In International Conference on Learning Representations (2020).
Johnson, J. et al. Accelerating 3D deep learning with PyTorch3D. ACM Digital Library https://doi.org/10.1145/3415263.3419160 (2020).
Bailey, P. et al. Differentiable graphics with TensorFlow 2.0. In ACM SIGGRAPH 2019 Courses 1–211 (ACM, 2019).
Jakob, W., Speierer, S., Roussel, N. & Vicini, D. Dr.JIT: a just-in-time compiler for differentiable rendering. ACM Trans. Graph. 41, 124 (2022).
Tewari, A. et al. Advances in neural rendering. Comput. Graph. Forum 41, 703–735 (W2022).
Gkioulekas, I., Zhao, S., Bala, K., Zickler, T. & Levin, A. Inverse volume rendering with material dictionaries. ACM Trans. Graph. 32, 162 (2013).
LeGendre, C. et al. DeepLight: learning illumination for unconstrained mobile mixed reality. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 5918–5928 (IEEE, 2019).
Weiss, S. & Westermann, R. Differentiable direct volume rendering. IEEE Trans. Vis. Comput. Graph. 28, 562–572 (2021).
Vicini, D., Speierer, S. & Jakob, W. Path replay backpropagation: differentiating light paths using constant memory and linear time. ACM Trans. Graph. 40, 108 (2021).
Shi, L. et al. Match: Differentiable material graphs for procedural material capture. ACM Trans. Graph 39, 196 (2020).
Habermann, M., Xu, W., Zollhofer, M., Pons-Moll, G. & Theobalt, C. DeepCap: monocular human performance capture using weak supervision. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 5052–5063 (IEEE, 2020).
Ritchie, D., Thomas, A., Hanrahan, P. & Goodman, N. Neurally-guided procedural models: amortized inference for procedural graphics programs using neural networks. In Proc. Advances in Neural Information Processing Systems Vol. 29 (eds Lee, D. et al.) (Curran Associates, 2016).
AlQuraishi, M. & Sorger, P. K. Differentiable biology: using deep learning for biophysics-based and data-driven modeling of molecular mechanisms. Nat. Methods 18, 1169–1180 (2021).
Schoenholz, S. & Cubuk, E. D. JAX, MD: a framework for differentiable physics. Proc. Advances in Neural Information Processing Systems Vol. 33 (eds Larochelle, H. et al.) (Curran Associates, 2020).
Innes, M. et al. A differentiable programming system to bridge machine learning and scientific computing. Preprint at https://arxiv.org/abs/1907.07587 (2019).
Moses, W. S. et al. Reverse-mode automatic differentiation and optimization of GPU kernels via enzyme. In Proc. International Conference for High Performance Computing, Networking, Storage and Analysis 1–16 (ACM, 2021).
Bettencourt, J., Johnson, M. J. & Duvenaud, D. Taylor-mode automatic differentiation for higher-order derivatives in JAX. Preprint at https://openreview.net/forum?id=SkxEF3FNPH (2019).
Ragan-Kelley, J. et al. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. ACM SIGPLAN Notices 48, 519–530 (2013).
Mullapudi, R. T., Adams, A., Sharlet, D., Ragan-Kelley, J. & Fatahalian, K. Automatically scheduling halide image processing pipelines. ACM Trans. Graph. 35, 1–11 (2016).
Griewank, A. & Walther, A. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (Society for Industrial and Applied Mathematics, 2008).
Acknowledgements
We thank A. Mordvintsev, A. Tagliasacchi, J. Valentin, O. Sanseviero and S. Bouaziz for early discussions on content and structure. This work was supported by a UKRI Future Leaders Fellowship (grant number G104084).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Machine Intelligence thanks Jinwei Ye, Thomas Mueller, and the other, anonymous, reviewer for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Spielberg, A., Zhong, F., Rematas, K. et al. Differentiable visual computing for inverse problems and machine learning. Nat Mach Intell 5, 1189–1199 (2023). https://doi.org/10.1038/s42256-023-00743-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s42256-023-00743-0