research-article

Hidden physics models: : Machine learning of nonlinear partial differential equations

Authors:

George Em KarniadakisAuthors Info & Claims

Volume 357, Issue C

Pages 125 - 141

https://doi.org/10.1016/j.jcp.2017.11.039

Published: 15 March 2018 Publication History

Abstract

While there is currently a lot of enthusiasm about “big data”, useful data is usually “small” and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from small data. In particular, we introduce hidden physics models, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier–Stokes, Schrödinger, Kuramoto–Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data.

References

[1]

Raissi, M.; Perdikaris, P.; Karniadakis, G.E. (2017): Numerical gaussian processes for time-dependent and non-linear partial differential equations. arXiv preprint arXiv:1703.10230.

[2]

S.H. Rudy, S.L. Brunton, J.L. Proctor, J.N. Kutz, Data-driven discovery of partial differential equations, Sci. Adv. 3 (2017).

[3]

J.P. Crutchfield, B.S. McNamara, Equations of motion from a data series, Complex Syst. 1 (1987) 121.

[4]

I.G. Kevrekidis, C.W. Gear, J.M. Hyman, P.G. Kevrekidid, O. Runborg, C. Theodoropoulos, et al., Equation-free, coarse-grained multiscale computation: enabling microscopic simulators to perform system-level analysis, Commun. Math. Sci. 1 (2003) 715–762.

[5]

R. Gonzalez-Garcia, R. Rico-Martinez, I. Kevrekidis, Identification of distributed parameter systems: a neural net based approach, Comput. Chem. Eng. 22 (1998) S965–S968.

[6]

H.U. Voss, P. Kolodner, M. Abel, J. Kurths, Amplitude equations from spatiotemporal binary-fluid convection data, Phys. Rev. Lett. 83 (1999) 3422.

[7]

G. Sugihara, R. May, H. Ye, C.-h. Hsieh, E. Deyle, M. Fogarty, S. Munch, Detecting causality in complex ecosystems, Science 338 (2012) 496–500.

[8]

H. Ye, R.J. Beamish, S.M. Glaser, S.C. Grant, C.-h. Hsieh, L.J. Richards, J.T. Schnute, G. Sugihara, Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling, Proc. Natl. Acad. Sci. 112 (2015) E1569–E1576.

[9]

A.J. Roberts, Model Emergent Dynamics in Complex Systems, SIAM, 2014.

[10]

M.D. Schmidt, R.R. Vallabhajosyula, J.W. Jenkins, J.E. Hood, A.S. Soni, J.P. Wikswo, H. Lipson, Automated refinement and inference of analytical models for metabolic networks, Phys. Biol. 8 (2011).

[11]

B.C. Daniels, I. Nemenman, Automated adaptive inference of phenomenological dynamical models, Nat. Commun. 6 (2015).

[12]

B.C. Daniels, I. Nemenman, Efficient inference of parsimonious phenomenological models of cellular dynamics using s-systems and alternating regression, PLoS ONE 10 (2015).

[13]

A.J. Majda, C. Franzke, D. Crommelin, Normal forms for reduced stochastic climate models, Proc. Natl. Acad. Sci. 106 (2009) 3649–3653.

[14]

D. Giannakis, A.J. Majda, Nonlinear laplacian spectral analysis for time series with intermittency and low-frequency variability, Proc. Natl. Acad. Sci. 109 (2012) 2222–2227.

[15]

I. Mezić, Spectral properties of dynamical systems, model reduction and decompositions, Nonlinear Dyn. 41 (2005) 309–325.

[16]

M. Budišić, R. Mohr, I. Mezić, Applied koopmanism a, Chaos 22 (2012).

[17]

I. Mezić, Analysis of fluid flows via spectral properties of the Koopman operator, Annu. Rev. Fluid Mech. 45 (2013) 357–378.

[18]

S.L. Brunton, B.W. Brunton, J.L. Proctor, E. Kaiser, J.N. Kutz, Chaos as an intermittently forced linear system, Nat. Commun. 8 (2017).

[19]

J. Bongard, H. Lipson, Automated reverse engineering of nonlinear dynamical systems, Proc. Natl. Acad. Sci. 104 (2007) 9943–9948.

[20]

M. Schmidt, H. Lipson, Distilling free-form natural laws from experimental data, Science 324 (2009) 81–85.

[21]

R. Tibshirani, Regression shrinkage and selection via the lasso, J. R. Stat. Soc. B (1996) 267–288.

[22]

S.L. Brunton, J.L. Proctor, J.N. Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. 113 (2016) 3932–3937.

[23]

N.M. Mangan, S.L. Brunton, J.L. Proctor, J.N. Kutz, Inferring biological networks by sparse identification of nonlinear dynamics, IEEE Trans. Mol. Biol. Multi-Scale Commun. 2 (2016) 52–63.

[24]

W.-X. Wang, R. Yang, Y.-C. Lai, V. Kovanis, C. Grebogi, Predicting catastrophes in nonlinear dynamical systems by compressive sensing, Phys. Rev. Lett. 106 (2011).

[25]

H. Schaeffer, R. Caflisch, C.D. Hauck, S. Osher, Sparse dynamics for partial differential equations, Proc. Natl. Acad. Sci. 110 (2013) 6634–6639.

[26]

V. Ozoliņš, R. Lai, R. Caflisch, S. Osher, Compressed modes for variational problems in mathematics and physics, Proc. Natl. Acad. Sci. 110 (2013) 18368–18373.

[27]

A. Mackey, H. Schaeffer, S. Osher, On the compressive spectral method, Multiscale Model. Simul. 12 (2014) 1800–1827.

[28]

S.L. Brunton, J.H. Tu, I. Bright, J.N. Kutz, Compressive sensing and low-rank libraries for classification of bifurcation regimes in nonlinear dynamical systems, SIAM J. Appl. Dyn. Syst. 13 (2014) 1716–1732.

[29]

J.L. Proctor, S.L. Brunton, B.W. Brunton, J. Kutz, Exploiting sparsity and equation-free architectures in complex systems, Eur. Phys. J. Spec. Top. 223 (2014) 2665–2684.

[30]

Z. Bai, T. Wimalajeewa, Z. Berger, G. Wang, M. Glauser, P.K. Varshney, Low-dimensional approach for reconstruction of airfoil data via compressive sensing, AIAA J. 53 (4) (2015) 920–933,.

[31]

Tran, G.; Ward, R. (2016): Exact recovery of chaotic systems from highly corrupted data. arXiv preprint arXiv:1607.01067.

[32]

M. Raissi, P. Perdikaris, G.E. Karniadakis, Inferring solutions of differential equations using noisy multi-fidelity data, J. Comput. Phys. 335 (2017) 736–746.

[33]

M. Raissi, P. Perdikaris, G.E. Karniadakis, Machine learning of linear differential equations using gaussian processes, J. Comput. Phys. 348 (2017) 683–693.

Digital Library

[34]

C.E. Rasmussen, C.K. Williams, Gaussian Processes for Machine Learning, vol. 1, MIT Press, Cambridge, 2006.

[35]

K.P. Murphy, Machine Learning: A Probabilistic Perspective, MIT Press, 2012.

Digital Library

[36]

R.M. Neal, Bayesian Learning for Neural Networks, vol. 118, Springer Science & Business Media, 2012.

[37]

V. Vapnik, The Nature of Statistical Learning Theory, Springer Science & Business Media, 2013.

Digital Library

[38]

B. Schölkopf, A.J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, 2002.

[39]

M.E. Tipping, Sparse Bayesian learning and the relevance vector machine, J. Mach. Learn. Res. 1 (2001) 211–244.

Digital Library

[40]

A. Tikhonov, Solution of incorrectly formulated problems and the regularization method, Soviet Math. Dokl. 4 (1963) 1035–1038.

[41]

A.N. Tikhonov, V.Y. Arsenin, Solutions of Ill-Posed Problems, W.H. Winston, 1977.

[42]

T. Poggio, F. Girosi, Networks for approximation and learning, Proc. IEEE 78 (1990) 1481–1497.

[43]

N. Aronszajn, Theory of reproducing kernels, Trans. Am. Math. Soc. 68 (1950) 337–404.

[44]

S. Saitoh, Theory of Reproducing Kernels and Its Applications, vol. 189, Longman, 1988.

[45]

A. Berlinet, C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics, Springer Science & Business Media, 2011.

[46]

Duvenaud, D.; Lloyd, J.R.; Grosse, R.; Tenenbaum, J.B.; Ghahramani, Z. (2013): Structure discovery in nonparametric regression through compositional kernel search. arXiv preprint arXiv:1302.4922.

[47]

Grosse, R.; Salakhutdinov, R.R.; Freeman, W.T.; Tenenbaum, J.B. (2012): Exploiting compositionality to explore a large space of model structures. arXiv preprint arXiv:1210.4856.

[48]

G. Malkomes, C. Schaff, R. Garnett, Bayesian optimization for automated model selection, in: Advances in Neural Information Processing Systems, pp. 2900–2908.

[49]

R. Calandra, J. Peters, C.E. Rasmussen, M.P. Deisenroth, Manifold Gaussian processes for regression, in: 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, pp. 3338–3345.

[50]

Raissi, M.; Karniadakis, G. (2016): Deep multi-fidelity Gaussian processes. arXiv preprint arXiv:1604.07484.

[51]

D.C. Liu, J. Nocedal, On the limited memory BFGS method for large scale optimization, Math. Program. 45 (1989) 503–528.

[52]

C.E. Rasmussen, Z. Ghahramani, Occam's razor, Adv. Neural Inf. Process. Syst. (2001) 294–300.

[53]

A.M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer. 19 (2010) 451–559.

[54]

E. Snelson, Z. Ghahramani, Sparse Gaussian processes using pseudo-inputs, in: Advances in Neural Information Processing Systems, pp. 1257–1264.

[55]

J. Hensman, N. Fusi, N.D. Lawrence, Gaussian processes for big data, in: Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI 2013, Bellevue, WA, USA, August 11–15, 2013.

[56]

Raissi, M. (2017): Parametric gaussian process regression for big data. arXiv preprint arXiv:1704.03144.

[57]

C. Basdevant, M. Deville, P. Haldenwang, J. Lacroix, J. Ouazzani, R. Peyret, P. Orlandi, A. Patera, Spectral and finite difference solutions of the Burgers equation, Comput. Fluids 14 (1986) 23–41.

[58]

Dauxois, T. (2008): Fermi, Pasta, Ulam and a mysterious lady. arXiv preprint arXiv:0801.1590.

[59]

J.M. Hyman, B. Nicolaenko, The Kuramoto–Sivashinsky equation: a bridge between pde's and dynamical systems, Physica D 18 (1986) 113–126.

[60]

B.I. Shraiman, Order, disorder, and phase turbulence, Phys. Rev. Lett. 57 (1986) 325.

[61]

B. Nicolaenko, B. Scheurer, R. Temam, Some global dynamical properties of the Kuramoto–Sivashinsky equations: nonlinear stability and attractors, Physica D 16 (1985) 155–183.

[62]

J.N. Kutz, S.L. Brunton, B.W. Brunton, J.L. Proctor, Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems, vol. 149, SIAM, 2016.

[63]

K. Taira, T. Colonius, The immersed boundary method: a projection approach, J. Comput. Phys. 225 (2007) 2118–2137.

Digital Library

[64]

T. Colonius, K. Taira, A fast immersed boundary method using a nullspace approach and multi-domain far-field boundary conditions, Comput. Methods Appl. Mech. Eng. 197 (2008) 2131–2146.

[65]

I. Podlubny, Fractional Differential Equations: An Introduction to Fractional Derivatives, Fractional Differential Equations, to Methods of Their Solution and Some of Their Applications, vol. 198, Academic Press, 1998.

[66]

J. Nolan, Stable Distributions: Models for Heavy-Tailed Data, Birkhauser, New York, 2003.

[67]

J.M. Chambers, C.L. Mallows, B. Stuck, A method for simulating stable random variables, J. Am. Stat. Assoc. 71 (1976) 340–344.

[68]

A. Weron, R. Weron, Computer simulation of Lévy α-stable variables and processes, in: Chaos—The Interplay Between Stochastic and Deterministic Behaviour, 1995, pp. 379–392.

[69]

J. Hartikainen, S. Särkkä, Kalman filtering and smoothing solutions to temporal Gaussian process regression models, in: 2010 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), IEEE, pp. 379–384.

Cited By

Schneider TAltintas IAtkins D(2024)Accelerating Scientific Discovery With AI-Aided AutomationComputing in Science and Engineering10.1109/MCSE.2024.335245125:5(27-30)Online publication date: 19-Apr-2024
https://dl.acm.org/doi/10.1109/MCSE.2024.3352451
Werner JBeisswanger PSchürger CKlaiber MTheissler A(2024)From Data to WisdomProcedia Computer Science10.1016/j.procs.2023.10.113225:C(1251-1260)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.10.113
Hu ZShukla KKarniadakis GKawaguchi K(2024)Tackling the curse of dimensionality with physics-informed neural networksNeural Networks10.1016/j.neunet.2024.106369176:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106369
Show More Cited By

Index Terms

Hidden physics models: Machine learning of nonlinear partial differential equations
1. Applied computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Deep hidden physics models: deep learning of nonlinear partial differential equations

We put forth a deep learning approach for discovering nonlinear partial differential equations from scattered and potentially noisy observations in space and time. Specifically, we approximate the unknown solution as well as the nonlinear dynamics by ...
Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations

We introduce the concept of numerical Gaussian processes, which we define as Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Numerical Gaussian processes, by construction, ...
Mittag--Leffler Euler Integrator for a Stochastic Fractional Order Equation with Additive Noise

Motivated by fractional derivative models in viscoelasticity, a class of semilinear stochastic Volterra integro-differential equations, and their deterministic counterparts, are considered. A generalized exponential Euler method, named here the Mittag--...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Computational Physics

Journal of Computational Physics Volume 357, Issue C

Mar 2018

375 pages

ISSN:0021-9991

Issue’s Table of Contents

Elsevier Inc.

Publisher

Academic Press Professional, Inc.

United States

Publication History

Published: 15 March 2018

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

122
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schneider TAltintas IAtkins D(2024)Accelerating Scientific Discovery With AI-Aided AutomationComputing in Science and Engineering10.1109/MCSE.2024.335245125:5(27-30)Online publication date: 19-Apr-2024
https://dl.acm.org/doi/10.1109/MCSE.2024.3352451
Werner JBeisswanger PSchürger CKlaiber MTheissler A(2024)From Data to WisdomProcedia Computer Science10.1016/j.procs.2023.10.113225:C(1251-1260)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.10.113
Hu ZShukla KKarniadakis GKawaguchi K(2024)Tackling the curse of dimensionality with physics-informed neural networksNeural Networks10.1016/j.neunet.2024.106369176:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106369
Xiong WHuang XZhang ZDeng RSun PTian Y(2024)Koopman neural operator as a mesh-free solver of non-linear partial differential equationsJournal of Computational Physics10.1016/j.jcp.2024.113194513:COnline publication date: 15-Sep-2024
https://dl.acm.org/doi/10.1016/j.jcp.2024.113194
Lu HTartakovsky D(2024)Data-driven models of nonautonomous systemsJournal of Computational Physics10.1016/j.jcp.2024.112976507:COnline publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1016/j.jcp.2024.112976
Stephany REarls C(2024) Weak-PDE-LEARNJournal of Computational Physics10.1016/j.jcp.2024.112950506:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.jcp.2024.112950
Johnson DFedkiw R(2024)Addressing discontinuous root-finding for subsequent differentiability in machine learning, inverse problems, and controlJournal of Computational Physics10.1016/j.jcp.2023.112624497:COnline publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1016/j.jcp.2023.112624
Chen YKoohy S(2024)GPT-PINNFinite Elements in Analysis and Design10.1016/j.finel.2023.104047228:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.finel.2023.104047
Pan YGuo JChen YCheng QLi WLiu Y(2024)A fundamental diagram based hybrid framework for traffic flow estimation and prediction by combining a Markovian model with deep learningExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122219238:PEOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122219
Es-haghi MAnitescu CRabczuk T(2024)Methods for enabling real-time analysis in digital twinsComputers and Structures10.1016/j.compstruc.2024.107342297:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.compstruc.2024.107342
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents