research-article

Variational Flow Graphical Model

Authors:

Ping LiAuthors Info & Claims

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 1493 - 1503

https://doi.org/10.1145/3534678.3539450

Published: 14 August 2022 Publication History

Abstract

This paper introduces a novel approach embedding flow-based models in hierarchical structures. The proposed model learns the representation of high-dimensional data via a message-passing scheme by integrating flow-based functions through variational inference. Meanwhile, our model produces a representation of the data using a lower dimension, thus overcoming the drawbacks of many flow-based models, usually requiring a high dimensional latent space involving many trivial variables. With the proposed aggregation nodes, our model provides a new approach for distribution modeling and numerical inference on datasets. Multiple experiments on synthetic and real-world datasets show the benefits of our~proposed~method and potentially broad applications.

References

[1]

James R Anderson and Carsten Peterson. A mean field theory learning algorithm for neural networks. Complex Systems, 1:995--1019, 1987.

[2]

Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862, 2017.

[3]

Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798--1828, 2013.

Digital Library

[4]

Yoshua Bengio, Tristan Deleu, Edward J Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. arXiv preprint arXiv:2111.09266, 2021.

[5]

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.

Digital Library

[6]

Christopher M Bishop, David Spiegelhalter, and John Winn. Vibes: A variational inference engine for bayesian networks. In NeurIPS, pages 793--800, 2003.

[7]

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859--877, 2017.

[8]

Samuel F Buck. A method of estimation of missing values in multivariate data suitable for use with an electronic computer. Journal of the Royal Statistical Society: Series B (Methodological), 22(2):302--306, 1960.

[9]

YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, 2020.

[10]

Adnan Darwiche. A differential approach to inference in bayesian networks. J. ACM, 50(3):280--305, 2003.

Digital Library

[11]

Rina Dechter and Robert Mateescu. AND/OR search spaces for graphical models. Artif. Intell., 171(2--3):73--106, 2007.

[12]

Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Proceedings of the 3rd International Conference on Learning Representations (ICLR Workshop), San Diego, CA, 2015.

[13]

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.

[14]

Guanhua Fang and Ping Li. On variational inference in biclustering models. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 3111--3121, Virtual Event, 2021.

[15]

Zoubin Ghahramani and Matthew J. Beal. Variational inference for bayesian mixtures of factor analysers. In Advances in Neural Information Processing Systems (NIPS), pages 449--455, Denver, CO, 1999.

[16]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014.

Digital Library

[17]

Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural networks: Tricks of the trade, pages 599--619. Springer, 2012.

[18]

Geoffrey E Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Lenny Pitt, editor, Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory (COLT), pages 5--13, Santa Cruz, CA, 1993.

Digital Library

[19]

Manfred Jaeger, Jens Dalgaard Nielsen, and Tomi Silander. Learning probabilistic decision graphs. Int. J. Approx. Reason., 42(1--2):84--100, 2006.

Digital Library

[20]

Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Mach. Learn., 37(2):183--233, 1999.

Digital Library

[21]

Ilyes Khemakhem, Ricardo Pio Monti, Robert Leech, and Aapo Hyvärinen. Causal autoregressive flows. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 3520--3528, Virtual, 2021.

[22]

Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational autoencoders with inverse autoregressive flow. In Advances in Neural Information Processing Systems (NIPS), 2016.

[23]

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, Canada, 2014.

[24]

Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In Proceedings of the Fourteenth International Conference on Principles of Knowledge Representation and Reasoning (KR), 2014.

Digital Library

[25]

Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.

Digital Library

[26]

Frances Y. Kuo, Ian H. Sloan, Grzegorz W. Wasilkowski, and Henryk Wozniakowski. On decompositions of multivariate functions. Math. Comput., 79(270):953--966, 2010.

[27]

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.

[28]

Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. MNIST handwritten digit database.

[29]

Radu Marinescu and Rina Dechter. AND/OR branch-and-bound for graphical models. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 224--229, Edinburgh, Scotland, UK, 2005.

[30]

Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Pranav Subramani, Nicola Di Mauro, Pascal Poupart, and Kristian Kersting. SPFlow: An easy and extensible library for deep probabilistic learning using sum-product networks. arXiv preprint arXiv:1901.03704, 2019.

[31]

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run MCMC toward energy-based model. In Advances in Neural Information Processing (NeurIPS), pages 5233--5243, 2019.

[32]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. J. Mach. Learn. Res., 12:2825--2830, 2011.

Digital Library

[33]

Hoifung Poon and Pedro M. Domingos. Sum-product networks: A new deep architecture. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence (UAI), pages 337--346, Barcelona, Spain, 2011.

[34]

Shaogang Ren, Dingcheng Li, Zhixin Zhou, and Ping Li. Estimate the implicit likelihoods of gans with application to anomaly detection. In Proceedings of the Web Conference (WWW), pages 2287--2297, Taipei, 2020.

Digital Library

[35]

Shaogang Ren, Haiyan Yin, Mingming Sun, and Ping Li. Causal discovery with flow-based conditional density estimation. In Proceedings of the IEEE International Conference on Data Mining (ICDM), pages 1300--1305, 2021.

[36]

Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1530--1538, Lille, France, 2015.

[37]

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning (ICML), 2014.

[38]

Raquel Sánchez-Cauce, Iago París, and Francisco Javier D'iez. Sum-product networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (early access), 2021.

[39]

Marco Scutari. Learning bayesian networks with the bnlearn r package. arXiv preprint arXiv:0908.3817, 2009.

[40]

Peter Sorrenson, Carsten Rother, and Ullrich Kö the. Disentanglement by nonlinear ICA with general incompressible-flow networks (GIN). In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.

[41]

Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.

[42]

Stef Van Buuren and Karin Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in R. Journal of statistical software, 45:1--67, 2011.

[43]

Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, and Max Welling. Sylvester normalizing flows for variational inference. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI), 2018.

[44]

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. J. Mach. Learn. Res., 9:2579--2605, 2008.

[45]

Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential families, and variational inference. Now Publishers Inc, 2008.

Digital Library

[46]

Antoine Wehenkel and Gilles Louppe. Graphical normalizing flows. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 37--45, Virtual Event, 2021.

[47]

John M. Winn and Christopher M. Bishop. Variational message passing. J. Mach. Learn. Res., 6:661--694, 2005.

Digital Library

[48]

Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. A theory of generative convnet. In Proceedings of the 33nd International Conference on Machine Learning (ICML), pages 2635--2644, New York City, NY, 2016.

[49]

Eric P. Xing, Michael I. Jordan, and Stuart Russell. A generalized mean field algorithm for variational inference in exponential families. In Proceedings of the 19th Conference in Uncertainty in Artificial Intelligence (UAI), 2003.

[50]

Haiyan Yin, Dingcheng Li, Xu Li, and Ping Li. Meta-cotgan: A meta cooperative training paradigm for improving adversarial text generation. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020.

[51]

Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based generative models via coarse-to-fine expanding and sampling. In Proceeding of the 9th International Conference on Learning Representations (ICLR), Virtual Event, 2021.

[52]

Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: continuous optimization for structure learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 9492--9503, 2018.

[53]

Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy-based models from a single natural image for internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2961--2970, virtual, 2021.

[54]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017.

[55]

Song-Chun Zhu, Ying Nian Wu, and David Mumford. Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. International Journal of Computer Vision (IJCV), 27(2):107--126, 1998.

Cited By

Ren SFei HLi DLi P(2023)Learning Latent Structural Relations with Message Passing Prior2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00530(5323-5332)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00530
Ren SLi PAl Hasan MXiong L(2022)Flow-based Perturbation for Cause-effect InferenceProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557326(1706-1715)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557326
Ren SLi DLi P(2022)Causal Effect Prediction with Flow-based Inference2022 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM54844.2022.00149(1167-1172)Online publication date: Nov-2022
https://doi.org/10.1109/ICDM54844.2022.00149

Index Terms

Variational Flow Graphical Model
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Probabilistic reasoning
  2. Machine learning
    1. Machine learning approaches

Recommendations

Stochastic variational inference

We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions. We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet ...
Variational inference in nonconjugate models

Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the ...
Bayesian Inference via Variational Approximation for Collaborative Filtering

Variational approximation method finds wide applicability in approximating difficult-to-compute probability distributions, a problem that is especially important in Bayesian inference to estimate posterior distributions. Latent factor model is a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2022

5033 pages

ISBN:9781450393850

DOI:10.1145/3534678

General Chairs:
Aidong Zhang
University of Virginia
,
Huzefa Rangwala
Amazon/George Mason University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '22

Sponsor:

KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2022

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
706
Total Downloads

Downloads (Last 12 months)154
Downloads (Last 6 weeks)5

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ren SFei HLi DLi P(2023)Learning Latent Structural Relations with Message Passing Prior2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00530(5323-5332)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00530
Ren SLi PAl Hasan MXiong L(2022)Flow-based Perturbation for Cause-effect InferenceProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557326(1706-1715)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557326
Ren SLi DLi P(2022)Causal Effect Prediction with Flow-based Inference2022 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM54844.2022.00149(1167-1172)Online publication date: Nov-2022
https://doi.org/10.1109/ICDM54844.2022.00149

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents