Building expressive and tractable probabilistic generative models: a review
IJCAI '24: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence
Article No.: 910, Pages 8234 - 8243
Abstract
We present a comprehensive survey of the advancements and techniques in the field of tractable probabilistic generative modeling, primarily focusing on Probabilistic Circuits (PCs). We provide a unified perspective on the inherent trade-offs between expressivity and tractability, highlighting the design principles and algorithmic extensions that have enabled building expressive and efficient PCs, and provide a taxonomy of the field. We also discuss recent efforts to build deep and hybrid PCs by fusing notions from deep neural models, and outline the challenges and open questions that can guide future research in this evolving field.
References
[1]
Tameem Adel, David Balduzzi, and Ali Ghodsi. Learning the structure of sum-product networks via an svd-based algorithm. In UAI, 2015.
[2]
Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. NeurIPS, 2022.
[3]
S. Bond-Taylor, A. Leach, Y. Long, and C. G. Willcocks. Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. TPAMI, 2022.
[4]
Cory J. Butz, Jhonatan de S. Oliveira, André E. dos Santos, and André L. Teixeira. Deep convolutional sum-product networks. In AAAI, 2019.
[5]
YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. 2020.
[6]
YooJung Choi, Tal Friedman, and Guy Van den Broeck. Solving marginal MAP exactly by probabilistic circuit transformations. In AISTATS, 2022.
[7]
Alvaro H. C. Correia, Gennaro Gala, Erik Quaeghebeur, Cassio de Campos, and Robert Peharz. Continuous mixtures of tractable probabilistic models. In AAAI, 2023.
[8]
Meihua Dang, Antonio Vergari, and Guy Broeck. Strudel: Learning structured-decomposable probabilistic circuits. In PGM, 2020.
[9]
Meihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In NeurIPS, 2022.
[10]
Adnan Darwiche. A differential approach to inference in bayesian networks. J. ACM, (3), 2003.
[11]
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 1977.
[12]
Aaron W. Dennis and Dan Ventura. Learning the architecture of sum-product networks using clustering on variables. In NeurIPS, 2012.
[13]
Nicola Di Mauro, Antonio Vergari, Teresa MA Basile, and Floriana Esposito. Fast and accurate density estimation with extremely randomized cutset networks. In ECML PKDD, 2017.
[14]
Nir Friedman and Daphne Koller. Being bayesian about network structure. a bayesian approach to structure discovery in bayesian networks. Machine learning, 2003.
[15]
Nir Friedman. The bayesian structural em algorithm. In UAI, 1998.
[16]
Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. In AISTATS, 2024.
[17]
Robert Gens and Pedro M. Domingos. Discriminative learning of sum-product networks. In NeurIPS, 2012.
[18]
Robert Gens and Pedro M. Domingos. Learning the structure of sum-product networks. In ICML, 2013.
[19]
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[22]
David Heckerman, Dan Geiger, and David M Chickering. Learning bayesian networks: The combination of knowledge and statistical data. Machine learning, 1995.
[23]
Athresh Karanam, Saurabh Mathur, Sahil Sidheekh, and Sriraam Natarajan. A unified framework for human-allied learning of probabilistic circuits. arXiv preprint, 2024.
[24]
Pasha Khosravi, YooJung Choi, Yitao Liang, Antonio Vergari, and Guy Van den Broeck. On tractable computation of expected predictions. In NeurIPS, 2019.
[25]
Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In KR, 2014.
[26]
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[27]
Steven Lang, Martin Mundt, Fabrizio Ventola, Robert Peharz, and Kristian Kersting. Elevating perceptual sample quality in pcs through differentiable sampling. In NeurIPS 2021 Workshop on Preregistration in Machine Learning, 2022.
[28]
Yang Li and Junier Oliva. Active feature acquisition with generative surrogate models. In Marina Meila and Tong Zhang, editors, ICML, 2021.
[29]
Bruce G Lindsay. Mixture models: theory, geometry, and applications. Ims, 1995.
[30]
Anji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless compression with probabilistic circuits. In ICLR, 2022.
[31]
Anji Liu, Honghua Zhang, and Guy Van den Broeck. Scaling up probabilistic circuits by latent variable distillation. In ICLR, 2023.
[32]
Xuejie Liu, Anji Liu, Guy Van den Broeck, and Yitao Liang. Understanding the distillation process from deep generative models to tractable probabilistic circuits. In ICML, 2023.
[33]
Anji Liu, Mathias Niepert, and Guy Van den Broeck. Image inpainting via tractable steering of diffusion models. In ICLR, 2024.
[34]
Lorenzo Loconte, Aleksanteri Mikulus Sladek, Stefan Mengel, Martin Trapp, Arno Solin, Nicolas Gillis, and Antonio Vergari. Subtractive mixture models via squaring: Representation and learning. In ICLR, 2024.
[35]
Antonio Mari, Gennaro Vessio, and Antonio Vergari. Unifying and understanding overparameterized circuit representations via low-rank tensor decompositions. In The 6th Workshop on Tractable Probabilistic Modeling, 2023.
[36]
Radu Marinescu and Rina Dechter. And/or branch-and-bound for graphical models. In IJCAI, 2005.
[37]
Saurabh Mathur, Vibhav Gogate, and Sriraam Natarajan. Knowledge intensive learning of cutset networks. In UAI, 2023.
[38]
Denis D Mauá, Fabio G Cozman, Diarmaid Conaty, and Cassio P Campos. Credal sum-product networks. In ISIPTA, 2017.
[39]
Denis Deratani Mauá, Diarmaid Conaty, Fabio Gagliardi Cozman, Katja Poppenhaeger, and Cassio Polpo de Campos. Robustifying sum-product networks. IJAR, 2018.
[40]
Nicola Di Mauro, Gennaro Gala, Marco Iannotta, and Teresa M. A. Basile. Random probabilistic circuits. In UAI, 2021.
[41]
Alejandro Molina, Antonio Vergari, Nicola Di Mauro, Sriraam Natarajan, Floriana Esposito, and Kristian Kersting. Mixed sum-product networks: A deep architecture for hybrid domains. In AAAI, 2018.
[42]
Milan Papez, Martin Rektoris, Vaclav Smidl, and Tomáš Pevný. Sum-product-set networks: Deep tractable models for tree-structured graphs. In ICLR, 2024.
[43]
Rohith Peddi, Tahrima Rahman, and Vibhav Gogate. Robust learning of tractable probabilistic models. In UAI, 2022.
[44]
Robert Peharz, Bernhard C Geiger, and Franz Pernkopf. Greedy part-wise learning of sum-product networks. In ECML PKDD, 2013.
[45]
Robert Peharz, Robert Gens, and Pedro Domingos. Learning selective sum-product networks. In LTPM workshop, ICML, 2014.
[46]
Robert Peharz, Robert Gens, Franz Pernkopf, and Pedro Domingos. On the latent variable interpretation in sum-product networks. TPAMI, 2016.
[47]
Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In ICML, 2020.
[48]
Robert Peharz, Antonio Vergari, Karl Stelzner, Alejandro Molina, Xiaoting Shao, Martin Trapp, Kristian Kersting, and Zoubin Ghahramani. Random sum-product networks: A simple and effective approach to probabilistic deep learning. In UAI, 2020.
[49]
Tomáš Pevný, Václav Smídl, Martin Trapp, Ondřej Poláček, and Tomáš Oberhuber. Sum-product-transform networks: Exploiting symmetries using invertible transformations. In PGM, 2020.
[50]
Hoifung Poon and Pedro M. Domingos. Sum-product networks: A new deep architecture. In UAI, 2011.
[51]
Tahrima Rahman, Prasanna Kothalkar, and Vibhav Gogate. Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of chow-liu trees. In ECML PKDD, 2014.
[52]
Amirmohammad Rooshenas and Daniel Lowd. Learning sum-product networks with direct and indirect variable interactions. In ICML, 2014.
[53]
Stuart J Russell. Artificial intelligence a modern approach. 2010.
[54]
Raquel Sánchez-Cauce, Iago París, and Francisco Javier Díez. Sum-product networks: A survey. TPAMI, 2021.
[55]
Nikil Roashan Selvam, Guy Van den Broeck, and YooJung Choi. Certifying fairness of probabilistic circuits. In AAAI, 2023.
[56]
Xiaoting Shao, Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Thomas Liebig, and Kristian Kersting. Conditional sum-product networks: Modular probabilistic circuits via gate functions. IJAR, 2022.
[57]
Or Sharir and Amnon Shashua. Sum-product-quotient networks. In AISTATS, 2018.
[58]
Andy Shih and Stefano Ermon. Probabilistic circuits for variational inference in discrete graphical models. In NeurIPS, 2020.
[59]
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Hyperspns: Compact and expressive probabilistic circuits. In NeurIPS, 2021.
[60]
Sahil Sidheekh, Kristian Kersting, and Sriraam Natarajan. Probabilistic flow circuits: Towards unified deep models for tractable probabilistic inference. In UAI, 2023.
[61]
Ping Liang Tan and Robert Peharz. Hierarchical decompositional mixtures of variational autoencoders. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, 2019.
[62]
Martin Trapp, Robert Peharz, Hong Ge, Franz Pernkopf, and Zoubin Ghahramani. Bayesian learning of sum-product networks. In NeurIPS, 2019.
[63]
Martin Trapp, Robert Peharz, Franz Pernkopf, and Carl Edward Rasmussen. Deep structured mixtures of gaussian processes. In AISTATS, 2020.
[64]
Fabrizio Ventola, Karl Stelzner, Alejandro Molina, and Kristian Kersting. Residual sum-product networks. In PGM, 2020.
[65]
Fabrizio Ventola, Steven Braun, Zhongjie Yu, Martin Mundt, and Kristian Kersting. Probabilistic circuits that know what they don't know. In UAI, 2023.
[66]
Antonio Vergari, Robert Peharz, Nicola Di Mauro, Alejandro Molina, Kristian Kersting, and Floriana Esposito. Sum-product autoencoding: Encoding and decoding representations using sum-product networks. In AAAI, 2018.
[67]
Antonio Vergari, Nicola Di Mauro, and Floriana Esposito. Visualizing and understanding sum-product networks. Machine Learning, 2019.
[68]
Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. In NeurIPS, 2021.
[69]
Benjie Wang and Marta Kwiatkowska. Compositional probabilistic and causal inference using tractable circuit models. In AISTATS, 2023.
[70]
Yang Yang, Gennaro Gala, and Robert Peharz. Bayesian structure scores for probabilistic circuits. In AISTATS, 2023.
[71]
Zhongjie Yu, Mingye Zhu, Martin Trapp, Arseny Skryagin, and Kristian Kersting. Leveraging probabilistic circuits for nonparametric multi-output regression. In UAI, 2021.
[72]
Zhongjie Yu, Devendra Singh Dhami, and Kristian Kersting. Sum-product-attention networks: Leveraging self-attention in energy-based probabilistic circuits. In The 5th Workshop on Tractable Probabilistic Modeling, 2022.
[73]
Zhongjie Yu, Fabrizio Ventola, Nils Thoma, Devendra Singh Dhami, Martin Mundt, and Kristian Kersting. Predictive whittle networks for time series. In UAI, 2022.
[74]
Zhongjie Yu, Martin Trapp, and Kristian Kersting. Characteristic circuits. In NeurIPS, 2024.
[75]
Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck. Tractable control for autoregressive language generation. In ICML, 2023.
[76]
Han Zhao, Pascal Poupart, and Geoffrey J Gordon. A unified approach for learning the parameters of sum-product networks. NeurIPS, 2016.
[77]
Pedro Zuidberg Dos Martires. Probabilistic neural circuits. In AAAI, 2024.
Index Terms
- Building expressive and tractable probabilistic generative models: a review
Index terms have been assigned to the content through auto-classification.
Recommendations
Tractable Bayesian learning of tree augmented Naive Bayes models
ICML'03: Proceedings of the Twentieth International Conference on International Conference on Machine LearningBayesian classifiers such as Naive Bayes or Tree Augmented Naive Bayes (TAN) have shown excellent performance given their simplicity and heavy underlying independence assumptions. In this paper we introduce a classifier taking as basis the TAN model and ...
Understanding the distillation process from deep generative models to tractable probabilistic circuits
ICML'23: Proceedings of the 40th International Conference on Machine LearningProbabilistic Circuits (PCs) are a general and unified computational framework for tractable probabilistic models that support efficient computation of various inference tasks (e.g., computing marginal probabilities). Towards enabling such reasoning ...
Continuous mixtures of tractable probabilistic models
AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial IntelligenceProbabilistic models based on continuous latent spaces, such as variational autoencoders, can be understood as uncountable mixture models where components depend continuously on the latent code. They have proven to be expressive tools for generative and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 International Joint Conferences on Artificial Intelligence.
Sponsors
- International Joint Conferences on Artifical Intelligence (IJCAI)
Publisher
Unknown publishers
Publication History
Published: 09 January 2025
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025