Article

Factorized asymptotic Bayesian inference for latent feature models

Authors:

Ryohei FujimakiAuthors Info & Claims

NIPS'13: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1

Pages 1214 - 1222

Published: 05 December 2013 Publication History

Abstract

This paper extends factorized asymptotic Bayesian (FAB) inference for latent feature models (LFMs). FAB inference has not been applicable to models, including LFMs, without a specific condition on the Hessian matrix of a complete log-likelihood, which is required to derive a "factorized information criterion" (FIC). Our asymptotic analysis of the Hessian matrix of LFMs shows that FIC of LFMs has the same form as those of mixture models. FAB/LFMs have several desirable properties (e.g., automatic hidden states selection and parameter identifiability) and empirically perform better than state-of-the-art Indian Buffet processes in terms of model selection, prediction, and computational efficiency.

References

[1]

T. Broderick, B. Kulis, and M. I. Jordan. MAD-Bayes: MAP-based Asymptotic Derivations from Bayes. In ICML, 2013.

[2]

F. Doshi-Velez and Z. Ghahramani. Accelerated sampling for the indian buffet process. In ICML, 2009.

[3]

F. Doshi-Velez, K. T. Miller, J. Van Gael, and Y. W. Teh. Variational inference for the Indian buffet process. In AISTATS, 2009.

[4]

A. Frank and A. Asuncion. UCI machine learning repository, 2010.

[5]

R. Fujimaki and K. Hayashi. Factorized asymptotic bayesian hidden markov model. In ICML, 2012.

[6]

R. Fujimaki and S. Morinaga. Factorized asymptotic bayesian inference for mixture modeling. In AISTATS, 2012.

[7]

A. S. Georghiades, P. N. Belhumeur, and D. J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23:643-660, 2001.

[8]

Z. Ghahramani. Factorial learning and the EM algorithm. In NIPS, 1995.

[9]

Z. Ghahramani, T. L. Griffiths, and P. Sollich. Bayesian nonparametric latent feature models (with discussion). In 8th Valencia International Meeting on Bayesian Statistics, 2006.

[10]

T. Griffiths and Z. Ghahramani. Infinite latent feature models and the indian buffet process, 2005.

[11]

T. L. Griffiths and Z. Ghahramani. The indian buffet process: An introduction and review. JMLR, 12:1185-1224, 2011.

[12]

U. Hoffmann, G. Garcia, J. M. Vesin, K. Diserens, and T. Ebrahimi. A boosting approach to p300 detection with application to brain-computer interfaces. In International IEEE EMBS Conference on Neural Engineering, pages 97-100. 2005.

[13]

J. J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550-554, 1994.

[14]

M. W. Kadous. Temporal Classification: Extending the Classification Paradigm to Multivariate Time Series. PhD thesis, School of Computer Science & Engineering, University of New South Wales, 2002.

[15]

J. Kivinen and M. K. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1):1-63, 1997.

[16]

K. Miller, T. Griffiths, and M. Jordan. Nonparametric latent feature models for link prediction. In NIPS, 2009.

[17]

K. T. Miller. Bayesian Nonparametric Latent Feature Models. PhD thesis, University of California, Berkeley, 2011.

[18]

S. Nakajima, M. Sugiyama, and D. Babacan. On bayesian PCA: Automatic dimensionality selection and analytic solution. In ICML, 2011.

[19]

K. Palla, D. A. Knowles, and Z. Ghahramani. An infinite latent attribute model for network data. In ICML, 2012.

[20]

C. Peterson and J. Anderson. A mean field theory learning algorithm for neural networks. Complex systems, 1:995-1019, 1987.

[21]

G. E. Poliner and D. P. W. Ellis. A discriminative model for polyphonic piano transcription. EURASIP Journal of Advances in Signal Processing, 2007(1):154, 2007.

[22]

C. Reed and Z. Ghahramani. Scaling the indian buffet process via submodular maximization. In ICML, 2013.

[23]

G. Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6(2):461-464, 1978.

[24]

M. Tipping and C. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society. Series B, 61(3):611-622, 1999.

[25]

M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.

[26]

S. Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13(4):899-933, 2001.

[27]

S. Watanabe. Algebraic Geometry and Statistical Learning Theory (Cambridge Monographs on Applied and Computational Mathematics). Cambridge University Press, 2009.

[28]

R. Wong. Asymptotic Approximation of Integrals (Classics in Applied Mathematics). SIAM, 2001.

[29]

A. L. Yuille and A. Rangarajan. The Concave-Convex procedure. Neural Computation, 15(4):915-936, 2003.

[30]

R. S. Zemel and G. E. Hinton. Learning population codes by minimizing description length. Neural Computation, 7(3):11-18, 1994.

Cited By

Yen ILee WChang SSuggala ALin SRavikumar P(2017)Latent feature lassoProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305890.3306089(3949-3957)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305890.3306089
Imaizumi MFujimaki R(2017)Factorized asymptotic bayesian policy search for POMDPsProceedings of the 26th International Joint Conference on Artificial Intelligence10.5555/3171837.3171894(4346-4352)Online publication date: 19-Aug-2017
https://dl.acm.org/doi/10.5555/3171837.3171894

Factorized asymptotic Bayesian inference for latent feature models
1. Computing methodologies

Recommendations

Bayesian nonparametric latent feature models
Mean-field variational approximate Bayesian inference for latent variable models

The ill-posed nature of missing variable models offers a challenging testing ground for new computational techniques. This is the case for the mean-field variational Bayesian inference. The behavior of this approach in the setting of the Bayesian probit ...
Online but accurate inference for latent variable models with local Gibbs sampling

We study parameter inference in large-scale latent variable models. We first propose a unified treatment of online inference for latent variable models from a non-canonical exponential family, and draw explicit links between several previously proposed ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

NIPS'13: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1

December 2013

3236 pages

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 05 December 2013

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yen ILee WChang SSuggala ALin SRavikumar P(2017)Latent feature lassoProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305890.3306089(3949-3957)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305890.3306089
Imaizumi MFujimaki R(2017)Factorized asymptotic bayesian policy search for POMDPsProceedings of the 26th International Joint Conference on Artificial Intelligence10.5555/3171837.3171894(4346-4352)Online publication date: 19-Aug-2017
https://dl.acm.org/doi/10.5555/3171837.3171894

View Options

View options

Media

Figures

Other

Tables

View Table of Contents