Abstract
The problem of low-rank matrix estimation recently received a lot of attention due to challenging applications. A lot of work has been done on rank-penalized methods [1] and convex relaxation [2], both on the theoretical and applied sides. However, only a few papers considered Bayesian estimation. In this paper, we review the different type of priors considered on matrices to favour low-rank. We also prove that the obtained Bayesian estimators, under suitable assumptions, enjoys the same optimality properties as the ones based on penalization.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bunea, F., She, Y., Wegkamp, M.H.: Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics 39(2), 1282–1309 (2011)
Candès, E., Tao, T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56(5), 2053–2080 (2009)
Bennett, J., Lanning, S.: The netflix prize. In: Proceedings of KDD Cup and Workshop 2007 (2007)
Reinsel, G.C., Velu, R.P.: Multivariate reduced-rank regression: theory and applications. Springer Lecture Notes in Statistics, vol. 136 (1998)
Koltchinskii, V., Lounici, K., Tsybakov, A.B.: Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics 39(5), 2302–2329 (2011)
Alquier, P., Butucea, C., Hebiri, M., Meziani, K., Morimae, T.: Rank-penalized estimation of a quantum system. Preprint arXiv:1206.1711 (2012)
Geweke, J.: Bayesian reduced rank regression in econometrics. Journal of Econometrics 75, 121–146 (1996)
Lim, Y.J., Teh, Y.W.: Variational Bayesian approach to movie rating prediction. In: Proceedings of KDD Cup and Workshop 2007 (2007)
Lawrence, N.D., Urtasun, R.: Non-linear matrix factorization with Gaussian processes. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 601–608. ACM, New York (2009)
Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization. In: Platt, J.C., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in Neural Information Processing Systems 20, NIPS 2007. MIT Press, Cambridge (2008)
Anderson, T.: Estimating linear restrictions on regression coefficients for multivariate normal distributions. Annals of Mathematical Statistics 22, 327–351 (1951)
Izenman, A.: Reduced rank regression for the multivariate linear model. Journal of Multivariate Analysis 5(2), 248–264 (1975)
Yuan, M., Ekici, A., Lu, Z., Monteiro, R.: Dimension reduction and coefficient estimation in multivariate linear regression. Journal of the Royal Statistical Society - Series B 69, 329–346 (2007)
Candès, E., Plan, Y.: Matrix completion with noise. Proceedings of the IEEE 98(6), 625–636 (2009)
Candès, E., Recht, B.: Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9(6), 717–772 (2009)
Gross, D.: Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory 57, 1548–1566 (2011)
Rohde, A., Tsybakov, A.B.: Estimation of high-dimensional low-rank matrices. The Annals of Statistics 39, 887–930 (2011)
Klopp, O.: Rank-penalized estimators for high-dimensionnal matrices. Electronic Journal of Statistics 5, 1161–1183 (2011)
Koltchinskii, V.: Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems. Springer Lecture Notes in Mathematics (2011)
Dreze, J.H.: Bayesian limited information analysis of the simultaneous equation model. Econometrica 44, 1045–1075 (1976)
Dreze, J.H., Richard, J.F.: Bayesian analysis of simultaneous equation models. In: Griliches, Z., Intriligater, J.F. (eds.) Handbook of Econometrics, vol. 1. North-Holland, Amsterdam (1983)
Zellner, A., Min, C., Dallaire, D.: Bayesian analysis of simultaenous equation and related models using the Gibbs sampler and convergence checks. H. G. B. Alexander Research Founsation working paper, University of Chicago (1993)
Kleibergen, F., van Dijk, H.K.: Bayesian simultaneous equation analysis using reduced rank structures. Econometric Theory 14, 699–744 (1998)
Bauwens, L., Lubrano, M.: Identification restriction and posterior densities in cointegrated gaussian var systems. In: Fomby, T.M., Carter Hill, R. (eds.) Advances in Econometrics, vol. 11(B). JAI Press, Greenwich (1993)
Kleibergen, F., van Dijk, H.K.: On the shape of the likelihood-posterior in cointegration models. Econometric Theory 10, 514–551 (1994)
Kleibergen, F., Paap, R.: Priors, posteriors and Bayes factors for a Bayesian analysis of cointegration. Journal of Econometrics 111, 223–249 (2002)
Corander, J., Villani, M.: Bayesian assessment of dimensionality in reduced rank regression. Statistica Neerlandica 58(3), 255–270 (2004)
Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th Annual International Conference on Machine Learning, ICML 2008. ACM, New York (2008)
Zhou, M., Wang, C., Chen, M., Paisley, J., Dunson, D., Carin, L.: Nonparametric Bayesian matrix completion. In: IEEE Sensor Array and Multichannel Signal Processing Workshop (2010)
Babacan, S.D., Luessi, M., Molina, R., Katsaggelos, A.K.: Low-rank matrix completion by variational sparse Bayesian learning. In: IEEE International Conference on Audio, Speech and Signal Processing, Prague (Czech Republic), pp. 2188–2191 (2011)
Paisley, J., Carin, L.: A nonparametric Bayesian model for kernel matrix completion. In: Proceedings of ICASSP 2010, Dallas, USA (2010)
Yu, K., Tresp, V., Schwaighofer, A.: Learning Gaussian processes for multiple tasks. In: Proceedings of the 22th Annual International Conference on Machine Learning, ICML 2005 (2005)
Yu, K., Lafferty, J., Zhu, S., Gong, Y.: Large-scale collaborative prediction using a non-parametric random effects model. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009. ACM, New York (2009)
Aoyagi, M., Watanabe, S.: The generalization error of reduced rank regression in Bayesian estimation. In: International Symposium on Information Theory and its Applications, ISITA 2004, Parma, Italy (2004)
Aoyagi, M., Watanabe, S.: Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks 18, 924–933 (2005)
van der Vaart, A.W.: Asymptotic Statistics. Cambridge University Press (1998)
Shawe-Taylor, J., Williamson, R.: A PAC analysis of a Bayes estimator. In: Proceedings of the Tenth Annual Conference on Computational Learning Theory, pp. 2–9. ACM, New York (1997)
McAllester, D.A.: Some pac-bayesian theorems. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, pp. 230–234. ACM (1998)
Catoni, O.: Statistical Learning Theory and Stochastic Optimization. Springer Lecture Notes in Mathematics (2004)
Catoni, O.: PAC-Bayesian Supervised Classification (The Thermodynamics of Statistical Learning). Lecture Notes-Monograph Series, vol. 56. IMS (2007)
Dalalyan, A.S., Tsybakov, A.B.: Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Machine Learning 72, 39–61 (2008)
Dalalyan, A.S., Tsybakov, A.B.: Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78(5), 1423–1443 (2012)
Dalalyan, A.S., Salmon, J.: Sharp oracle inequalities for aggregation of affine estimators. The Annals of Statistics 40(4), 2327–2355 (2012)
Alquier, P., Lounici, K.: PAC-Bayesian bounds for sparse regression estimation with exponential weights. Electronic Journal of Statistics 5, 127–145 (2011)
Audibert, J.Y., Catoni, O.: Robust linear least squares regression. The Annals of Statistics 39, 2766–2794 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Alquier, P. (2013). Bayesian Methods for Low-Rank Matrix Estimation: Short Survey and Theoretical Study. In: Jain, S., Munos, R., Stephan, F., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2013. Lecture Notes in Computer Science(), vol 8139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40935-6_22
Download citation
DOI: https://doi.org/10.1007/978-3-642-40935-6_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40934-9
Online ISBN: 978-3-642-40935-6
eBook Packages: Computer ScienceComputer Science (R0)