Abstract
In this paper we present and study a new class of regularized kernel methods for learning vector fields, which are based on filtering the spectrum of the kernel matrix. These methods include Tikhonov regularization as a special case, as well as interesting alternatives such as vector valued extensions of L2-Boosting. Our theoretical and experimental analysis shows that spectral filters that yield iterative algorithms, such as L2-Boosting, are much faster than Tikhonov regularization and attain the same prediction performances. Finite sample bounds for the different filters can be derived in a common framework and highlight different theoretical properties of the methods. The theory of vector valued reproducing kernel Hilbert space is a key tool in our study.
Chapter PDF
Similar content being viewed by others
References
Micchelli, C.A., Pontil, M.: On learning vector–valued functions. Neural Computation 17 (2005)
Lo Gerfo, L., Rosasco, L., Odone, F., De Vito, E., Verri, A.: Spectral algorithms for supervised learning. Neural Computation 20 (2008)
De Vito, E., Rosasco, L., Caponnetto, A., De Giovannini, U., Odone, F.: Learning from examples as an inverse problem. JMLR 6 (2005)
Poggio, T., Rifkin, R., Mukherjee, S., Niyogi, P.: General conditions for predictivity in learning theory. Nature 428 (2004)
Bousquet, O., Elisseeff, A.: Stability and generalization. JMLR 2 (2002)
Bühlmann, P., Yu, B.: Boosting with the l 2-loss: Regression and classification. JASA 98 (2002)
Caruana, R.: Multitask learning. Mach. Learn. 28 (1997)
Argyriou, A., Maurer, A., Pontil, M.: An algorithm for transfer learning in a heterogeneous environment. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part I. LNCS (LNAI), vol. 5211, pp. 71–85. Springer, Heidelberg (2008)
Boyle, P., Frean, M.: Dependent gaussian processes. In: NIPS (2005)
Chai, K., Williams, C., Klanke, S., Vijayakumar, S.: Multi-task gaussian process learning of robot inverse dynamics. In: NIPS (2009)
Breiman, L., Friedman, J.H.: Predicting multivariate responses in multiple linear regression. J. R. Statist. Soc. B 59(1) (1997)
Izenman, A.: Reduced-rank regression for the multivariate linear model. J. Multiv. Anal. 5 (1975)
van der Merwe, A., Zidek, J.V.: Multivariate regression analysis and canonical variates. Can. J. Stat. 8 (1980)
Wold, S., Ruhe, H., Wold, H., Dunn III, W.: The collinearity problem in linear regression. The partial least squares (pls) approach to generalizes inverses. SIAM J. Sci. Comput. 5 (1984)
Stein, M.L.: Interpolation of spatial data. Springer, Heidelberg (1999)
Brudnak, M.: Vector-valued support vector regression. In: IJCNN (2006)
Vazquez, E., Walter, E.: Multi output support vector regression. In: SYSID (2003)
Caponnetto, A., De Vito, E.: Optimal rates for regularized least-squares algorithm. Found. Comp. Math. (2006)
Carmeli, C., De Vito, E., Toigo, A.: Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem. JAA 4 (2006)
Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-posed Problems. John Wiley, Chichester (1977)
Evgeniou, T., Pontil, M., Poggio, T.: Regularization networks and support vector machines. Advances in Computational Mathematics 13(1), 1–50 (2000)
Bauer, F., Pereverzev, S., Rosasco, L.: On regularization algorithms in learning theory. J. Complex. 23(1) (2007)
Engl, H.W., Hanke, M., Neubauer, A.: Regularization of inverse problems. Kluwer, Dordrecht (1996)
Smola, A., Kondor, R.: Kernels and regularization on graphs. In: Schölkopf, B., Warmuth, M.K. (eds.) COLT/Kernel 2003. LNCS (LNAI), vol. 2777, pp. 144–158. Springer, Heidelberg (2003)
Evgeniou, T., Micchelli, C.A., Pontil, M.: Learning multiple tasks with kernel methods. JMLR 6 (2005)
Sheldon, D.: Graphical multi-task learning. Technical report, Cornell University (2008) (preprint)
Caponnetto, A., Micchelli, C.A., Pontil, M., Ying, Y.: Universal kernels for multi-task learning. JMLR 9 (2008)
Macêdo, I., Castro, R.: Learning divergence-free and curl-free vector fields with matrix-valued kernels. Technical report, Instituto Nacional de Matematica Pura e Aplicada (2008)
Micchelli, C.A., Pontil, M.: Kernels for multi-task learning. In: NIPS (2004)
Steinwart, I.: On the influence of the kernel on the consistency of support vector machines. JMLR 2 (2002)
Carmeli, C., De Vito, E., Toigo, A., Umanitá, V.: Vector valued reproducing kernel hilbert spaces and universality. JAA 4 (2010)
Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, Heidelberg (1996)
Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. IJCV 12(1) (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Baldassarre, L., Rosasco, L., Barla, A., Verri, A. (2010). Vector Field Learning via Spectral Filtering. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2010. Lecture Notes in Computer Science(), vol 6321. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15880-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-15880-3_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15879-7
Online ISBN: 978-3-642-15880-3
eBook Packages: Computer ScienceComputer Science (R0)