Abstract
In transfer learning the aim is to solve new learning tasks using fewer examples by using information gained from solving related tasks. Existing transfer learning methods have been used successfully in practice and PAC analysis of these methods have been developed. But the key notion of relatedness between tasks has not yet been defined clearly, which makes it difficult to understand, let alone answer, questions that naturally arise in the context of transfer, such as, how much information to transfer, whether to transfer information, and how to transfer information across tasks. In this paper we look at transfer learning from the perspective of Algorithmic Information Theory, and formally solve these problems in the same sense Solomonoff Induction solves the problem of inductive inference. We define universal measures of relatedness between tasks, and use these measures to develop universally optimal Bayesian transfer learning methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Caruana, R.: Multitask learning. Machine Learning 28, 41–75 (1997)
Thrun, S., Mitchell, T.: Lifelong robot learning. Robotics and Autonomous Systems 15, 25–46 (1995)
Thrun, S., Pratt, L.Y. (eds.): Learning To Learn. Kluwer Academic Publishers, Boston (1998)
Ben-David, S., Schuller, R.: Exploiting task relatedness for learning multiple tasks. In: Proceedings of the 16th Annual Conference on Learning Theory (2003)
Baxter, J.: A model of inductive bias learning. Journal of Artificial Intelligence Research 12, 149–198 (2000)
Juba, B.: Estimating relatedness via data compression. In: Proceedings of the 23rd International Conference on Machine Learning (2006)
Bennett, C., Gacs, P., Li, M., Vitanyi, P., Zurek, W.: Information distance. IEEE Transactions on Information Theory 44(4), 1407–1423 (1998)
Li, M., Vitanyi, P.: An Introduction to Kolmogorov Complexity and its Applications, 2nd edn. Springer, New York (1997)
Zvonkin, A.K., Levin, L.A.: The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Math. Surveys 25(6), 83–124 (1970)
Hutter, M.: Optimality of Bayesian universal prediction for general loss and alphabet. Journal of Machine Learning Research 4, 971–1000 (2003)
Li, M., Chen, X., Ma, B., Vitanyi, P.: The similarity metric. IEEE Transactions on Information Theory 50(12), 3250–3264 (2004)
Cilibrasi, R., Vitanyi, P.: Clustering by compression. IEEE Transactions on Information theory 51(4), 1523–1545 (2004)
Solomonoff, R.J.: Complexity-based induction systems: comparisons and convergence theorems. IEEE Transactions on Information Theory 24(4), 422–432 (1978)
Hutter, M.: Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Springer, Berlin (2004)
Mahmud, M.M.H., Ray, S.: Transfer learning using Kolmogorov complexity:basic theory and empirical evaluations. Technical Report UIUC-DCS-R-2007-2875, Department of Computer Science, University of Illinois at Urbana-Champaign (2007)
Hutter, M.: The fastest and shortest algorithm for all well defined problems. International Journal of Foundations of Computer Science 13(3), 431–443 (2002)
Grunwald, P., Vitanyi, P.: Shannon information and Kolmogorov complexity. IEEE Transactions on Information Theory (submitted, 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mahmud, M.M.H. (2007). On Universal Transfer Learning. In: Hutter, M., Servedio, R.A., Takimoto, E. (eds) Algorithmic Learning Theory. ALT 2007. Lecture Notes in Computer Science(), vol 4754. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75225-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-540-75225-7_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-75224-0
Online ISBN: 978-3-540-75225-7
eBook Packages: Computer ScienceComputer Science (R0)