Abstract
We consider the problem of online combinatorial optimization under semi-bandit feedback. The goal of the learner is to sequentially select its actions from a combinatorial decision set so as to minimize its cumulative loss. We propose a learning algorithm for this problem based on combining the Follow-the-Perturbed-Leader (FPL) prediction method with a novel loss estimation procedure called Geometric Resampling (GR). Contrary to previous solutions, the resulting algorithm can be efficiently implemented for any decision set where efficient offline combinatorial optimization is possible at all. Assuming that the elements of the decision set can be described with d-dimensional binary vectors with at most m non-zero entries, we show that the expected regret of our algorithm after T rounds is \(O(m\sqrt{dT\log d})\). As a side result, we also improve the best known regret bounds for FPL, in the full information setting to \(O(m^{3/2}\sqrt{T\log d})\), gaining a factor of \(\sqrt{d/m}\) over previous bounds for this algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Allenberg, C., Auer, P., Györfi, L., Ottucsák, G.: Hannan consistency in on-line learning in case of unbounded losses under partial monitoring. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds.) ALT 2006. LNCS (LNAI), vol. 4264, pp. 229–243. Springer, Heidelberg (2006)
Audibert, J.-Y., Bubeck, S.: Regret bounds and minimax policies under partial monitoring. Journal of Machine Learning Research 11, 2635–2686 (2010)
Audibert, J.Y., Bubeck, S., Lugosi, G.: Regret in online combinatorial optimization. To appear in Mathematics of Operations Research (2013)
Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The nonstochastic multiarmed bandit problem. SIAM J. Comput. 32(1), 48–77 (2002)
Awerbuch, B., Kleinberg, R.D.: Adaptive routing with end-to-end feedback: distributed learning and geometric approaches. In: Proceedings of the 36th ACM Symposium on Theory of Computing, pp. 45–53 (2004)
Bubeck, S., Cesa-Bianchi, N., Kakade, S.M.: Towards minimax policies for online linear optimization with bandit feedback. In: Proceedings of the 25th Annual Conference on Learning Theory (COLT), pp. 1–14 (2012)
Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press, New York (2006)
Cesa-Bianchi, N., Lugosi, G.: Combinatorial bandits. Journal of Computer and System Sciences 78, 1404–1422 (2012)
Dani, V., Hayes, T., Kakade, S.: The price of bandit information for online optimization. In: Advances in Neural Information Processing Systems (NIPS), vol. 20, pp. 345–352 (2008)
György, A., Linder, T., Lugosi, G., Ottucsák, G.: The on-line shortest path problem under partial monitoring. Journal of Machine Learning Research 8, 2369–2403 (2007)
Hannan, J.: Approximation to Bayes risk in repeated play. Contributions to the Theory of Games 3, 97–139 (1957)
Kalai, A., Vempala, S.: Efficient algorithms for online decision problems. Journal of Computer and System Sciences 71, 291–307 (2005)
Koolen, W., Warmuth, M., Kivinen, J.: Hedging structured concepts. In: Proceedings of the 23rd Annual Conference on Learning Theory (COLT), pp. 93–105 (2010)
McMahan, H.B., Blum, A.: Online geometric optimization in the bandit setting against an adaptive adversary. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS (LNAI), vol. 3120, pp. 109–123. Springer, Heidelberg (2004)
Poland, J.: FPL analysis for adaptive bandits. In: Lupanov, O.B., Kasim-Zade, O.M., Chaskin, A.V., Steinhöfel, K. (eds.) SAGA 2005. LNCS, vol. 3777, pp. 58–69. Springer, Heidelberg (2005)
Suehiro, D., Hatano, K., Kijima, S., Takimoto, E., Nagano, K.: Online prediction under submodular constraints. In: Bshouty, N.H., Stoltz, G., Vayatis, N., Zeugmann, T. (eds.) ALT 2012. LNCS, vol. 7568, pp. 260–274. Springer, Heidelberg (2012)
Takimoto, E., Warmuth, M.: Paths kernels and multiplicative updates. Journal of Machine Learning Research 4, 773–818 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Neu, G., Bartók, G. (2013). An Efficient Algorithm for Learning with Semi-bandit Feedback. In: Jain, S., Munos, R., Stephan, F., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2013. Lecture Notes in Computer Science(), vol 8139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40935-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-642-40935-6_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40934-9
Online ISBN: 978-3-642-40935-6
eBook Packages: Computer ScienceComputer Science (R0)