Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Deep Reinforcement Learning with a Classifier System – First Steps

  • Conference paper
  • First Online:
Architecture of Computing Systems (ARCS 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13642))

Included in the following conference series:

Abstract

Organic Computing enables self-* properties in technical systems for mastering them in the face of complexity and for improving robustness and efficiency. Key technology for self-improving adaptation decisions is reinforcement learning (RL). In this paper, we argue that traditional deep RL concepts are not applicable due to their limited interpretability. In contrast, approaches from the field of rule-based evolutionary RL are less powerful. We propose to fuse both technical concepts while maintaining their advantages – allowing for an applicability especially suited for Organic Computing applications. We present initial steps and the first evaluation of standard RL scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bishop, J.: xcsfrl. https://github.com/jtbish/xcsfrl. Accessed 09 May 2022

  2. Bishop, J.T., Gallagher, M.: Optimality-based analysis of XCSF compaction in discrete reinforcement learning. In: Bäck, T., et al. (eds.) PPSN 2020. LNCS, vol. 12270, pp. 471–484. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58115-2_33

    Chapter  Google Scholar 

  3. Bodnar, C., Day, B., Lió, P.: Proximal distilled evolutionary reinforcement learning. In: Proceedings of AAAI Conference on AI, vol. 34(4), pp. 3283–3290 (2020)

    Google Scholar 

  4. Bu, S.-J., Cho, S.-B.: A hybrid system of deep learning and learning classifier system for database intrusion detection. In: Martínez de Pisón, F.J., Urraca, R., Quintián, H., Corchado, E. (eds.) HAIS 2017. LNCS (LNAI), vol. 10334, pp. 615–625. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59650-1_52

    Chapter  Google Scholar 

  5. Bull, L., O’Hara, T.: Accuracy-based neuro and neuro-fuzzy classifier systems. In: Proceedings of GECCO 2002, p. 7 (2002)

    Google Scholar 

  6. Butz, M., Wilson, S.W.: An algorithmic description of XCS. In: Revised Papers from the 3rd IWLCS, pp. 253–272. IWLCS 2000, Springer (2000). https://doi.org/10.1007/s005000100111

  7. Butz, M., Kovacs, T., Lanzi, P., Wilson, S.: Toward a theory of generalization and learning in XCS. IEEE Trans. on Evol. Comp. 8(1), 28–46 (2004)

    Article  Google Scholar 

  8. Dam, H., Abbass, H., Lokan, C.: Xin Yao: neural-based learning classifier systems. IEEE Trans. Knowl. Data Eng. 20(1), 26–39 (2008)

    Article  Google Scholar 

  9. Giani, A., Baiardi, F., Starita, A.: PANIC: a parallel evolutionary rule based system. In: Proceedings of (EP)95, pp. 753–771 MIT Press (1995)

    Google Scholar 

  10. Holland, J.H.: Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. University of Michigan Press, Ann Arbor (1975)

    Google Scholar 

  11. Holland, J.H., Reitman, J.S.: Cognitive systems based on adaptive algorithms. In: Waterman, D., Haey-Roth, F. (eds.) Pattern-directed inference systems, pp. 313–329. Academic Press (1978)

    Google Scholar 

  12. Lanzi, P.L., Loiacono, D.: XCSF with neural prediction. In: 2006 IEEE International Conference on Evolutionary Computation, pp. 2270–2276. IEEE (2006)

    Google Scholar 

  13. Lanzi, P.L., Loiacono, D.: Classifier systems that compute action mappings. In: GECCO 2007: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, pp. 1822–1829 (2007)

    Google Scholar 

  14. Lanzi, P.L., Loiacono, D., Wilson, S.W., Goldberg, D.E.: Extending XCSF beyond linear approximation. In: GECCO 2005: Proceedings of the 7th annual conference on Genetic and evolutionary computation, pp. 1827–1834 (2005)

    Google Scholar 

  15. Loiacono, D., Lanzi, P.L.: Evolving neural networks for classifier prediction with XCSF. Technical report, AIRLab, Milano, Italy and IlliGAL, University of Illinois at Urbana Champaign (2014)

    Google Scholar 

  16. Müller-Schloer, C., Tomforde, S.: Organic Computing - Technical Systems for Survival in the Real World. Birkhäuser (2017)

    Google Scholar 

  17. O’Hara, T., Bull, L.: Prediction calculation in accuracy-based neural learning classifier systems. UWELCSG 04–004, UWE Bristol, England (2004)

    Google Scholar 

  18. O’Hara, T., Bull, L.: A memetic accuracy-based neural learning classifier system. In: Proceedings of CEC05, vol. 3, pp. 2040–2045. IEEE (2005)

    Google Scholar 

  19. O’Hara, T., Bull, L.: Building anticipations in an accuracy-based learning classifier system by use of an artificial neural network. In: 2005 IEEE Congress on Evolutionary Computation, vol. 3, pp. 2046–2052 (2005). ISSN: 1941–0026

    Google Scholar 

  20. O’Hara, T., Bull, L.: Backpropagation in accuracy-based neural learning classifier systems. In: Kovacs, T., Llorà, X., Takadama, K., Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) Learning Classifier Systems, pp. 25–39. Springer, Cham (2007)

    Chapter  Google Scholar 

  21. OpenAI: frozen lake - gym documentation. https://www.gymlibrary.ml/environments/toy_text/frozen_lake/. Accessed 15 May 2022

  22. Preen, R.J., Bull, L.: Deep learning with a classifier system: initial results. arXiv:2103.01118 [cs] (2021)

  23. Preen, R.J., Pätzel, D.: XCSF. https://github.com/rpreen/xcsf (2021). Accessed 03 May 2022

  24. Preen, R.J., Wilson, S.W., Bull, L.: Autoencoding with a classifier system. IEEE Trans. Evol. Comput. 25, 1079–1090 (2021)

    Article  Google Scholar 

  25. Prothmann, H., Tomforde, S., Branke, J., Hähner, J., Müller-Schloer, C., Schmeck, H.: Organic Traffic Control. In: Müller-Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing — A Paradigm Shift for Complex Systems, vol. 1, pp. 431–446. Springer, Cham (2011). https://doi.org/10.1007/978-3-0348-0130-0_28

  26. Rosenbauer, L., Stein, A., Maier, R., Pätzel, D., Hähner, J.: XCS as a reinforcement learning approach to automatic test case prioritization. In: Proceedings of GECCO 2020, pp. 1798–1806 (2020)

    Google Scholar 

  27. Schönberner, C.: Deep Reinforcement Learning with a Classifier System. Master’s thesis, Kiel University, Kiel, Germany (2022)

    Google Scholar 

  28. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359. Nature Publishing Group (2017)

    Google Scholar 

  29. Stein, A., Maier, R., Rosenbauer, L., Hähner, J.: XCS classifier system with experience replay. In: Proceedings of GECCO20, pp. 404–413. ACM (2020)

    Google Scholar 

  30. Stein, A., Rauh, D., Tomforde, S., Hähner, J.: Interpolation in the extended classifier system: An architectural perspective. J. Sys. Arch. 75, 79–94 (2017)

    Article  Google Scholar 

  31. Stein, A., Rudolph, S., Tomforde, S., Hähner, J.: Self-learning smart cameras - harnessing the generalization capability of XCS. In: IJCCI17, pp. 129–140 (2017)

    Google Scholar 

  32. Tomforde, S., Hähner, J.: Organic network control: turning standard protocols into evolving systems. In: Lio, P., Verma, D. (eds.) Biologically Inspired Networking and Sensing - Algorithms and Architectures, pp. 11–35. IGI Global (2012)

    Google Scholar 

  33. Tomforde, S., Hähner, J., Sick, B.: Interwoven systems. Inform. Spektrum 37(5), 483–487 (2014)

    Google Scholar 

  34. Tomforde, S., Prothmann, H., Branke, J., et al.: Observation and Control of Organic Systems. In: Müller-Schloer, C., Schmeck, H., Ungerer, T. (eds.) Organic Computing — A Paradigm Shift for Complex Systems, pp. 325–338. Springer, Cham (2011). https://doi.org/10.1007/978-3-0348-0130-0_21

  35. Tomforde, S., Sick, B., Müller-Schloer, C.: Organic computing in the spotlight. arXiv:1701.08125v1 [cs.MA] (2017)

  36. von Mammen, S., Tomforde, S., Höhner, J., et al.: OCbotics: an organic computing approach to collaborative robotic swarms. In: 2014 IEEE Symposium on Swarm Intelligence, pp. 1–8 (2014)

    Google Scholar 

  37. Wilson, S.W.: Classifier fitness based on accuracy. Evol. Comput. 3(2), 149–175 (1995)

    Article  Google Scholar 

  38. Wilson, S.W.: Get real! XCS with continuous-valued inputs. In: Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 1999. LNCS (LNAI), vol. 1813, pp. 209–219. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45027-0_11

    Chapter  Google Scholar 

  39. Wilson, S.W.: Classifiers that approximate functions. Nat. Comput. 1(2), 211–234 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  40. Wurman, P.R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T.J., et al.: Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature 602(7896), 223–228 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Systems, Department of Computer Science, Kiel University, Kiel, Germany

    Connor Schönberner & Sven Tomforde

Authors
  1. Connor Schönberner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven Tomforde
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Connor Schönberner .

Editor information

Editors and Affiliations

  1. Technical University of Munich, Garching, Germany

    Martin Schulz

  2. Technical University of Munich, Heilbronn, Germany

    Carsten Trinitis

  3. Chalmers University of Technology, Gothenburg, Sweden

    Nikela Papadopoulou

  4. Otto-von-Guericke-Universität Magdeburg, Magdeburg, Germany

    Thilo Pionteck

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schönberner, C., Tomforde, S. (2022). Deep Reinforcement Learning with a Classifier System – First Steps. In: Schulz, M., Trinitis, C., Papadopoulou, N., Pionteck, T. (eds) Architecture of Computing Systems. ARCS 2022. Lecture Notes in Computer Science, vol 13642. Springer, Cham. https://doi.org/10.1007/978-3-031-21867-5_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-21867-5_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-21866-8

  • Online ISBN: 978-3-031-21867-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation