Abstract
The optimization of complex cyber-physical systems is a crucial task for their correct functioning, usability, and commercial viability. Due to their complexity, scale and resource intensiveness, conventional manual optimization is infeasible in many instances. We investigate the combination of the Digital Twin paradigm and Reinforcement Learning framework to address the long response times, limited availability of data, and the intractability of such systems. Here, the Digital Twin functions as the training environment in different development phases of the optimization. In this position paper we showcase our ongoing research on developing a reference architecture of a Digital Twin-Artificial Intelligence optimization system. This includes presenting the development process of the optimization system in terms of phases, an architecture from four viewpoints and an exemplary implementation.
The research is carried out as part of the ITEA4 20216 ASIMOV project. The ASIMOV activities are supported by the Netherlands Organization for Applied Scientific Research TNO and the Dutch Ministry of Economic Affairs and Climate (project number: AI211006). This research is also partially funded by the German Federal Ministry of Education and Research (BMBF) within the project ASIMOV-D under grant agreement No. 01IS21022B [AVL], 01IS21022D [Liangdao], 01IS21022F [TrianGraphics] and 01IS21022G [DLR] based on a decision of the German Bundestag.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abdolmaleki, A., Springenberg, J.T., Tassa, Y., Munos, R., Heess, N., Riedmiller, M.: Maximum a posteriori policy optimisation (2018). https://doi.org/10.48550/arXiv.1806.06920
Aheleroff, S., Xu, X., Zhong, R.Y., Lu, Y.: Digital twin as a service (DTAAS) in industry 4.0: An architecture reference model. Adv. Eng. Inform. 47, 101225 (2021). https://doi.org/10.1016/j.aei.2020.101225
Alexopoulos, K., Nikolakis, N., Chryssolouris, G.: Digital twin-driven supervised machine learning for the development of artificial intelligence applications in manufacturing. Int. J. Comput. Integr. Manuf. 33(5), 429–439 (2020). https://doi.org/10.1080/0951192X.2020.1747642
Amiranashvili, A., Argus, M., Hermann, L., Burgard, W., Brox, T.: Pre-training of deep RL agents for improved learning under domain randomization. eprint arXiv:2104.14386 (2021). https://doi.org/10.48550/arXiv.2104.14386
Barricelli, B.R., Casiraghi, E., Fogli, D.: A survey on digital twin: definitions, characteristics, applications, and design implications. IEEE Access 7, 167653–167671 (2019). https://doi.org/10.1109/ACCESS.2019.2953499
Brankovic, B., Binder, C., Draxler, D., Neureiter, C., Lastro, G.: Towards a cross-domain modeling approach in system-of-systems architectures. In: Boy, G.A., Guegan, A., Krob, D., Vion, V. (eds.) CSDM 2019, pp. 164–175. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-34843-4_14
Cobbe, K., Klimov, O., Hesse, C., Kim, T., Schulman, J.: Quantifying generalization in reinforcement learning. In: 36th International Conference on Machine Learning, vol. PMLR 97. PMLR, Long Beach, USA (2019)
Degrave, J., et al.: Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602(7897), 414–419 (2022). https://doi.org/10.1038/s41586-021-04301-9
Dulac-Arnold, G., et al.: Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Mach. Learn. 110(9), 2419–2468 (2021)
Ferko, E., Bucaioni, A., Behnam, M.: Architecting digital twins. IEEE Access 10, 50335–50350 (2022). https://doi.org/10.1109/ACCESS.2022.3172964
Gan, X., Zuo, Y., Zhang, A., Li, S., Tao, F.: Digital twin-enabled adaptive scheduling strategy based on deep reinforcement learning. Sci. China Technol. Sci. 1–15 (2023)
Grieves, M., Vickers, J., (None): Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. In: Kahlen, F.J., Flumerfelt, S., Alves, A. (eds.) Transdisciplinary Perspectives on Complex Systems, pp. 85–113. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-38756-7
Haj-Ali, A., Ahmed, N.K., Willke, T., Gonzalez, J., Asanovic, K., Stoica, I.: A view on deep reinforcement learning in system optimization
Hankel, M., Rexroth, B.: Das Referenzarchitekturmodell Industrie 4.0 (RAMI 4.0)
IEEE: IEEE standard for modeling and simulation (m &s) high level architecture (HLA): Framework and rules (2010). https://standards.ieee.org/ieee/1516/3744/
Jamil, S., Rahman, M.: Fawad: a comprehensive survey of digital twins and federated learning for industrial internet of things (IIOT), internet of vehicles (IOV) and internet of drones (IOD). Appl. Syst. Innov. 5(3), 56 (2022)
Jones, D., Snider, C., Nassehi, A., Yon, J., Hicks, B.: Characterising the digital twin: a systematic literature review. CIRP J. Manuf. Sci. Technol. 29, 36–52 (2020). https://doi.org/10.1016/j.cirpj.2020.02.002
Ju, H., Juan, R., Gomez, R., Nakamura, K., Li, G.: Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nature Mach. Intell. 4(12), 1077–1087 (2022). https://doi.org/10.1038/s42256-022-00573-6
Julian, R., Swanson, B., Sukhatme, G.S., Levine, S., Finn, C., Hausman, K.: Never stop learning: the effectiveness of fine-tuning in robotic reinforcement learning. arXiv preprint arXiv:2004.10190 (2020)
Kairouz, P., et al. (eds.): Advances and Open Problems in Federated Learning, Foundation and Trends in Machine Learning, vol. 14. Now Publishers Inc. (2021). https://doi.org/10.1561/2200000083
Matulis, M., Harvey, C.: A robot arm digital twin utilising reinforcement learning. Comput. Graph. 95, 106–114 (2021). https://doi.org/10.1016/j.cag.2021.01.011
Menzel, T., Bagschik, G., Maurer, M.: 2018 IEEE Intelligent Vehicles Symposium (IV): 26–30 June 2018, Piscataway, NJ. IEEE (2018)
Modelica Association Project FMI: Functional mock-up interface for model exchange and co-simulation (2019)
Neurohr, C., Westhofen, L., Henning, T., de Graaff, T., Möhlmann, E., Böde, E.: Fundamental considerations around scenario-based testing for automated driving. In: IEEE Intelligent Vehicles Symposium Proceedings, pp. 121–127 (2020). https://doi.org/10.1109/IV47402.2020.9304823
Osiński, B., et al.: Simulation-based reinforcement learning for real-world autonomous driving. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6411–6418. IEEE (2020)
Pohl, K., Broy, M., Daembkes, H., Hönninger, H. (eds.): Advanced Model-Based Engineering of Embedded Systems. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48003-9
Shen, G., et al.: Deep reinforcement learning for flocking motion of multi-UAV systems: learn from a digital twin. IEEE Internet Things J. 9(13), 11141–11153 (2021)
Shen, G., Lei, L., Zhang, X., Li, Z., Cai, S., Zhang, L.: Multi-UAV cooperative search based on reinforcement learning with a digital twin driven training framework. IEEE Trans. Veh. Technol. 72, 8354–8368 (2023)
Stark, R., Damerau, T.: Digital twin. In: Chatti, S., Tolio, T. (eds.) CIRP Encyclopedia of Production Engineering, pp. 1–8. Springer Berlin Heidelberg, Heidelberg (2019). https://doi.org/10.1007/978-3-642-35950-7_16870-1
Sun, W., Lei, S., Wang, L., Liu, Z., Zhang, Y.: Adaptive federated learning and digital twin for industrial internet of things. IEEE Trans. Industr. Inf. 17(8), 5605–5614 (2021). https://doi.org/10.1109/TII.2020.3034674
Sutton, R.S., Barto, A.: Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning, 2nd edn. The MIT Press, Cambridge, Massachusetts and London, England (2018)
Wagg, D.J., Worden, K., Barthorpe, R.J., Gardner, P.: Digital twins: state-of-the-art and future directions for modeling and simulation in engineering dynamics applications. ASCE - ASME J. Risk Uncertainty Eng. Syst. 6(3) (2020). https://doi.org/10.1115/1.4046739
Zhang, Z., Zahng, D., Qiu, R.C.: Deep reinforcement learning for power system: an overview. CSEE J. Power Energy Syst. 6(1) (2020). https://doi.org/10.17775/CSEEJPES.2019.00920
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Modrakowski, E. et al. (2024). Architecture for Digital Twin-Based Reinforcement Learning Optimization of Cyber-Physical Systems. In: Tekinerdoğan, B., Spalazzese, R., Sözer, H., Bonfanti, S., Weyns, D. (eds) Software Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium. ECSA 2023. Lecture Notes in Computer Science, vol 14590. Springer, Cham. https://doi.org/10.1007/978-3-031-66326-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-66326-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-66325-3
Online ISBN: 978-3-031-66326-0
eBook Packages: Computer ScienceComputer Science (R0)