Abstract
Ensemble-based component systems have been used for many years to develop collective adaptive systems (CAS). The DEECo component model offers a framework for modeling and implementing ensemble-based component systems. Being expressive enough and having semantics specifically tailored towards dynamically evolving systems, DEECo has proven to be fairly powerful in modeling complex and dynamic architectures. At the same time, its specific semantics turned out to be a hurdle for newcomers when expressing their design intention and understanding its implications and side effects. We see quite a potential in employing large language models (LLMs) to simplify creating and refining the DEECo architectures. Since this constitutes a large research scope, we focus in this paper on initial experiments demonstrating how well generic LLMs understand the advanced concepts of ensemble-based CAS embodied in DEECo and asses what expectation from LLMs is realistic in this context. Our results indicate that LLMs can indeed understand ensemble-based architectures, but it depends on the form in which the architecture is presented in the textual form. Using external DSL, which is very self-explanatory gave good results out of the box. Specifications embedded in existing programming languages needed prior explanation of how to interpret them.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Ahmad, A., Waseem, M., Liang, P., et al.: Towards human-bot collaborative software architecting with ChatGPT. In: Proceedings of EASE 2023, Oulu, Finland, pp. 279–285 (2023). https://doi.org/10.1145/3593434.3593468
Al Ali, R., Bures, T., Hnetynka, P., et al.: Dynamic security specification through autonomic component ensemble. In: Proceedings of ISoLA 2018, Limassol, Cyprus. LNCS. Springer (2018). https://doi.org/10.1007/978-3-030-03424-5_12
Al Ali, R., Bures, T., Hnetynka, P., et al.: Toward autonomically composable and context-dependent access control specification through ensembles. Int. J. Softw. Tools Technol. Transfer 22(4), 511–522 (2020). https://doi.org/10.1007/s10009-020-00556-1
Brown, T.B., Mann, B., Ryder, N., et al.: Language models are few-shot learners. In: Proceedings NIPS 2020, Vancouver, Canada, pp. 1877–1901 (2020) https://doi.org/10.1007/s10009-020-00556-1
Bures, T., Gerostathopoulos, I., Hnetynka, P., et al.: DEECO: an ensemble-based component system. In: Proceedings of CBSE 2013, Vancouver, Canada, pp. 81–90 (2013). https://doi.org/10.1145/2465449.2465462
Clark, P., Cowhey, I., Etzioni, O., et al.: Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, (2018). https://doi.org/10.48550/arxiv1803.05457. arXiv: 1803.05457 [cs.AI]
Cobbe, K., Kosaraju, V., Bavarian, M., et al.: Training Verifiers to Solve Math Word Problems, (2021). https://doi.org/10.48550/arXiv.2110.14168arXiv: 2110.14168 [cs.LG]
Dhar, R., Vaidhyanathan, K., Varma, V.: Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study (2024). https://doi.org/10.48550/arxiv.2403.01709. arXiv: 2403.01709 [cs.SE]. (accepted to ICSA 2024)
Donakanti, R., Jain, P., Kulkarni, S., et al.: Reimagining Self-Adaptation in the Age of Large Language Models (2024). https://doi.org/10.48550/arxiv.2404.09866arXiv: 2404.09866 [cs.SE]
Hendrycks, D., Burns, C., Basart, S., et al.: Measuring Massive Multitask Language Understanding, (2021). https://doi.org/10.48550/arxiv.2009.03300arXiv: 2009.03300 [cs.CY]
Hnetynka, P., Bures, T., Gerostathopoulos, I., et al.: Using component ensembles for modeling autonomic component collaboration in smart farming. In: Proceedings of SEAMS 2020, Seoul, Korea, pp. 156–162. ACM (2020). https://doi.org/10.1145/3387939.3391599
Krijt, F., Jiracek, Z., Bures, T., et al.: Intelligent ensembles - a declarative group description language and java framework. In: Proceedings of SEAMS 2017, Buenos Aires, Argentina, pp. 116–122 (2017). https://doi.org/10.1109/SEAMS.2017.17
Li, B., Wu, W., Tang, Z., et al.: DevBench: a comprehensive benchmark for software development (2024). https://doi.org/10.48550/arxiv.2403.08604arXiv: 2403.08604 [cs.CL]
Liang, P., Bommasani, R., Lee, T., et al.: Holistic Evaluation of Language Models (2023). https://doi.org/10.48550/arxiv.2211.09110. arXiv:2211. 09110 [cs.CL]
Mernik, M., Heering, J., Sloane, A.M.: When and how to develop domain-specific languages. ACM Comput. Surv. 37(4), 316–344 (2005). https://doi.org/10.1145/1118890.1118892
Sakaguchi, K., Bras, R.L., Bhagavatula, C., et al.: WinoGrande: an adversarial winograd schema challenge at scale (2019). https://doi.org/10.48550/arxiv.1907.10641.arXiv: 1907.10641 [cs.CL]
Sutawika, L., Schoelkopf, H., Gao, L., et al.: EleutherAI/lm-evaluation-harness: v0.4.2, version v0.4.2 (2024). https://doi.org/10.5281/zenodo.10829972
Topfer, M., Abdullah, M., Krulis, M., et al.: ML-DEECo: a machine-learning- enabled framework for self-organizing components. In: Companion proceedings of ACSOS 2022, Virtual event, pp. 66–69. IEEE (2022). https://doi.org/10.1109/ACSOSC56246.2022.00033
Wirsing, M., H–lzl, M., Koch, N., et al.: Software Engineering for Collective Autonomic Systems (The ASCENS Approach). Springer (2015)
Zellers, R., Holtzman, A., Bisk, Y., et al.: HellaSwag: Can a Machine Really Finish Your Sentence? (2019). https://doi.org/10.48550/arxiv.1905.07830arXiv:1905.07830 [cs.CL]
Acknowledgment
This work has been partially supported by the EU project ExtremeXP grant agreement 101093164, partially by the Charles University Grant Agency projects 408622 and 269723, and partially by Charles University institutional funding SVV 260698.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Töpfer, M., Khalyeyev, D., Bureš, T., Hnětynka, P., Plášil, F. (2025). How Well Do LLMs Understand DEECo Ensemble-Based Component Architectures. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. Rigorous Engineering of Collective Adaptive Systems. ISoLA 2024. Lecture Notes in Computer Science, vol 15220. Springer, Cham. https://doi.org/10.1007/978-3-031-75107-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-75107-3_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-75106-6
Online ISBN: 978-3-031-75107-3
eBook Packages: Computer ScienceComputer Science (R0)