research-article

Criteria for the evaluation of self-* systems

Authors:

Claudia Raibulet,

Jean-Pierre Georgé,

Gauthier Picard,

Marie-Pierre GleizesAuthors Info & Claims

SEAMS '10: Proceedings of the 2010 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems

Pages 29 - 38

https://doi.org/10.1145/1808984.1808988

Published: 03 May 2010 Publication History

Abstract

In the last years, the growing complexity of the current applications has led to the design of self-adapting systems presenting self-* properties. These systems are composed of several autonomous interactive entities. They behave autonomously and present enhanced characteristics allowing them to handle dynamics coming from exogenous and endogenous changes.

In this paper, we propose a set of criteria for the description and evaluation of the adaptive properties of such systems. They aim to provide a concrete mechanism to analyze the quality of the design of adaptive systems, to evaluate the effect of self-* properties on the performances and to compare the adaptive features of different systems. The criteria are grouped into different categories: methodological, architectural, intrinsic, and runtime evaluation. They have been identified and specified by analyzing several case studies, which address self-adaptivity issues through different approaches with different objectives in various application contexts.

References

[1]

C. P. Bass, L. and R. Kazman. Software Architecture in Practice. Addison Wesley, USA, 2003.

Digital Library

[2]

I. Brito, F. Herrero, and P. Meseguer. On the evaluation of discsp algorithms. The Fifth International Workshop on Distributed Constraint Reasoning (DCR04) Toronto, Canada September 27, 2004 held in conjunction with Tenth International Conference on Principles and Practice of Constraint Programming (CP 2004), (5):142--151, 2004.

[3]

B. H. C. Cheng, R. De Lemos, H. Giese, P. Inverardi, and J. Magee, editors. Software Engineering for Self-Adaptive Systems, volume 5525 of Lecture Notes in Computer Science. Springer, 2009.

Digital Library

[4]

G. Clair, E. Kaddoum, M.-P. Gleizes, and G. Picard. Self-Regulation in Self-Organising Multi-Agent Systems for Adaptive and Intelligent Manufacturing Control. In IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO), Venice Italy, page (on line). IEEE Computer Society, 2008.

Digital Library

[5]

P. De Bra, A. Aerts, B. Berden, B. de Lange, B. Rousseau, T. Santic, D. Smits, and N. Stash. Aha! the adaptive hypermedia architecture. In HYPERTEXT '03: Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, pages 81--84, New York, NY, USA, 2003. ACM.

Digital Library

[6]

G. Di Marzo Serugendo, M.-P. Gleizes, and A. Karageorgos. Self-Organisation and Emergence in Multi-Agent Systems: An Overview. Informatica, 30(1):45--54, janvier 2006. ISSN 0350-5596.

[7]

F. Gaillard, Y. Kubera, P. Mathieu, and S. Picault. A reverse engineering form for multi agent systems. In Proceedings of the 9th International Workshop Engineering Societies in the Agents World (ESAW'2008), 2008.

[8]

D. Garlan, S.-W. Cheng, A.-C. Huang, B.-R. Schmerl, and P. Steenkiste. Rainbow: Architecture-based self-adaptation with reusable infrastructure. IEEE Computer, 37(10):46--54, 2004.

Digital Library

[9]

D. Garlan and M. Shaw. An introduction to software architecture. Technical Report CMU-CS-94-166, Carnegie Mellon University, January 1994.

Digital Library

[10]

M.-P. Gleizes, V. Camps, J.-P. Georgé, and D. Capera. Engineering Systems which Generate Emergent Functionalities. In EEMMAS 07, number 5049 in Lecture Notes in Artificial Intelligence (LNAI), page (on line). Springer-Verlag, juillet 2008.

Digital Library

[11]

I. Gorton, Y. Liu, and N. Trivedi. An extensible and lightweight architecture for adaptive server applications. Software - Practice and Experience Journal, 38(8):853--883, 2008.

Digital Library

[12]

K. M. Hansen, W. Zhang, and M. Ingstrup. Towards self-managed executable petri nets. In SASO '08: Proceedings of the 2008 Second IEEE International Conference on Self-Adaptive and Self-Organizing Systems, pages 287--296, Washington, DC, USA, 2008. IEEE Computer Society.

Digital Library

[13]

J. He, T. Gao, W. Hao, I.-L. Yen, and B. F. A flexible content adaptation system using a rule-based approach. IEEE Transactions on Knowledge and Data Engineering, 19:127--140, 2007.

Digital Library

[14]

E. Kaddoum, M.-P. Gleizes, J.-P. Georgé, and G. Picard. Characterizing and evaluating problem solving self-* systems. In Proceedings of the ADAPTIVE 2009 Conference, IEEE Press.

[15]

G. Karsai, A. Lédeczi, J. Sztipanovits, G. Peceli, G. Simon, and T. Kovácsházy. An approach to self-adaptive software based on supervisory control. In LNCS 2614, pages 77--92, 2003.

Digital Library

[16]

N. Lynch. Distributed Algorithms. Morgan-Kaufmann, 1996.

Digital Library

[17]

P. McKinley, S. Sadjadi, E. Kasten, and B. Cheng. Composing adaptive software. Computer, 37:56--64, 2004.

Digital Library

[18]

D. Meignan, J.-C. Creput, and A. Koukam. A coalition-based metaheuristic for the vehicle routing problem. pages 1176--1182, June 2008.

[19]

A. Meisels, E. Kaplansky, I. Razgon, and R. Zivan. Comparing performance of distributed constraints processing algorithms. In AAMAS-02 Workshop on Distributed Constraint Reasoning, number 5, pages 86--93, 2002.

[20]

C. H. Papadimitriou, A. A. Schäffer, and M. Yannakakis. On the complexity of local search. In STOC '90: Proceedings of the twenty-second annual ACM symposium on Theory of computing, pages 438--445, New York, NY, USA, 1990. ACM.

Digital Library

[21]

M. Pezzè and J. Wuttke. Automatic generation of runtime failure detectors from property templates. pages 223--240, 2009.

[22]

G. Picard and M.-P. Gleizes. The ADELFE Methodology -- Designing Adaptive Cooperative Multi-Agent Systems, chapter 8, pages 157--176. Kluwer Publishing, 2004.

[23]

C. Raibulet. Facets of adaptivity. In Proceedings of the 2nd European Conference on Software Architecture, LNCS 5292, pages 342--345, 2008.

Digital Library

[24]

C. Raibulet and L. Masciadri. Towards evaluation mechanisms for runtime adaptivity: from case studies to metrics. In Proceedings of the ADAPTIVE 2009 Conference, IEEE Press.

[25]

C. Raibulet and L. Masciadri. Evaluation of dynamic adaptivity through metrics: an achievable target? In WICSA/ECSA, pages 341--344, 2009.

[26]

P. Robertson, R. Laddaga, and H. Shrobe. Introduction: the first international workshop on self-adaptive software. In Proceedings of the 1st IWSAS, volume 1936 of LNCS, pages 1--10. Springer, 2000.

Digital Library

[27]

H. Sadat and A. Ghorbani. On the evaluation of adaptive web systems. In Workshop on Web-based Support Systems, pages 127--136, 2004.

[28]

N. Subramanian and L. Chung. Metrics for adaptability. In Journal of Applied Technology Division, pages 95--108, 1999.

[29]

R. P. Würtz. Organic Computing. Springer, 2008.

[30]

S. Weibelzahl. Evaluation of adaptive systems. In User Modeling, pages 292--294, 2001.

Digital Library

Cited By

Li QZeng F(2024)Enhancing Software Architecture Adaptability: A Comprehensive Evaluation MethodSymmetry10.3390/sym1607089416:7(894)Online publication date: 13-Jul-2024
https://doi.org/10.3390/sym16070894
Kraaijveld KRaibulet C(2024)Adaptive Strategies Metric SuiteEvaluation of Novel Approaches to Software Engineering10.1007/978-3-031-64182-4_14(296-320)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-64182-4_14
Goller MThomsen IAl-Falouji GTomforde S(2023)Abnormal Behaviour Detection of Self-Adaptive Agents in Traffic Environments2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C)10.1109/ACSOS-C58168.2023.00034(41-46)Online publication date: 25-Sep-2023
https://doi.org/10.1109/ACSOS-C58168.2023.00034
Show More Cited By

Index Terms

Criteria for the evaluation of self-* systems

Recommendations

Toward a web-based multi-criteria decision support system for the layered evaluation of interactive adaptive systems
Abstract
The choice of suitable evaluation methods for the layered evaluation of Interactive Adaptive Systems (IAS) needs the consideration of different factors, leading to a multi-criteria decision analysis problem. This paper proposes the first step ...
Communicability evaluation in systems created by designer-users
IHC '16: Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems

The number of people who strive to build their own systems grows every day, and for this type of activity we use the expression End-User Development (EUD). Communicability defines whether a system can make the user understands what the designer wanted ...
Evaluation Probes
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

We introduce evaluation probes for conducting emic, experiential evaluation of urban technologies "in the wild" without direct researcher presence. We commence with a thorough discussion and analysis of the original cultural probes, used by Gaver, Dunne ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SEAMS '10: Proceedings of the 2010 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems

May 2010

146 pages

ISBN:9781605589718

DOI:10.1145/1808984

General Chair:
Rogèrio de Lemos
University of Kent, UK
,
Program Chair:
Mauro Pezzè
University of Lugano, Switzerland and University of Milan-Bicocca, Italy

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 May 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '10

Sponsor:

SIGSOFT

ICSE '10: 32nd International Conference on Software Engineering

May 3 - 4, 2010

Cape Town, South Africa

Acceptance Rates

Overall Acceptance Rate 17 of 31 submissions, 55%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
567
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li QZeng F(2024)Enhancing Software Architecture Adaptability: A Comprehensive Evaluation MethodSymmetry10.3390/sym1607089416:7(894)Online publication date: 13-Jul-2024
https://doi.org/10.3390/sym16070894
Kraaijveld KRaibulet C(2024)Adaptive Strategies Metric SuiteEvaluation of Novel Approaches to Software Engineering10.1007/978-3-031-64182-4_14(296-320)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-64182-4_14
Goller MThomsen IAl-Falouji GTomforde S(2023)Abnormal Behaviour Detection of Self-Adaptive Agents in Traffic Environments2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C)10.1109/ACSOS-C58168.2023.00034(41-46)Online publication date: 25-Sep-2023
https://doi.org/10.1109/ACSOS-C58168.2023.00034
Jilderda HRaibulet C(2023)MAPE-K Based Guidelines for Designing Reactive and Proactive Self-adaptive SystemsSoftware Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium10.1007/978-3-031-66326-0_4(53-68)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-66326-0_4
Goller MTomforde S(2022)Runtime Assessment of the Parameter Utilisation in Adaptive Systems2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops53856.2022.9767230(200-205)Online publication date: 21-Mar-2022
https://doi.org/10.1109/PerComWorkshops53856.2022.9767230
Gerostathopoulos IRaibulet CAlberts E(2022)Assessing Self-Adaptation Strategies Using Cost-Benefit Analysis2022 IEEE 19th International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C54293.2022.00023(92-95)Online publication date: Mar-2022
https://doi.org/10.1109/ICSA-C54293.2022.00023
Naqvi MMalik SAstekin MMoonen L(2022)On Evaluating Self-Adaptive and Self-Healing Systems using Chaos Engineering2022 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)10.1109/ACSOS55765.2022.00018(1-10)Online publication date: Sep-2022
https://doi.org/10.1109/ACSOS55765.2022.00018
Tomforde SGoller M(2021)Beyond Homeostasis: A Novel Approach for Assessing the Stability and Coherence of Self-Adaptive Systems2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)10.1109/DASC-PICom-CBDCom-CyberSciTech52372.2021.00018(10-17)Online publication date: Oct-2021
https://doi.org/10.1109/DASC-PICom-CBDCom-CyberSciTech52372.2021.00018
Tomforde SGoller M(2020)To Adapt or Not to Adapt: A Quantification Technique for Measuring an Expected Degree of Self-AdaptationComputers10.3390/computers90100219:1(21)Online publication date: 18-Mar-2020
https://doi.org/10.3390/computers9010021
Ghahremani SGiese H(2020)Evaluation of Self-Healing Systems: An Analysis of the State-of-the-Art and Required ImprovementsComputers10.3390/computers90100169:1(16)Online publication date: 27-Feb-2020
https://doi.org/10.3390/computers9010016
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents