research-article

A Catalog of Performance Measures for Self-Adaptive Systems

Authors:

Maike Bezerra da Silva,

Emanuel Coutinho,

Paulo Henrique MaiaAuthors Info & Claims

SBQS '21: Proceedings of the XX Brazilian Symposium on Software Quality

Article No.: 1, Pages 1 - 10

https://doi.org/10.1145/3493244.3493259

Published: 14 December 2021 Publication History

Abstract

[Context] Self-adaptive systems (SAS) can evaluate their own behavior at runtime and change it when necessary to avoid failures. Many of those systems require a high processing power to carry out their dynamic and complex operations successfully. In this realm, performance arises as an important quality characteristic that has been widely used in the literature to drive the necessary adaptation strategies in SAS to achieve or maintain the QoS requirements. However, few studies evaluate the SAS performance efficiently due to the lack of knowledge on all possible performance measures. Still, there is a variety of measures scattered throughout the body of work in this area. [Objective] Our objective is to create a catalog of performance measures for assessing self-adaptive systems. [Method] To do that, we identified and analyzed the performance measures used in 32 primary studies found through a systematic literature mapping. [Results] As a result, 18 measures have been inserted in the catalog, having most of them addressed the sub-characteristics of time behavior and resource utilization. To validate the catalog, we implemented a subset of measures in a healthcare SAS.

References

[1]

Germán H Alférez, Vicente Pelechano, Raúl Mazo, Camille Salinesi, and Daniel Diaz. 2014. Dynamic adaptation of service compositions with variability models. Journal of Systems and Software 91 (2014), 24–47. http://www.sciencedirect.com/science/article/pii/S0164121213001465

Digital Library

[2]

Mohammad Alrifai and Thomas Risse. 2009. Combining Global Optimization with Local Selection for Efficient QoS-Aware Service Composition. In Proceedings of the 18th International Conference on World Wide Web(WWW ’09). Association for Computing Machinery, New York, NY, USA, 881–890. https://doi.org/10.1145/1526709.1526828

Digital Library

[3]

D. M. Barbosa, R. G. De Moura Lima, P. H. Mendes Maia, and E. Costa. 2017. Lotus@Runtime: A Tool for Runtime Monitoring and Verification of Self-Adaptive Systems. In 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). 24–30. https://doi.org/10.1109/SEAMS.2017.18

Digital Library

[4]

Simona Bernardi, José Merseguer, and Dorina C Petriu. 2011. A dependability profile within MARTE. Software & Systems Modeling 10, 3 (2011), 313–336. https://doi.org/10.1007/s10270-009-0128-1

Digital Library

[5]

Kate M. Bowers, Erik M. Fredericks, and Betty H. C. Cheng. 2018. Automated Optimization of Weighted Non-functional Objectives in Self-adaptive Systems. In Search-Based Software Engineering. Springer, Cham, 182–197. https://doi.org/10.1007/978-3-319-99241-9_9

[6]

Radu Calinescu, Lars Grunske, Marta Kwiatkowska, Raffaela Mirandola, and Giordano Tamburrelli. 2011. Dynamic QoS management and optimization in service-based systems. IEEE Transactions on Software Engineering 37, 3 (2011), 387–409.

Digital Library

[7]

Javier Cámara, Rogério de Lemos, Marco Vieira, Raquel Almeida, and Rafael Ventura. 2013. Architecture-based resilience evaluation for self-adaptive systems. Computing 95, 8 (2013), 689–722. https://doi.org/10.1007/s00607-013-0311-7

Digital Library

[8]

Javier Cámara, David Garlan, and Bradley Schmerl. 2019. Synthesizing tradeoff spaces with quantitative guarantees for families of software systems. Journal of Systems and Software 152 (2019), 33–49.

Digital Library

[9]

Javier Cámara, David Garlan, Bradley Schmerl, and Ashutosh Pandey. 2015. Optimal Planning for Architecture-Based Self-Adaptation via Model Checking of Stochastic Games. In Proceedings of the 30th Annual ACM Symposium on Applied Computing(SAC ’15). Association for Computing Machinery, New York, NY, USA, 428–435. https://doi.org/10.1145/2695664.2695680

Digital Library

[10]

Mauro Caporuscio, Antinisca Di Marco, and Paola Inverardi. 2007. Model-based system reconfiguration for dynamic performance management. Journal of Systems and Software 80, 4 (2007), 455–473. http://www.sciencedirect.com/science/article/pii/S0164121206002068 Software Performance.

Digital Library

[11]

Rainara Maia Carvalho, Rossana Maria de Castro Andrade, and Káthia Marçal de Oliveira. 2018. AQUARIUM - A suite of software measures for HCI quality evaluation of ubiquitous mobile applications. Journal of Systems and Software 136 (2018), 101–136. https://doi.org/10.1016/j.jss.2017.11.022

Digital Library

[12]

Bihuan Chen, Xin Peng, and Wenyun Zhao. 2009. Towards runtime optimization of software quality based on feedback control theory. In Proceedings of the First Asia-Pacific Symposium on Internetware(Internetware ’09). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/1640206.1640216

Digital Library

[13]

Maxime Cordy, Andreas Classen, Patrick Heymans, Axel Legay, and Pierre-Yves Schobbens. 2013. Model checking adaptive software with featured transition systems. In Assurances for Self-Adaptive Systems: Principles, Models, and Techniques. Springer, Berlin, Heidelberg, 1–29. https://doi.org/10.1007/978-3-642-36249-1_1

[14]

Rui A Santos Cruz, Mário Serafim Nunes, Leandro Menezes, and João Domingues. 2011. IPTV architecture for an IMS environment with dynamic QoS adaptation. Multimedia Tools and Applications 53, 3 (2011), 557–589. https://doi.org/10.1007/s11042-010-0537-8

Digital Library

[15]

Rogério De Lemos, David Garlan, Carlo Ghezzi, Holger Giese, Jesper Andersson, Marin Litoiu, Bradley Schmerl, Danny Weyns, Luciano Baresi, Nelly Bencomo, 2017. Software engineering for self-adaptive systems: Research challenges in the provision of assurances. In Software Engineering for Self-Adaptive Systems III. Assurances. Springer, 3–30.

[16]

Amanda Oliveira de Sousa, Carla I. M. Bezerra, Rossana M. C. Andrade, and José M. S. M. Filho. 2019. Quality Evaluation of Self-Adaptive Systems: Challenges and Opportunities. In Proceedings of the XXXIII Brazilian Symposium on Software Engineering(SBES 2019). Association for Computing Machinery, New York, NY, USA, 213–218. https://doi.org/10.1145/3350768.3352455

Digital Library

[17]

Antinisca Di Marco, Paola Inverardi, and Romina Spalazzese. 2013. Synthesizing Self-Adaptive Connectors Meeting Functional and Performance Concerns. In Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems(SEAMS ’13). IEEE Press, 133–142.

Digital Library

[18]

Ali Farahani, Eslam Nazemi, Giacomo Cabri, and Alireza Rafizadeh. 2016. An evaluation method for Self-Adaptive systems. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, Budapest, Hungary, 2814–2820. https://doi.org/10.1109/SMC.2016.7844665

Digital Library

[19]

Ilias Gerostathopoulos, Thomas Vogel, Danny Weyns, and Patricia Lago. 2021. How do we Evaluate Self-adaptive Software Systems?: A Ten-Year Perspective of SEAMS. In 2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). 59–70. https://doi.org/10.1109/SEAMS51251.2021.00018

[20]

Sukhpal Singh Gill and Rajkumar Buyya. 2019. Resource provisioning based scheduling framework for execution of heterogeneous and clustered workloads in clouds: from fundamental to autonomic offering. Journal of Grid Computing 17, 3 (2019), 385–417. https://doi.org/10.1007/s10723-017-9424-0

Digital Library

[21]

Thomas Goldschmidt, Stefan Hauck-Stattelmann, Somayeh Malakuti, and Sten Grüner. 2018. Container-based architecture for flexible industrial control applications. Journal of Systems Architecture 84 (2018), 28–36. https://doi.org/10.1016/j.sysarc.2018.03.002

[22]

Daniel Dominguez Gouvêa, Cyro de A Assis D Muniz, Gilson A Pinto, Alberto Avritzer, Rosa Maria Meri Leao, Edmundo de Souza e Silva, Morganna Carmem Diniz, Vittorio Cortellessa, Luca Berardinelli, Julius CB Leite, 2013. Experience with model-based performance, reliability, and adaptability assessment of a complex industrial architecture. Software & Systems Modeling 12, 4 (2013), 765–787. https://doi.org/10.1007/s10270-012-0264-x

Digital Library

[23]

Haithem Haggui, Sofiène Affes, and Faouzi Bellili. 2019. FPGA-SDR Integration and Experimental Validation of a Joint DA ML SNR and Doppler Spread Estimator for 5G Cognitive Transceivers. IEEE Access 7(2019), 69464–69480. https://doi.org/10.1109/ACCESS.2019.2919978

[24]

ISO/IEC. 2011. ISO-IEC 25010: 2011 Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-System and Software Quality Models. ISO.

[25]

G Maria Kalavathy, N Edison Rathinam, and P Seethalakshmi. 2012. Self-adaptable media service architecture for guaranteeing reliable multimedia services. Multimedia Tools and Applications 57, 3 (2012), 633–650.

Digital Library

[26]

G. Lau, M. Al-Sabah, M. Jaseemuddin, H. Razavi, and M. Bhuiyan. 2017. Context-aware RAON middleware for opportunistic network. Pervasive and Mobile Computing 41 (2017), 28 – 45. https://doi.org/10.1016/j.pmcj.2017.07.004

Digital Library

[27]

Chung-Horng Lung, Xu Zhang, and Pragash Rajeswaran. 2016. Improving software performance and reliability in a distributed and concurrent environment with an architecture-based self-adaptive framework. Journal of Systems and Software 121 (2016), 311 – 328. https://doi.org/10.1016/j.jss.2016.06.102

Digital Library

[28]

S. Maaloul, M. Afif, and S. Tabbane. 2016. Handover Decision in Heterogeneous Networks. In 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA). 588–595.

[29]

Moreno Marzolla and Raffaela Mirandola. 2010. Performance aware reconfiguration of software systems. In European Performance Engineering Workshop. Springer, 51–66.

[30]

Jhonny Mertz and Ingrid Nunes. 2019. On the Practical Feasibility of Software Monitoring: A Framework for Low-Impact Execution Tracing. In Proceedings of the 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems(SEAMS ’19). IEEE Press, 169–180. https://doi.org/10.1109/SEAMS.2019.00030

Digital Library

[31]

Afaf Mousa, Jamal Bentahar, and Omar Alam. 2019. Context-aware composite SaaS using feature model. Future Generation Computer Systems 99 (2019), 376 – 390. https://doi.org/10.1016/j.future.2019.04.032

Digital Library

[32]

Nicola Nostro, Romina Spalazzese, Felicita Di Giandomenico, and Paola Inverardi. 2016. Achieving functional and non functional interoperability through synthesized connectors. Journal of Systems and Software 111 (2016), 185 – 199. https://doi.org/10.1016/j.jss.2015.09.038

Digital Library

[33]

Nuno Oliveira and Luís S Barbosa. 2015. Self-adaptation by coordination-targeted reconfigurations. Journal of Software Engineering Research and Development 3, 1(2015), 1–31.

[34]

Marc Oriol, Xavier Franch, and Jordi Marco. 2015. Monitoring the service-based system lifecycle with SALMon. Expert Systems with Applications 42, 19 (2015), 6507 – 6521. https://doi.org/10.1016/j.eswa.2015.03.027

Digital Library

[35]

Anthony Peruma and Daniel Krutz. 2018. Security: a critical quality attribute in self-adaptive systems. In 2018 IEEE/ACM 13th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). IEEE, 188–189.

Digital Library

[36]

A. Perveen, M. Patwary, and A. Aneiba. 2019. Dynamically Reconfigurable Slice Allocation and Admission Control within 5G Wireless Networks. In 2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring). 1–7.

[37]

Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (2015), 1–18.

Digital Library

[38]

Nenad Petrovic and Milorad Tosic. 2020. SMADA-Fog: Semantic model driven approach to deployment and adaptivity in fog computing. Simulation Modelling Practice and Theory 101 (2020), 102033. https://doi.org/10.1016/j.simpat.2019.102033 Modeling and Simulation of Fog Computing.

[39]

Linh Manh Pham and Tuan-Minh Pham. 2017. Autonomic fine-grained replication and migration at component level on multicloud. Vietnam Journal of Computer Science 4, 1 (2017), 39–49.

Digital Library

[40]

Pasqualina Potena. 2013. Optimization of adaptation plans for a service-oriented architecture with cost, reliability, availability and performance tradeoff. Journal of Systems and Software 86, 3 (2013), 624–648.

Digital Library

[41]

A Rafique, D Van Landuyt, E Truyen, 2019. SCOPE: self-adaptive and policy-based data management middleware for federated clouds. J Internet Serv Appl 10.

[42]

Claudia Raibulet. 2014. Hints on Quality Evaluation of Self-Systems. In 2014 IEEE Eighth International Conference on Self-Adaptive and Self-Organizing Systems. IEEE, 185–186.

[43]

Claudia Raibulet and Francesca Arcelli Fontana. 2017. Evaluation of Self-Adaptive Systems: A Women Perspective(ECSA ’17). Association for Computing Machinery, New York, NY, USA, 23–30. https://doi.org/10.1145/3129790.3129825

Digital Library

[44]

C Raibulet, F Arcelli Fontana, R Capilla, and C Carrillo. 2017. An Overview on Quality Evaluation of Self-Adaptive Systems. In Managing Trade-Offs in Adaptable Software Architectures. Elsevier, 325–352.

[45]

Claudia Raibulet and Laura Masciadri. 2009. Evaluation of dynamic adaptivity through metrics: an achievable target?. In 2009 Joint Working IEEE/IFIP Conference on Software Architecture & European Conference on Software Architecture. IEEE, 341–344.

[46]

Andres J Ramirez, David B Knoester, Betty HC Cheng, and Philip K McKinley. 2011. Plato: a genetic algorithm approach to run-time reconfiguration in autonomic computing systems. Cluster Computing 14, 3 (2011), 229–244.

Digital Library

[47]

Ajita Rattani, Gian Luca Marcialis, and Fabio Roli. 2011. Self adaptive systems: An experimental analysis of the performance over time. In 2011 IEEE Workshop on Computational Intelligence in Biometrics and Identity Management (CIBIM). IEEE, 36–43.

[48]

Fabio Roli, Luca Didaci, and Gian Luca Marcialis. 2008. Adaptive biometric systems that can improve with use. In Advances in Biometrics. Springer, 447–471.

[49]

C. Ryan and P. Rossi. 2005. Software, performance and resource utilisation metrics for context-aware mobile applications. In 11th IEEE International Software Metrics Symposium (METRICS’05). 10 pp.–12.

[50]

L Emiliano Sanchez, J Andres Diaz-Pace, Alejandro Zunino, Sabine Moisan, and Jean-Paul Rigault. 2015. An approach based on feature models and quality criteria for adapting component-based systems. Journal of Software Engineering Research and Development 3, 1(2015), 10.

[51]

Cristina Silvano, Giovanni Agosta, Andrea Bartolini, Andrea R. Beccari, Luca Benini, Loïc Besnard, João Bispo, Radim Cmar, João M.P. Cardoso, Carlo Cavazzoni, Daniele Cesarini, Stefano Cherubin, Federico Ficarelli, Davide Gadioli, Martin Golasowski, Antonio Libri, Jan Martinovič, Gianluca Palermo, Pedro Pinto, Erven Rohou, Kateřina Slaninová, and Emanuele Vitali. 2019. The ANTAREX domain specific language for high performance computing. Microprocessors and Microsystems 68 (2019), 58 – 73. https://doi.org/10.1016/j.micpro.2019.05.005

Digital Library

[52]

Filippo Sironi, Davide B Bartolini, Simone Campanoni, Fabio Cancare, Henry Hoffmann, Donatella Sciuto, and Marco D Santambrogio. 2012. Metronome: operating system level performance management via self-adaptive computing. In Proceedings of the 49th Annual Design Automation Conference. ACM, 856–865.

Digital Library

[53]

Mehdi Sliem, Nabila Salmi, and Malika Ioualalen. 2015. Using performance modelling and analysis for self-adaptive resources allocation systems: A case study. In 2015 12th International Symposium on Programming and Systems (ISPS). IEEE, 1–10.

[54]

Gabriel Tamura, Norha M. Villegas, Hausi A. Müller, Laurence Duchien, and Lionel Seinturier. 2013. Improving Context-Awareness in Self-Adaptation Using the DYNAMICO Reference Model. In Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems(SEAMS ’13). IEEE Press, 153–162.

Digital Library

[55]

Danny Weyns. 2019. Software Engineering of Self-adaptive Systems. In Handbook of Software Engineering. Springer, 399–443.

[56]

Danny Weyns and Radu Calinescu. 2015. Tele assistance: A self-adaptive service-based system exemplar. In 2015 IEEE/ACM 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. IEEE, 88–92.

Digital Library

Cited By

Kraaijveld KRaibulet C(2024)Adaptive Strategies Metric SuiteEvaluation of Novel Approaches to Software Engineering10.1007/978-3-031-64182-4_14(296-320)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-64182-4_14
Hasan SElkhatib Y(2023)On the Interplay Between Self-adaptation and Energy EfficiencyProceedings of the 9th International Conference on Advanced Intelligent Systems and Informatics 202310.1007/978-3-031-43247-7_12(130-139)Online publication date: 18-Sep-2023
https://doi.org/10.1007/978-3-031-43247-7_12
Naqvi MMalik SAstekin MMoonen L(2022)On Evaluating Self-Adaptive and Self-Healing Systems using Chaos Engineering2022 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)10.1109/ACSOS55765.2022.00018(1-10)Online publication date: Sep-2022
https://doi.org/10.1109/ACSOS55765.2022.00018

Index Terms

A Catalog of Performance Measures for Self-Adaptive Systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Quality Evaluation of Self-Adaptive Systems: Challenges and Opportunities
SBES '19: Proceedings of the XXXIII Brazilian Symposium on Software Engineering

Self-adaptive systems (SAS) can adapt their behavior to suit user preferences or contexts, as well as monitor their performance and adjust it if necessary. In addition to adaptation operations, self-adaptive systems communicate with sensors, actuators, ...
Percent perfect performance (PPP)

An information retrieval performance measure that is interpreted as the percent of perfect performance (PPP) can be used to study the effects of the inclusion of specific document features or feature classes or techniques in an information retrieval ...
Towards Self-Aware Performance and Resource Management in Modern Service-Oriented Systems
SCC '10: Proceedings of the 2010 IEEE International Conference on Services Computing

Modern service-oriented systems have increasingly complex loosely-coupled architectures that often exhibit poor performance and resource efficiency and have high operating costs. This is due to the inability to predict at run-time the effect of dynamic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBQS '21: Proceedings of the XX Brazilian Symposium on Software Quality

November 2021

320 pages

ISBN:9781450395533

DOI:10.1145/3493244

Editors:
Rodrigo Santos
UNIRIO
,
Sandro Ronaldo Bezerra Oliveira
UFPA
,
Ismayle de Sousa Santos
UFC
,
Sabrina Marczak
PUCRS
,
Davi Viana
UFMA
,
Edna Dias Canedo
UnB
,
Adriano Bessa Albuquerque
UNIFOR
,
Marco Paludo
PUCPR
,
Ana Regina Rocha
UFRJ
,
Ivan Machado
UFBA
,
Uirá Kulesza
UFRN
,
Vinícius Garcia
UFPE
,
Gleison Santos
UNIRIO
,
Natasha Valentim
UFPR
,
Monalessa Barcellos
UFES
,
Adriano Bessa Albuquerque
UNIFOR
,
Ana Regina Cavalcanti da Rocha
COPPE/UFRJ
,
Gleison Santos
UNIRIO
,
Ivan Machado
UFBA
,
Monalessa Perini Barcellos
UFES
,
Sheila Reinehr
PUCPR
,
Tayana Conte
UFAM
,
Awdren Fontão
UFMS
,
Crescêncio Lima
IFBA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SBQS '21

SBQS '21: XX Brazilian Symposium on Software Quality

November 8 - 11, 2021

Virtual Event, Brazil

Acceptance Rates

Overall Acceptance Rate 35 of 99 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
131
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 19 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kraaijveld KRaibulet C(2024)Adaptive Strategies Metric SuiteEvaluation of Novel Approaches to Software Engineering10.1007/978-3-031-64182-4_14(296-320)Online publication date: 10-Jul-2024
https://doi.org/10.1007/978-3-031-64182-4_14
Hasan SElkhatib Y(2023)On the Interplay Between Self-adaptation and Energy EfficiencyProceedings of the 9th International Conference on Advanced Intelligent Systems and Informatics 202310.1007/978-3-031-43247-7_12(130-139)Online publication date: 18-Sep-2023
https://doi.org/10.1007/978-3-031-43247-7_12
Naqvi MMalik SAstekin MMoonen L(2022)On Evaluating Self-Adaptive and Self-Healing Systems using Chaos Engineering2022 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)10.1109/ACSOS55765.2022.00018(1-10)Online publication date: Sep-2022
https://doi.org/10.1109/ACSOS55765.2022.00018

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents