ATheNA-S: A Testing Tool for Simulink Models Driven by Software Requirements and Domain Expertise
Authors: 
Federico Formica, Mohammad Mahdi Mahboob, Mehrnoosh Askarpour, Claudio MenghiAuthors Info & Claims
Pages 587 - 591
Abstract
Search-based software testing (SBST) is widely used to verify software systems. SBST iteratively generates new test inputs driven by fitness functions, i.e., objective functions that guide the test case generation. In previous work, we proposed ATheNA, a novel SBST framework that combines fitness functions automatically generated from requirements' specifications with those manually defined by engineers, and showed its effectiveness. This tool demonstration paper describes ATheNA-S, an instance of ATheNA that targets Simulink models. We demonstrate our tool using an automotive case study and present our implementation and design decisions. A video walkthrough of the case study is available on YouTube: youtu.be/dhw9rwO7L4k.
References
[1]
Aleti, A., Moser, I., Grunske, L.: Analysing the fitness landscape of search-based software testing problems. Automated Software Engineering 24(3), 603–621 (2017).
[2]
Almulla, H., Gay, G.: Learning how to search: Generating exception-triggering tests through adaptive fitness function selection. In: International Conference on Software Testing, Validation and Verification. pp. 63–73. IEEE (2020)
[3]
Amal, B., Kessentini, M., Bechikh, S., Dea, J., Said, L.B.: On the use of machine learning and search-based software engineering for ill-defined fitness function: A case study on software refactoring. In: Le Goues, C., Yoo, S. (eds.) International Symposium on Search-Based Software Engineering. pp. 31–45. Springer, Cham (2014)
[4]
Annpureddy, Y., Liu, C., Fainekos, G., Sankaranarayanan, S.: S-taliro: A tool for temporal logic falsification for hybrid systems. In: Tools and Algorithms for the Construction and Analysis of Systems. pp. 254–257. Springer (2011)
[5]
Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: International Conference on Software Engineering. pp. 1–10. ACM (2011)
[6]
ATheNA. https://github.com/ATheNA-SBST/ATheNA (January 2024 [Online])
[7]
ATheNA Add-on. https://www.mathworks.com/matlabcentral/fileexchange/116095-athena (May 2024 [Online])
[8]
Automotive Electrical System Simulation and Control. https://it.mathworks.com/matlabcentral/fileexchange/25674-automotive-electrical-system-simulation-and-control (April 2022 [Online])
[9]
Bajaj, A., Sangwan, O.P.: A systematic literature review of test case prioritization using genetic algorithms. IEEE Access 7, 126355–126375 (2019).
[10]
United States Department of Energy. https://www.energy.gov/ (April 2022 [Online])
[11]
Ernst, G., Arcaini, P., Fainekos, G., Formica, F., Inoue, J., Khandait, T., Mahboob, M.M., Menghi, C., Pedrielli, G., Waga, M., Yamagata, Y., Zhang, Z.: ARCH-COMP 2022 Category Report: Falsification with Ubounded Resources. In: International Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH22). EPiC Series in Computing, vol. 90, pp. 204–221. EasyChair (2022)
[12]
Fainekos, G.E., Pappas, G.J.: Robustness of temporal logic specifications for continuous-time signals. Theoretical Computer Science 410(42), 4262–4291 (2009).
[13]
Formica, F., Fan, T., Menghi, C.: Search-based software testing driven by automatically generated and manually defined fitness functions. Transactions on Software Engineering and Methodology (2023).
[14]
General Motors. https://www.gm.com/ (April 2022 [Online])
[15]
Harman, M., Clark, J.: Metrics are fitness functions too. In: International Symposium on Software Metrics. pp. 58–69. IEEE (2004)
[16]
Hart, E., Ross, P.: Gavel-a new tool for genetic algorithm visualization. Transactions on Evolutionary Computation 5(4), 335–348 (2001).
[17]
ATheNA Demo Walkthrough. https://www.youtube.com/watch?v=dhw9rwO7L4k (January 2024 [Online])
[18]
ATheNA Installation Guide. https://www.youtube.com/watch?v=F8hhTQ8nLts (January 2024 [Online])
[19]
Jin, X., Deshmukh, J.V., Kapinski, J., Ueda, K., Butts, K.: Powertrain control verification benchmark. In: International conference on Hybrid systems: computation and control. pp. 253–262. ACM (2014)
[20]
Kapinski, J., Deshmukh, J.V., Jin, X., Ito, H., Butts, K.: Simulation-based approaches for verification of embedded control systems: An overview of traditional and advanced modeling, testing, and verification techniques. Control Systems Magazine 36(6), 45–64 (2016).
[21]
Kim, Y.H., Moon, B.R.: Visualization of the fitness landscape, a steady-state genetic search, and schema traces. In: Annual Conference on Genetic and Evolutionary Computation. pp. 686–686. Morgan Kaufmann (2002)
[22]
Kim, Y.H., Moon, B.R.: New usage of sammon’s mapping for genetic visualization. In: Cantú-Paz, E., Foster, J.A., Deb, K., Davis, L., Roy, R., O’Reilly, U., Beyer, H., Standish, R.K., Kendall, G., Wilson, S.W., Harman, M., Wegener, J., Dasgupta, D., Potter, M.A., Schultz, A.C., Dowsland, K.A., Jonoska, N., Miller, J.F. (eds.) Genetic and Evolutionary Computation Conference. pp. 1136–1147. Springer, Berlin, Heidelberg (2003)
[23]
MathWorks: MathWorks. https://www.mathworks.com (April 2022 [Online])
[24]
MathWorks: The EcoCAR Mobility Challenge. https://it.mathworks.com/academia/student-competitions/ecocar.html (April 2022 [Online])
[25]
Menghi, C., Arcaini, P., Baptista, W., Ernst, G., Fainekos, G., Formica, F., Gon, S., Khandait, T., Kundu, A., Pedrielli, G., et al.: ARCH-COMP 2023 Category Report: Falsification. In: 10th International Workshop on Applied Verification of Continuous and Hybrid Systems. ARCH23. vol. 96, pp. 151–169 (2023)
[26]
Menghi, C., Nejati, S., Briand, L., Parache, Y.I.: Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification. In: International Conference on Software Engineering. pp. 372–384. IEEE/ACM (2020)
[27]
Miller, S.: Medical ventilator model in simscape. https://www.mathworks.com/matlabcentral/fileexchange/75012-medical-ventilator-model-in-simscape (September 2022 [Online])
[28]
Nejati, S., Sorokin, L., Safin, D., Formica, F., Mahboob, M.M., Menghi, C.: Reflections on surrogate-assisted search-based testing: A taxonomy and two replication studies based on industrial adas and simulink models. Information and Software Technology 163, 107286 (2023).
[29]
Pohlheim, H.: Visualization of evolutionary algorithms-set of standard techniques and multidimensional visualization. In: Genetic and Evolutionary Computation Conference. vol. 1, pp. 533–540 (1999)
[30]
Sahin, O., Akay, B.: Comparisons of metaheuristic algorithms and fitness functions on software test data generation. Applied Soft Computing 49, 1202–1214 (2016).
[31]
Salahirad, A., Almulla, H., Gay, G.: Choosing the fitness function for the job: Automated generation of test suites that detect real faults. Software Testing, Verification and Reliability 29(4-5), e1701 (2019).
[32]
Thibeault, Q., Anderson, J., Chandratre, A., Pedrielli, G., Fainekos, G.: Psy-taliro: A python toolbox for search-based test generation for cyber-physical systems. In: International Conference on Formal Methods for Industrial Critical Systems. pp. 223–231. Springer (2021)
[33]
Tuncali, C.E., Hoxha, B., Ding, G., Fainekos, G., Sankaranarayanan, S.: Experience report: Application of falsification methods on the uxas system. In: NASA Formal Methods Symposium. pp. 452–459. Springer, Cham (2018)
[34]
Wilkerson, J.L., Tauritz, D.R.: A guide for fitness function design. In: Annual Conference Companion on Genetic and Evolutionary Computation. p. 123–124. ACM (2011)
Index Terms
- ATheNA-S: A Testing Tool for Simulink Models Driven by Software Requirements and Domain Expertise
Recommendations
Search-Based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions
Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the ...
A dynamic fitness function for search based software testing
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference CompanionSearch Based Software Testing (SBST) formulates testing as an optimization problem, hence some search algorithms can be used to tackle it. The goal of SBST is to improve various test adequacy criteria. There are different types of coverage criteria, and ...
Automated Software Testing for Matlab
Software testing can improve software quality. To test effectively, scientists and engineers should know how to write and run tests, define appropriate test cases, determine expected outputs, and correctly handle floating-point arithmetic. Using Matlab ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Natural Sciences and Engineering Research Council of Canada
Conference
FSE '24
Sponsor:
FSE '24: 32nd ACM International Conference on the Foundations of Software Engineering
July 15 - 19, 2024
Porto de Galinhas, Brazil
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 33Total Downloads
- Downloads (Last 12 months)33
- Downloads (Last 6 weeks)5
Reflects downloads up to 28 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in