Abstract
As our dependence on automated systems grows, so does the need for guaranteeing their safety, cybersecurity, and privacy (SCP). Dedicated methods for verification and validation (V&V) must be used to this end and it is necessary that the methods and their characteristics can be clearly differentiated. This can be achieved via method classifications. However, we have experienced that existing classifications are not suitable to categorise V&V methods for SCP of automated systems. They do not pay enough attention to the distinguishing characteristics of this system type and of these quality concerns. As a solution, we present a new classification developed in the scope of a large-scale industry-academia project. The classification considers both the method type, e.g., testing, and the concern addressed, e.g., safety. Over 70 people have successfully used the classification on 53 methods. We argue that the classification is a more suitable means to categorise V&V methods for SCP of automated systems and that it can help other researchers and practitioners.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amalthea4public project: D3.1 - Analysis of state of the art V&V techniques (2015)
Arfelt, E., Basin, D., Debois, S.: Monitoring the GDPR. In: Sako, K., Schneider, S., Ryan, P.Y.A. (eds.) ESORICS 2019. LNCS, vol. 11735, pp. 681–699. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29959-0_33
Avizienis, A., et al.: Fundamental concepts of dependability. University of Newcastle (2001)
Barbosa, R., et al.: The VALU3S ECSEL project: verification and validation of automated systems safety and security. In: DSD 2020 (2020)
Bartocci, E., Manjunath, N., Mariani, L., Mateis, C., Ničković, D.: Automatic failure explanation in CPS models. In: Ölveczky, P.C., Salaün, G. (eds.) SEFM 2019. LNCS, vol. 11724, pp. 69–86. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30446-1_4
Belmonte, L., et al.: Feeling of safety and comfort towards a socially assistive unmanned aerial vehicle that monitors people in a virtual home. Sensors 21(3), 908 (2021)
Bozzano, M., Cimatti, A., Griggio, A., Mattarei, C.: Efficient anytime techniques for model-based safety analysis. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 603–621. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_41
Cassar, I., et al: A survey of runtime monitoring instrumentation techniques. PrePost@iFM (2017)
CENELEC: EN 50128 - Railway applications - Communication, signalling and processing systems - Software for railway control and protection systems (2020)
Cimatti, A., Tian, C., Tonetta, S.: Assumption-based runtime verification with partial observability and resets. In: Finkbeiner, B., Mariani, L. (eds.) RV 2019. LNCS, vol. 11757, pp. 165–184. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32079-9_10
Clarke, E.M., et al.: Handbook of Model Checking. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-319-10575-8
de la Vara, J.L., et al.: Assurance and certification of cyber-physical systems: the AMASS open source ecosystem. J. Syst. Softw. 171, 110812 (2021)
Dias, R., et al.: Verifying concurrent programs using contracts. In: ICST 2017 (2017)
Duckham, M., Kulik, L.: Simulation of obfuscation and negotiation for location privacy. In: Cohn, A.G., Mark, D.M. (eds.) COSIT 2005. LNCS, vol. 3693, pp. 31–48. Springer, Heidelberg (2005). https://doi.org/10.1007/11556114_3
Fonseca, J., et al.: Analysis of field data on web security vulnerabilities. IEEE Trans. Dependable Secure Comput. 11(2), 89–100 (2014)
Gallina, B., et al.: Multi‐concern dependability‐centered assurance for space systems via ConcertoFLA. Ada-Europe (2018)
Halfind, W.G.J., et al.: A classification of SQL injection attacks and countermeasures. In: ISSSE 2006 (2006)
Herdt, V., et al.: Efficient cross-level testing for processor verification: a RISC-V case-study. In: FDL 2020 (2020)
Humbatova, N., et al.: Taxonomy of real faults in deep learning systems. In: ICSE 2020 (2020)
IEC: IEC 61508 - Functional safety of electrical/electronic/programmable electronic safety-related systems (2011)
Hähnle, R., Huisman, M.: Deductive software verification: from pen-and-paper proofs to industrial tools. In: Steffen, B., Woeginger, G. (eds.) Computing and Software Science. LNCS, vol. 10000, pp. 345–373. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-91908-9_18
IEEE: IEEE Std 1012 - IEEE Standard for System, Software, and Hardware V&V (2016)
Kammueller, F.: Formal modeling and analysis of data protection for GDPR compliance of IoT healthcare systems. In: SMC 2018 (2018)
Khalastchi, E., Kalech, M.: On fault detection and diagnosis in robotic systems. ACM Comput. Surv. 51(1), 9 (2018)
Kuhn, T., Antonino, P.O., Bachorek, A.: A simulator coupling architecture for the creation of digital twins. In: Muccini, H., et al. (eds.) ECSA 2020. CCIS, vol. 1269, pp. 326–339. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59155-7_25
Kramer, A., Legeard, B.: Model-Based Testing Essentials. Wiley, Hoboken (2016)
Laskey, M., et al.: DART: noise injection for robust imitation learning. In: CoRL 2017 (2017)
Luckcuck, M., et al.: Formal specification and verification of autonomous robotic systems: a survey. ACM Comput. Surv. 52(5), 100 (2019)
Nair, S., et al.: An extended systematic literature review on provision of evidence for safety certification. Inf. Softw. Technol. 56(7), 689–717 (2014)
Natella, R., et al.: Assessing dependability with software fault injection: a survey. ACM Comput. Surv. 48(3), 44 (2016)
Oxford UK Dictionary: method (2021). https://www.lexico.com/definition/method
Pan, L., et al.: Cyber security attacks to modern vehicular systems. J. Inf. Secur. Appl. 36, 30–100 (2017)
Pandit, H.J., O’Sullivan, D., Lewis, D.: Test-driven approach towards GDPR compliance. In: Acosta, M., Cudré-Mauroux, P., Maleshkova, M., Pellegrini, T., Sack, H., Sure-Vetter, Y. (eds.) SEMANTiCS 2019. LNCS, vol. 11702, pp. 19–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33220-4_2
Parra, E., et al.: Advances in artefact quality analysis for safety-critical systems. In: ISSRE 2019 (2019)
Paz, A., El Boussaidi, G.: A requirements modelling language to facilitate avionics software verification and certification. In: RET 2019 (2019)
Rival, X., Yi, K.: Introduction to Static Analysis. An Abstract Interpretation Perspective. MIT Press (2020)
Sangchoolie, B., et al.: A study of the interplay between safety and security using model-implemented fault injection. In: EDCC 2018 (2018)
Savary, A., Frappier, M., Leuschel, M., Lanet, J.-L.: Model-based robustness testing in event-B using mutation. In: Calinescu, R., Rumpe, B. (eds.) SEFM 2015. LNCS, vol. 9276, pp. 132–147. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22969-0_10
Skoglund, M., et al.: Black-box testing for security-informed safety of automated driving systems. In: VTC 2021-Spring (2021)
Timperley, C.S., et al.: Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In: ICST 2018 (2018)
Tsachouridis, V.A., et al.: Formal analysis of the Schulz matrix inversion algorithm: a paradigm towards computer aided verification of general matrix flow solvers. Numer. Algebra Control Optim. 10(2), 177–206 (2020)
US DoD: Defense Modeling & Simulation Coordination Office, V&V Technique Taxonomy (2001). https://vva.msco.mil/default.htm?Ref_Docs/VVTechniques/
VALU3S project: D3.1 - V&V methods for SCP evaluation of automated systems (2021)
Yang, Y., et al.: Man-in-the-middle attack test-bed investigating cyber-security vulnerabilities in smart grid SCADA systems. In: SUPERGEN 2012 (2012)
Acknowledgments
The research leading to this paper has received funding from the VALU3S (H2020-ECSEL grant agreement no 876852; Spain’s MICINN ref. PCI2020-112001), iRel4.0 (H2020-ECSEL grant agreement no 876659; MICINN ref. PCI2020-112240), and Treasure (JCCM SBPLY/19/180501/000270; European Regional Development Fund) projects, and from the Ramon y Cajal Program (MICINN RYC-2017-22836; European Social Fund). We are also grateful to all the VALU3S partners that have provided input and feedback for the development of the classification.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
de la Vara, J.L. et al. (2021). A Proposal for the Classification of Methods for Verification and Validation of Safety, Cybersecurity, and Privacy of Automated Systems. In: Paiva, A.C.R., Cavalli, A.R., Ventura Martins, P., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2021. Communications in Computer and Information Science, vol 1439. Springer, Cham. https://doi.org/10.1007/978-3-030-85347-1_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-85347-1_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85346-4
Online ISBN: 978-3-030-85347-1
eBook Packages: Computer ScienceComputer Science (R0)