Natural language requirements testability measurement based on requirement smells
Pages 13051 - 13085
Abstract
Requirements form the basis for defining software systems’ obligations and tasks. Testable requirements help prevent failures, reduce maintenance costs, and make it easier to perform acceptance tests. However, despite the importance of measuring and quantifying requirements testability, no automatic approach for measuring requirements testability has been proposed based on the requirements smells, which are at odds with the requirements testability. This paper presents a mathematical model to evaluate and rank the natural language requirements testability based on an extensive set of nine requirements smells, detected automatically, and acceptance test efforts determined by requirement length and its application domain. Most of the smells stem from uncountable adjectives, context-sensitive, and ambiguous words. A comprehensive dictionary is required to detect such words. We offer a neural word embedding technique to generate such a dictionary automatically. Using the dictionary, we could automatically detect Polysemy smell (domain-specific ambiguity) for the first time in 10 application domains. Our empirical study on nearly 1000 software requirements from six well-known industrial and academic projects demonstrates that the proposed smell detection approach outperforms Smella, a state-of-the-art tool, in detecting requirements smells. The Precision and Recall of smell detection are improved with an average of 0.03 and 0.33, respectively, compared to the state-of-the-art. The proposed requirement testability model measures the testability of 985 requirements with a mean absolute error of 0.12 and a mean squared error of 0.03, demonstrating the model’s potential for practical use.
References
[1]
dos Santos EC and Vilain P Automated acceptance tests as software requirements: an experiment to compare the applicability of fit tables and Gherkin language 2018 Cham Springer 104-119
[2]
Femmer H and Vogelsang A Requirements quality is quality in use IEEE Softw 2019 36 3 83-91
[3]
Hayes JH, Li W, Yu T, Han X, Hays M, Woodson C (2015) Measuring requirement quality to predict testability. In: 2015 IEEE second international workshop on artificial intelligence for requirements engineering (AIRE). IEEE, pp 1–8. URL: http://ieeexplore.ieee.org/document/7337622/
[4]
ISO/IEC/IEEE (2017a) ISO/IEC/IEEE 24765:2017 Systems and software engineering-vocabulary. [Online]. Available: https://www.iso.org/standard/71952.html
[5]
Garousi V, Felderer M, and KıLıçaslan FN A survey on software testability Inf Softw Technol 2019 108 35-64
[6]
Khan RA and Mustafa K Metric based testability model for object oriented design (MTMOOD) ACM SIGSOFT Softw Eng Notes 2009 34 2 1
[7]
Shaheen MR, Du Bousquet L (2014) Survey of source code metrics for evaluating testability of object oriented systems. Technical Report, Inria France. URL: https://hal.inria.fr/hal-00953403
[8]
Terragni V, Salza P, Pezzè M (2020) Measuring software testability modulo test quality. In: Proceedings of the 28th international conference on program comprehension, pp 241–251, New York, NY, USA, ACM. ISBN 9781450379588. URL: https://dl.acm.org/doi/10.1145/3387904.3389273
[9]
Zakeri-Nasrabadi M, Parsa S (2021) Learning to predict software testability. pp 1–5. IEEE, 3. ISBN 978-1-6654-1241-4.
[10]
Zakeri-Nasrabadi M and Parsa S An ensemble meta-estimator to predict source code testability Appl Soft Comput 2022 129
[11]
Ammann P and Offutt J Introduction to software testing 2016 Cambridge Cambridge University Press
[12]
Beer A, Junker M, Femmer H, Felderer M (2017) Initial investigations on the influence of requirement smells on test-case design. In: 2017 IEEE 25th international requirements engineering conference workshops (REW), pp 323–326. IEEE. ISBN 978-1-5386-3488-2. URL: http://ieeexplore.ieee.org/document/8054872/
[13]
Izosimov V, Ingelsson U, Wallin A (2012) Requirement decomposition and testability in development of safety-critical automotive components. pp 74–86.
[14]
Gonzalo G, Fuentes José M, Juan L, Omar H, and Valentín M A framework to measure and improve the quality of textual requirements Requir Eng 2013 18 1 25-41
[15]
Femmer H, Fernández DM, Juergens E, Klose M, Zimmer I, Zimmer J (2014) Rapid requirements checks with requirements smells: two case studies. In: Proceedings of the 1st international workshop on rapid continuous software engineering-RCoSE 2014, pp 10–19, New York, New York, USA, ACM Press. ISBN 9781450328562.
[16]
Henning F, Méndez FD, Stefan W, and Sebastian E Rapid quality assurance with requirements smells J Syst Softw 2017 123 190-213
[17]
Jurafsky D and Martin JH Speech and language processing 2009 2 Upper Saddle River Prentice-Hall Inc.
[18]
Petrov S, Das D, McDonald R (2011) A universal part-of-speech tagset. Computing Research Repository-CORR
[19]
Christopher D Manning; and Hinrich Schütze. Foundations of statistical natural language processing 1999 Cambridge MIT Press
[20]
Wilson WM, Rosenberg LH, Hyatt LE (1997) Automated analysis of requirement specifications. In: Proceedings of the 19th international conference on Software engineering-ICSE ’97, pp 161–171, New York, New York, USA, ACM Press. ISBN 0897919149.
[21]
Fabbrini F, Fusani M, Gnesi S, Lami G (2001) The linguistic approach to the natural language requirements quality: benefit of the use of an automatic tool. In: Proceedings 26th annual NASA goddard software engineering workshop, pp 97–105, IEEE Comput. Soc, 2001. ISBN 0-7695-1456-1. URL: http://ieeexplore.ieee.org/document/992662/
[22]
Tjong SF, Berry DM (2013) The design of SREE-a prototype potential ambiguity finder for requirements specifications and lessons learned. pp 80–95.
[23]
Gleich B, Creighton O, Kof L (2010) Ambiguity detection: towards a tool explaining ambiguity sources. pp 218–232.
[24]
Berry DM, Kamsties E, Krieger MM (2003) From contract drafting to software specification: linguistic sources of ambiguity. https://cs.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf
[25]
Ferrari A, Donati B, Gnesi S (2017) Detecting domain-specific ambiguities: an NLP approach based on Wikipedia crawling and word embeddings. In: 2017 IEEE 25th international requirements engineering conference workshops (REW). pp 393–399. IEEE. ISBN 978-1-5386-3488-2. URL: http://ieeexplore.ieee.org/document/8054883/
[26]
Ferrari A, Esuli A, Gnesi S (2018) Identification of cross-domain ambiguity with language models. In: 2018 5th international workshop on artificial intelligence for requirements engineering (AIRE), pp 31–38. IEEE. ISBN 978-1-5386-8404-7. URL: https://ieeexplore.ieee.org/document/8501308/
[27]
Alessio F and Andrea E An NLP approach for cross-domain ambiguity detection in requirements engineering Autom Softw Eng 2019 26 3 559-598
[28]
Chetan A, Mehrdad S, Lionel B, and Frank Z Automated checking of conformance to requirements templates using natural language processing IEEE Trans Softw Eng 2015 41 10 944-968
[29]
Liping Z, Waad A, Alessio F, Letsholo Keletso J, Ajagbe Muideen A, Erol-Valeriu C, and Batista-Navarro Riza T Natural language processing for requirements engineering: a systematic mapping study ACM Comput Surv 2021
[30]
IEEE (1990) IEEE 610.12-1990-IEEE Standard Glossary of Software Engineering Terminology. URL: https://standards.ieee.org/standard/610_12-1990.html
[31]
ISO/IEC/IEEE (2017b) ISO/IEC/IEEE 12207:2017(en) Systems and software engineering-software life cycle processes. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec-ieee:12207:ed-1:v1:en
[32]
ISO/IEC/IEEE (2018) ISO/IEC/IEEE 29148:2018(en) Systems and software engineering-life cycle processes-requirements engineering. URL: https://www.iso.org/obp/ui#iso:std:iso-iec-ieee:29148:ed-2:v1:en
[33]
Lami G, Gnesi S, Fabbrini F, Fusani M, Trentanni G (2004) An automatic tool for the analysis of natural language requirements. Informe técnico, CNR Information Science and Technology Institute, Pisa, Italia, Setiembre
[34]
Hui Y, de Roeck A, Vincenzo G, Alistai W, and Bashar N Analysing anaphoric ambiguity in natural language requirements Requir Eng 2011 16 3 163-189
[35]
Alessio F, Gloria G, Benedetta R, Iacopo T, Stefano B, Alessandro F, and Stefania G Detecting requirements defects with NLP patterns: an industrial experience in the railway domain Empir Softw Eng 2018 23 6 3684-3733
[36]
Dalpiaz F, van der Schalk I, Brinkkemper S, Aydemir FB, and Lucassen G Detecting terminological ambiguity in user stories: tool and experimentation Inf Softw Technol 2019 110 3-16
[37]
Webber FDS (2015) Semantic folding theory and its application in semantic fingerprinting. arXiv:1511.08855
[38]
Ezzini S, Abualhaija S, Arora C, Sabetzadeh M, Briand LC (2021) Using domain-specific corpora for improved handling of ambiguity in requirements. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE). pp 1485–1497. IEEE. ISBN 978-1-6654-0296-5. URL: https://ieeexplore.ieee.org/document/9402055/
[39]
Fantechi A, Gnesi S, Semini L (2019) Applying the QuARS tool to detect variability. In: Proceedings of the 23rd international systems and software product line conference volume B - SPLC ’19, pp 1–4, New York, New York, USA, ACM Press. ISBN 9781450366687. URL: http://dl.acm.org/citation.cfm?doid=3307630.3342388
[40]
Chantree F, Nuseibeh B, de Roeck A, Willis A (2006) Identifying nocuous ambiguities in natural language requirements. In: 14th IEEE international requirements engineering conference (RE’06), pp 59–68. IEEE. ISBN 0-7695-2555-5. URL: http://ieeexplore.ieee.org/document/1704049/
[41]
Tom Y, Devamanyu H, Soujanya P, and Erik C Recent trends in deep learning based natural language processing IEEE Comput Intell Mag 2018 13 3 55-75
[42]
de Bruijn F and Dekkers HL Roel W and Anne P Ambiguity in natural language software requirements: a case study Requirements engineering: foundation for software quality 2010 Berlin Springer 233-247
[43]
QRACorp. ( 2021) QVscribe. [Online]. Available: https://qracorp.com
[44]
Visure Solutions Inc. (2020) Visure Quality Analyzer-requirements quality metrics. [Online]. Available: https://visuresolutions.com/visure-quality-analyzer-write-requirements
[45]
Ferrari A, Spagnolo GO, Fiscella A, Parente G (2019) QuOD: an NLP tool to improve the quality of business process descriptions. pp 267–281. URL: http://link.springer.com/10.1007/978-3-030-30985-5_17
[46]
Shevchenko A, Lytvyn M, Lider D (2009) Grammarly. [Online]. Available: https://www.grammarly.com/grammar-check
[47]
Song X, Wu N, Song S, and Stojanovic V Switching-like event-triggered state estimation for reaction-diffusion neural networks against dos attacks Neural Proc Lett 2013 55 8997-9018
[48]
Tengda W, Xiaodi L, and Vladimir S Input-to-state stability of impulsive reaction-diffusion neural networks with infinite distributed delays Nonlinear Dyn 2021 103 1733-1755
[49]
Xiaona S, Peng S, Shuai S, and Vladimir S Quantized neural adaptive finite-time preassigned performance control for interconnected nonlinear systems Neural Comput Appl 2023 35 15429-15446
[50]
Zakeri-Nasrabadi M, Parsa S, and Kalaee A Format-aware learn & fuzz: deep test data generation for efficient fuzzing Neural Comput Appl 2021
[51]
Zhilu X, Xiaodi L, and Vladimir S Exponential stability of nonlinear state-dependent delayed impulsive systems with applications Nonlinear Anal Hybrid Syst 2021 42
[52]
Basili VR, Caldiera G, Rombach HD (1994) The goal question metric approach. Encycl Softw Eng, 528–532
[53]
Fischbach J, Vogelsang A, Spies D, Wehrle A, Junker M, Freudenstein D (2020) SPECMATE: automated creation of test cases from acceptance criteria. In: 2020 IEEE 13th international conference on software testing, validation and verification (ICST), pp 321–331. IEEE, ISBN 978-1-7281-5778-8. URL: https://ieeexplore.ieee.org/document/9159056/
[54]
Robertson S and Robertson J Mastering the requirements process 2006 Boston ACM Press books, Addison-Wesley
[55]
Fabbrini F, Fusani M, Gnesi S, Lami G (2000) Quality evaluation of software requirement specifications. In: Proceedings of the software and internet quality week 2000 conference, pp 1–18
[56]
Ian F (2002) Alexander and Richard Stevens. Addison-Wesley Professional, Writing better requirements. 0321131630
[57]
Huertas C, Juárez-Ramírez R (2013) Towards assessing the quality of functional requirements using english/spanish controlled languages and context free grammar. In: Proc. third international conference on digital information and communication technology and its applications (DICTAP 2013), Ostrava, Czech Republic on, pp 234–241. Citeseer
[58]
Beer A, Felderer M (2018) Measuring and improving testability of system requirements in an industrial context by applying the goal question metric approach. In: Proceedings of the 5th international workshop on requirements engineering and testing, RET ’18, pp 25–32, New York, NY, USA. Association for Computing Machinery. ISBN 9781450357494.
[59]
Tomas M, Kai C, Greg C, Jeffrey D (2013) Efficient estimation of word representations in vector space
[60]
Giatsoglou M, Vozalis MG, Diamantaras K, Vakali A, Sarigiannidis G, and Chatzisavvas KC Sentiment analysis leveraging emotions and word embeddings Expert Syst Appl 2017 69 214-224
[61]
Juergens E, Deissenboeck F, Feilkas M, Hummel B, Schaetz B, Wagner S, Domann C, Streit J (2010) Can clone detection support quality assessments of requirements specifications? In: Proceedings of the 32nd ACM/IEEE international conference on software engineering-ICSE ’10, volume 2, pp 79, New York, New York, USA. ACM Press. ISBN 9781605587196. URL: http://portal.acm.org/citation.cfm?doid=1810295.1810308
[62]
Fontana FA, Ferme V, Zanoni M, Roveda R (2015) Towards a prioritization of code debt: a code smell Intensity Index. In: 2015 IEEE 7th international workshop on managing technical debt (MTD), pp 16–24. IEEE, oct 2015. ISBN 978-1-4673-7378-4. URL: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=7332620
[63]
Dubrova E Fault-tolerant design 2013 Berlin Springer
[64]
Ian S Software engineering 2016 10 Harlow Pearson Education Limited
[65]
Goldsmith RF Discovering real business requirements for software project success 2004 Boston Artech House Inc.
[66]
Loria S (2020) TextBlob: simplified text processing. [Online]. Available: https://textblob.readthedocs.io/en/dev/
[67]
Rehurek R (2020) Gensim. [Online]. Available: https://radimrehurek.com/gensim/
[68]
Adrian H, Simon W (2020) Django. [Online]. Available: https://www.djangoproject.com/
[69]
Ferrari A, Spagnolo GO, Gnesi S (2017) PURE: a dataset of public requirements documents. In: 2017 IEEE 25th international requirements engineering conference (RE). pp 502–505. IEEE. ISBN 978-1-5386-3191-1. URL: http://ieeexplore.ieee.org/document/8049173/
[70]
ISO/IEC/IEEE (2011) IEEE/ISO/IEC 29148-2011 - ISO/IEC/IEEE International Standard-Systems and software engineering–Life cycle processes–Requirements engineering. URL: https://standards.ieee.org/standard/29148-2011.html
[71]
Florian S and Brian B A literature survey on international standards for systems requirements engineering Proc Comput Sci 2013 16 796-805
[72]
Google (2020) Google colab. [Online]. Available: https://colab.research.google.com
[73]
DiCiccio TJ and Efron B Bootstrap confidence intervals Statist Sci 1996 11 3 189-228
[74]
Fabiano P, Di ND, De RC, and De LA A large empirical assessment of the role of data balancing in machine-learning-based code smell detection J Syst Softw 2020 169
[75]
Christopher Frey H and Patil SR Identification and review of sensitivity analysis methods Risk Anal 2002 22 553-578
[76]
Wiegers KE and Joy B Software requirements 3 2013 Redmond Microsoft Press
[77]
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, and Duchesnay E Scikit-learn: machine learning in python J Mach Learn Res 2011 12 2825-2830
[78]
Pennington J, Socher R, Manning CD (2014) GloVe: global vectors for word representation. In: Empirical methods in natural language processing (EMNLP), pp 1532–1543, 2014. URL: http://www.aclweb.org/anthology/D14-1162
[79]
Goodfellow I, Bengio Y, and Courville A Deep learning 2016 Cambridge MIT Press
[80]
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, and Kaiser L Guyon I, Von Luxburg U, Bengio S, Wallach H, Fergus R, Vishwanathan S, and Garnett R Attention is all you need Advances in neural information processing systems 2017 Glasgow Curran Associates Inc
Recommendations
Natural language processing for software requirement specifications
CASCON '21: Proceedings of the 31st Annual International Conference on Computer Science and Software EngineeringSoftware Requirement Specifications (SRS) describe the functionality and expected performance for software products. It is one of the most important documents in the software development life cycle process that affects all the subsequent phases in ...
Toward a text classification system for the quality assessment of software requirements written in natural language
SOQUA '07: Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meetingRequirements Engineering (RE) is concerned with the gathering, analyzing, specifying and validating of user requirements that are documented mostly in natural language. The artifact produced by the RE process is the software requirements specification (...
Rapid requirements checks with requirements smells: two case studies
RCoSE 2014: Proceedings of the 1st International Workshop on Rapid Continuous Software EngineeringBad requirements quality can have expensive consequences during the software development lifecycle. Especially, if iterations are long and feedback comes late - the faster a problem is found, the cheaper it is to fix.
We propose to detect issues in ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 24 April 2024
Accepted: 25 March 2024
Received: 02 March 2022
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 27 Feb 2025