article

Artificial neural networks as multi-networks automated test oracle

Authors:

Seyed Reza Shahamiri,

Wan M. Wan-Kadir,

Suhaimi Ibrahim,

Siti Zaiton HashimAuthors Info & Claims

Automated Software Engineering, Volume 19, Issue 3

Pages 303 - 334

https://doi.org/10.1007/s10515-011-0094-z

Published: 01 September 2012 Publication History

Abstract

One of the important issues in software testing is to provide an automated test oracle. Test oracles are reliable sources of how the software under test must operate. In particular, they are used to evaluate the actual results produced by the software. However, in order to generate an automated test oracle, it is necessary to map the input domain to the output domain automatically. In this paper, Multi-Networks Oracles based on Artificial Neural Networks are introduced to handle the mapping automatically. They are an enhanced version of previous ANN-Based Oracles. The proposed model was evaluated by a framework provided by mutation testing and applied to test two industry-sized case studies. In particular, a mutated version of each case study was provided and injected with some faults. Then, a fault-free version of it was developed as a Golden Version to evaluate the capability of the proposed oracle finding the injected faults. Meanwhile, the quality of the proposed oracle is measured by assessing its accuracy, precision, misclassification error and recall. Furthermore, the results of the proposed oracle are compared with former ANN-based Oracles. Accuracy of the proposed oracle was up to 98.93%, and the oracle detected up to 98% of the injected faults. The results of the study show the proposed oracle has better quality and applicability than the previous model.

References

[1]

Aggarwal, K.K., Singh, Y., Kaur, A., Sangwan, O.P.: A neural net based approach to test oracle. ACM SIGSOFT Softw. Eng. Notes 29(3), 1-6 (2004).

Digital Library

[2]

Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008).

[3]

Bourlard, H.A., Morgan, N.: Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic, Norwel (1994).

[4]

Briand, L.C., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51(1), 16-30 (2009).

Digital Library

[5]

Hall, R.J.: A method and tools for large scale scenarios. Autom. Softw. Eng. 15(2), 113-148 (2008).

Digital Library

[6]

Heiat, A.: Comparison of artificial neural network and regression models for estimating software development effort. Inf. Softw. Technol. 44(15), 911-922 (2002).

[7]

Jin, H., Wang, Y., Chen, N.-W., Gou, Z.-J., Wang, S.: Artificial neural network for automatic test oracles generation. Paper presented at the Proceedings of the 2008 International Conference on Computer Science and Software Engineering, Wuhan, Hubei, 12-14 Dec. (2008).

[8]

Jorgensen, P.C.: Software Testing: A Craftsman's Approach, 2nd edn. CRC Press LLC, Boca Raton (2002).

[9]

Kessentini, M.K.M., Sahraoui, H., Boukadoum, M.: Example-based model-transformation testing. Autom. Softw. Eng. 18(2), 199-224 (2011a).

Digital Library

[10]

Kessentini, M., Sahraoui, H., Boukadoum, M.: Example-based model-transformation testing. Autom. Softw. Eng. 18(2), 199-224 (2011b).

Digital Library

[11]

Khoshgoftaar, T.M., Pandya, A.S., More, H.B.: A neural network approach for predicting software development faults. In: Pandya, A.S. (ed.) Third International Symposium on Software Reliability Engineering, Research Triangle Park, NC, pp. 83-89 (1992).

[12]

Khoshgoftaar, T.M., Szabo, R.M., Guasti, P.J.: Exploring the behavior of neural network software quality models. Softw. Eng. J. 10(3), 89-96 (1995).

[13]

Khoshgoftaar, T.M., Allen, E.B., Xu, Z.: Predicting testability of program modules using a neural network. In: Allen, E.B. (ed.) 3rd IEEE Symposium on Application-Specific Systems and Software Engineering Technology, Richardson, TX, USA, pp. 57-62 (2000).

[14]

Last, M., Freidman, M.: Black-box testing with info-fuzzy networks. In: Last, M., Kandel, A., Bunke, H. (eds.) Artificial Intelligence Methods in Software Testing. Series in Machine Perceprion and Artificial Intelligence, vol. 56, pp. 21-50. World Scientific, Singapore (2004).

[15]

Last, M., Friendman, M., Kandel, A.: Using data mining for automated software testing. Int. J. Softw. Eng. Knowl. Eng. 14(4), 369-393 (2004).

[16]

Lu, Y., Ye, M.: Oracle model based on RBF neural networks for automated software testing. Inf. Technol. J. 6(3), 469-474 (2007).

[17]

Manolache, L.I., Kourie, D.G.: Software testing using model programs. Softw. Pract. Exp. 31(13), 1211- 1236 (2001).

Digital Library

[18]

Mao, Y., Boqin, F., Li, Z., Yao, L.: Neural networks based automated test oracle for software testing. In: Proceedings of the 13th International Conference Neural Information Processing, Hong Kong, China, 2006. Lecture Notes in Computer Science, pp. 498-507. Springer, Heidelberg (2006).

[19]

McCaffrey, J.D.: Software Testing: Fundamental Principles and Essential Knowledge. BookSurge Publishing, Charleston (2009).

[20]

Memon, A.M.: Automated GUI regression testing using AI planning. In: Last, M., Kandel, A., Bunke, H. (eds.) Atificial Intelligence Methods in Software Testing. Series in Machine Perception and Artificial Intelligence, vol. 56, pp. 51-99. World Scientific, Singapore (2004).

[21]

Memon, A.M., Pollack, M.E., Soffa, M.L.: Automated test oracles for GUIs. SIGSOFT Softw. Eng. Notes 25(6), 30-39 (2000). http://doi.acm.org/10.1145/357474.355050

Digital Library

[22]

Memon, A., Nagarajan, A., Xie, Q.: Automating regression testing for evolving GUI software. J. Softw. Maint. Evol. 17(1 Spec Iss.), 27-64 (2005).

[23]

Menhaj, M.B.: Basics of Neural Networks, 1st edn. Amirkabir Technology University, Tehran (2001).

[24]

Michael, C., McGraw, G.: Automated software test data generation for complex programs. Paper presented at the Proceedings of the 13th IEEE International Conference on Automated Software Engineering, Honolulu, Hawaii, October (1998).

[25]

Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27(12), 1085-1110 (2001).

Digital Library

[26]

Myers, G.J.: The Art of Software Testing, 2nd edn. Wiley, New York (2004).

[27]

Ntafos, S.C.: On comparisons of random, partition, and proportional partition testing. IEEE Trans. Softw. Eng. 27(10), 949-960 (2001).

Digital Library

[28]

Patton, R.: Software Testing, 2nd edn. Sams Publishing, Indianapolis (2005).

[29]

Peters, D., Parnas, D.L.: Generating a test oracle from program documentation. In: Proceedings of the 1994 ACM SIGSOFT International Symposium on Software Testing and Analysis, Seattle, WA, USA, p. 58. ACM, New York (1994).

[30]

Peters, D.K., Parnas, D.L.: Using test oracles generated from program documentation. IEEE Trans. Softw. Eng. 24(3), 161-173 (1998).

Digital Library

[31]

Pfleeger, S.L., Hatton, L.: Investigating the influence of formal methods. Computer 30(2), 33-43 (1997).

Digital Library

[32]

Ran, L., Dyreson, C., Andrews, A.A., Bryce, R., Mallery, C.: Building test cases and oracles to automate the testing of web database applications. Inf. Softw. Technol. 51(2), 460-477 (2009).

Digital Library

[33]

Richardson, D.J., Aha, S.L., O'Malley, T.O.: Specification-based test oracles for reactive systems. In: Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105-118. ACM, New York (1992).

Digital Library

[34]

Saraph, P., Last, M., Kandell, A.: Test case generation and reduction by automated input-output analysis. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Washington DC, United States, pp. 768-773. IEEE, New York (2003).

[35]

Schalkoff, R.J.: Artificial Neural Networks. McGraw-Hill, New York (1997).

[36]

Schroeder, P.J., Korel, B.: Black-box test reduction using input-output analysis. SIGSOFT Softw. Eng. Notes 25(5), 173-177 (2000). http://doi.acm.org/10.1145/347636.349042

[37]

Schroeder, P.J., Faherty, P., Korel, B.: Generating expected results for automated black-box testing. In: Proceedings 17th IEEE International Conference on Automated Software Engineering ASE, Edinburgh, UK, pp. 139-148 (2002).

[38]

Shahamiri, S.R., Wan, M.N.W.K.: Intelligent and automated software testingmethods classification. In: 4th Postgraduate Annual Research Seminar (PARS'08), UTM Skudai, Malaysia, 1-3 July 2008 (2008).

[39]

Shahamiri, S.R., Kadir, W.M.N.W., Mohd-Hashim, S.Z.: A comparative study on automated software test oracle methods. In: Proceedings of the 2009 Fourth International Conference on Software Engineering Advances, Porto, Portugal, September 20-25, 2009, pp. 140-145. IEEE Computer Society, Los Alamitos (2009).

Digital Library

[40]

Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S.: An automated oracle approach to test decision-making structures. In: Proceedings of the 2010 3rd IEEE International Conference on Computer Science and Information Technology, Chengdu, China, July 9-11, 2010, pp. 30-34. IEEE, New York (2010a).

[41]

Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S.: A single-network ANN-based oracle to verify logical software modules. In: Proceedings of the 2010 2nd International Conference on Software Technology and Engineering, Puerto Rico, USA, October 3-5, 2010, pp. 272-276. IEEE, New York (2010b).

[42]

Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S., Hashim, S.Z.M.: An automated framework for software test oracle. Inf. Softw. Technol. 53(7), 774-788 (2011).

Digital Library

[43]

Sofokleous, A.A., Andreou, A.S.: Automatic evolutionary test data generation for dynamic software testing. J. Syst. Softw. 81(11), 1883-1898 (2008).

Digital Library

[44]

Spillner, A., Linz, T., Schaefer, H.: Software Testing Foundations, 2nd edn. Rocky Nook Inc, Santa Barbara (2007).

[45]

Stocks, P., Carrington, D.: A framework for specification-based testing. IEEE Trans. Softw. Eng. 22(11), 777-793 (1996).

Digital Library

[46]

Su, Y., Huang, C.: Neural-network-based approaches for software reliability estimation using dynamic weighted combinational models. J. Syst. Softw. 80(4), 606-615 (2007).

Digital Library

[47]

Sun, Y., Bosch, L.T., Boves, L.: Hybrid HMM/BLSTM-RNN for robust speech recognition. Paper presented at the Proceedings of the 13th International Conference on Text, Speech and Dialogue, Brno, Czech Republic (2010).

[48]

Vanmali, M., Last, M., Kandel, A.: Using a neural network in the software testing process. Int. J. Intell. Syst. 17(1), 45-62 (2002).

[49]

Whittaker, J.A.: What is software testing? And why is it so hard? IEEE Softw. 17(1), 70-79 (2000).

Digital Library

[50]

Woodward, M.R.: Mutation testing--its origin and evolution. Inf. Softw. Technol. 35(3), 163-169 (1993).

[51]

Woodward, M.R., Hennell, M.A.: On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC. Inf. Softw. Technol. 48(7), 433-440 (2006).

Cited By

Wang XAli SArrieta AArcaini PArratibel Md'Amorim M(2024)Application of Quantum Extreme Learning Machines for QoS Prediction of Elevators’ Software in an Industrial ContextCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663859(399-410)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663859
Ren XYe XLin YXing ZLi SLyu MChandra SBlincoe KTonella P(2023)API-Knowledge Aware Search-Based Software Testing: Where, What, and HowProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616269(1320-1332)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616269
Geethal CBöhme MPham V(2023)Human-in-the-Loop Automatic Program RepairIEEE Transactions on Software Engineering10.1109/TSE.2023.330505249:10(4526-4549)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3305052
Show More Cited By

Artificial neural networks as multi-networks automated test oracle
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

An automated framework for software test oracle

Context: One of the important issues of software testing is to provide an automated test oracle. Test oracles are reliable sources of how the software under test must operate. In particular, they are used to evaluate the actual results that produced by ...
A Comparative Study on Automated Software Test Oracle Methods
ICSEA '09: Proceedings of the 2009 Fourth International Conference on Software Engineering Advances

Software testing has been used to find software faults in order to improve its quality. To verify the software behavior, testers require test oracle. Test oracle is a reliable source of expected software behavior that provides outputs for any input ...
Applying Artificial Neural Networks to Logic Built-in Self-test: Improving Test Point Insertion
Abstract
This study applies artificial neural networks (ANNs) to increase stuck-at and delay fault coverage of logic built-in self-test (LBIST) through test point insertion (TPI). Increasing TPI quality is essential for modern logic circuits, but the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Automated Software Engineering

Automated Software Engineering Volume 19, Issue 3

September 2012

146 pages

ISSN:0928-8910

Issue’s Table of Contents

Copyright © Copyright © 2012 Springer Science+Business Media, LLC.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 September 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang XAli SArrieta AArcaini PArratibel Md'Amorim M(2024)Application of Quantum Extreme Learning Machines for QoS Prediction of Elevators’ Software in an Industrial ContextCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663859(399-410)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663859
Ren XYe XLin YXing ZLi SLyu MChandra SBlincoe KTonella P(2023)API-Knowledge Aware Search-Based Software Testing: Where, What, and HowProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616269(1320-1332)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616269
Geethal CBöhme MPham V(2023)Human-in-the-Loop Automatic Program RepairIEEE Transactions on Software Engineering10.1109/TSE.2023.330505249:10(4526-4549)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3305052
Ibrahimzada AVarli YTekinoglu DJabbarvand RRoychoudhury ACadar CKim M(2022)Perfect is the enemy of test oracleProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549086(70-81)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549086
Fontes AGay GJahangirova GTerragni V(2021)Using machine learning to generate test oracles: a systematic literature reviewProceedings of the 1st International Workshop on Test Oracles10.1145/3472675.3473974(1-10)Online publication date: 24-Aug-2021
https://dl.acm.org/doi/10.1145/3472675.3473974
Geethal CGrundy J(2021)Training automated test oracles to identify semantic bugsProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678886(1051-1055)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678886
Molina FGrundy JLe Goues CLo D(2020)Applying learning techniques to oracle synthesisProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3415287(1153-1157)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3415287
Kamaraj KArvind CSrihari K(2020)A weight optimized artificial neural network for automated software test oracleSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-020-05197-924:17(13501-13511)Online publication date: 1-Sep-2020
https://dl.acm.org/doi/10.1007/s00500-020-05197-9
Jahan HFeng ZMahmud SDong P(2019)Version specific test case prioritization approach based on artificial neural networkJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-18199836:6(6181-6194)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.3233/JIFS-181998
Molina FDegiovanni RPonzio PRegis GAguirre NFrias MAtlee JBultan TWhittle J(2019)Training binary classifiers as data structure invariantsProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00084(759-770)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00084
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents