Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

A semantic approach for automated test oracle generation

Published: 01 April 2016 Publication History

Abstract

This paper presents the design, implementation, and applications of a software testing tool, TAO, which allows users to specify and generate test cases and oracles in a declarative way. Extended from its previous grammar-based test generation tool, TAO provides a declarative notation for defining denotational semantics on each productive grammar rule, such that when a test case is generated, its expected semantics will be evaluated automatically as well, serving as its test oracle. TAO further provides a simple tagging mechanism to embed oracles into test cases for bridging the automation between test case generation and software testing. Two practical case studies are used to illustrate how automated oracle generation can be effectively integrated with grammar-based test generation in different testing scenarios: locating fault-inducing input patterns on Java applications; and Selenium-based automated web testing. HighlightsTAO provides a declarative framework for automated test and oracle generation.TAO integrates test and oracle generation as a whole to promote better automated software testing.TAO is the first attempt to apply the methodology of denotational semantics in test and oracle generation.

References

[1]
Guo H-F, Cao L, Song Y, Qiu Z. Automated test oracle generation via denotational semantics. In: 14th International conference on quality software; 2014. p. 139-44.
[2]
B. Hailpern, P. Santhanam, Software debugging, testing, and verification, IBM Syst J, 41 (2002) 4-12.
[3]
J. Offutt, S. Liu, A. Abdurazik, P. Ammann, Generating test data from state based specifications, J Softw Test Verif Reliab, 13 (2003) 25-53.
[4]
S. Anand, E.K. Burke, T.Y. Chen, J. Clark, M.B. Cohen, W. Grieskamp, An orchestrated survey of methodologies for automated software test case generation, J Syst Softw, 86 (2013) 1978-2001.
[5]
Autotest Developers. Fully automated testing under linux. {http://autotest.github.io/}last visited on October 2015.
[6]
Yahoo Developer Network, Yeti. {http://autotest.github.io/}last visited on 2013.
[7]
A. Zeller, R. Hildebrandt, Simplifying and isolating failure-inducing input, IEEE Trans Softw Eng, 28 (2002) 183-200.
[8]
V. Debroy, W.E. Wong, A consensus-based strategy to improve the quality of fault localization, Softw: Pract Exp, 43 (2013) 989-1011.
[9]
Cleve H, Zeller A. Locating causes of program failures. In: Proceedings of the 27th international conference on software engineering, ICSE '05, ACM, New York, NY, USA; 2005. p. 342-51.
[10]
Yang X, Chen Y, Eide E, Regehr J. Finding and understanding bugs in c compilers. In: The 32nd Conference on programming language design and implementation. New York, NY, USA: ACM; 2011. p. 283-94.
[11]
Baresi L, Young M. Test oracles. Technical report CIS-TR-01-02. University of Oregon, Department of Computer and Information Science, {http://www.cs.uoregon.edu/~michal/pubs/oracles.html}; August 2001.
[12]
Shahamiri SR, Kadir WMNW, Mohd-Hashim SZ. A comparative study on automated software test oracle methods. In: the 2009 Fourth international conference on software engineering advances, 2009. p. 140-5.
[13]
Staats M, Whalen MW, Heimdahl MP. Better testing through oracle selection. In: Proceedings of the 33rd international conference on software engineering (NIER Track). New York, NY, USA: ACM; 2011. p. 892-5.
[14]
E. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo, The oracle problem in software testing, IEEE Trans Softw Eng, 41 (2015) 507-525.
[15]
Staats M, Whalen MW, Heimdahl MP. Programs, tests, and oracles: the foundations of testing revisited. In: the 33rd International conference on software engineering; 2011. p. 391-400.
[16]
Li N, Offutt J. An empirical analysis of test oracle strategies for model-based testing. In: Seventh IEEE international conference on software testing, verification and validation, 2014. p. 363-72.
[17]
Q. Xie, AM. Memon, Designing and comparing automated test oracles for gui-based software applications, ACM Trans Softw Eng Methodol, 16 (2007).
[18]
Peters D, Parnas DL. Generating a test oracle from program documentation: Work in progress. In: The 1994 ACM SIGSOFT international symposium on software testing and analysis; 1994. p. 58-65.
[19]
Day JD, Gannon JD. A test oracle based on formal specifications. In: Proceedings of the second conference on software development tools, techniques, and alternatives. Los Alamitos, CA, USA: IEEE Computer Society Press; 1985. p. 126-30.
[20]
Schroeder PJ, Faherty P, Korel B. Generating expected results for automated black-box testing. In: The 17th IEEE international conference on automated software engineering, 2002. p. 139-.
[21]
Richardson DJ, Aha SL, O'Malley TO. Specification-based test oracles for reactive systems. In: The 14th international conference on software engineering. New York, NY, USA: ACM; 1992. p. 105-18.
[22]
Dillon LK, Ramakrishna YS. Generating oracles from your favorite temporal logic specifications. In: Proceedings of the 4th ACM SIGSOFT symposium on foundations of software engineering, SIGSOFT '96. New York, NY, USA: ACM; 1996. p. 106-17. http://dx.doi.org/10.1145/239098.239116.
[23]
Callahan JR, Easterbrook SM, Montgomery TL. Generating test oracles via model checking. Technical report. NASNWVU Software Research Lab; 1998.
[24]
M.S. Feather, B. Smith, Automatic generation of test oracles-from pilot studies to application, Automat Softw Eng, 8 (2001) 31-61.
[25]
Shrestha K, Rutherford M. An empirical evaluation of assertions as oracles. In: IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST), 2011. p. 110-9.
[26]
Di Nardo D, Alshahwan N, Briand L, Fourneret E, Nakic-Alfirevic T, Masquelier V. Model based test validation and oracles for data acquisition systems. In: 2013 IEEE/ACM 28th international conference on automated software engineering (ASE); 2013. p. 540-50.
[27]
L. Padgham, J. Thangarajah, Z. Zhang, T. Miller, Model-based test oracle generation for automated unit testing of agent systems, IEEE Trans Softw Eng, 39 (2013) 1230-1244.
[28]
E.J. Weyuker, On testing non-testable programs, Comput J, 25 (1982) 465-470.
[29]
D. Hoffman, Heuristic test oracles, Softw Test Qual Eng Mag (1999) 29-32.
[30]
Mayer J, Guderlei R. Test oracles using statistical methods. In: The first international workshop on software quality; 2004. p. 179-89.
[31]
M. Last, M. Friedman, A. Kandel, Using data mining for automated software testing, Int J Softw Eng Knowl Eng, 14 (2004) 369-393.
[32]
Xie T. Augmenting automatically generated unit-test suites with regression oracle checking. In: The 20th European conference on object-oriented programming; 2006. p. 380-403.
[33]
Scott D, Strachey C. Toward a mathematical semantics for computer languages. Oxford programming research group technical monograph, PRG-6; 1971.
[34]
R. Milne, C. Strachey, A theory of programming language semantics, Chapman and Hall, London, 1976.
[35]
D.A. Schmidt, Denotational semantics: a methodology for language development, Wm. C. Brown Publishers, Hamburg, Iowa, USA, 1986.
[36]
H.-F. Guo, Z. Qiu, A dynamic stochastic model for automatic grammar-based test generation, Softw: Pract Exp, 45 (2015) 1457-1595.
[37]
Lab UL. Tao online. URL {http://laser.ist.unomaha.edu/tao_home/}; 2014.
[38]
J. de Bakker, E. de Vink, Denotational models for programming languages, Topol Appl, 85 (1998) 35-52.
[39]
Gupta G. Horn logic denotations and their applications. In: The logic programming paradigm. Berlin, Heidelberg: Springer; 1999. p. 127-59.
[40]
Majumdar R, Xu R-G. Directed test generation using symbolic grammars. In: The 22nd IEEE/ACM international conference on automated software engineering; 2007. p. 134-43.
[41]
P.M. Maurer, Generating test data with enhanced context-free grammars, IEEE Softw, 7 (1990) 50-55.
[42]
W. McKeeman, Differential testing for software, Digit Techn J Digit Equip Corp, 10 (1998) 100-107.
[43]
Sirer EG, Bershad BN. Using production grammars in software testing. In: The 2nd conference on domain-specific languages; 1999. p. 1-13.
[44]
Lämmel R. Grammar testing. In: Proceedings of the fundamental approaches to software engineering (FASE) 2001. Berlin, Heidelberg: Springer-Verlag; 2001. p. 201-16.
[45]
Lämmel R, Schulte W. Controllable combinatorial coverage in grammar-based testing. In: International conference on testing of communicating systems; 2006. p. 19-38.
[46]
D.M. Hoffman, D. Ly-Gagnon, P. Strooper, H.-Y. Wang, Grammar-based test generation with yougen, Softw: Pract Exp, 41 (2011) 427-447.
[47]
H.-F. Guo, G. Gupta, Simplifying dynamic programming via mode-directed tabling, Softw: Pract Exp, 38 (2008) 75-94.
[48]
Misherghi G, Su Z. Hdd: hierarchical delta debugging. In: The 28th international conference on software engineering. New York, NY, USA: ACM; 2006. p. 142-51.
[49]
Zeller A. Isolating cause-effect chains from computer programs. In: Proceedings of the 10th ACM SIGSOFT symposium on foundations of software engineering. New York, NY, USA: ACM; 2002. p. 1-10.
[50]
Selenium. Selenium browser automation, {http://www.seleniumhq.org/}accessed 30.08.12.
[51]
M. Mernik, M. ¿repinšek, T. Kosar, D. Rebernak, V. ¿umer, Grammar-based systems, Informatica (Slovenia), 28 (2004) 245-255.
[52]
Godefroid P, Kiezun A, Levin MY. Grammar-based whitebox fuzzing. In: The 29th ACM SIGPLAN conference on programming language design and implementation; New York, NY, USA: ACM. 2008. p. 206-15.
[53]
H. Wu, J. Gray, M. Mernik, Grammar-driven generation of domain-specific language debuggers, Softw: Pract Exp, 38 (2008) 1073-1103.
[54]
M. Mernik, J. Heering, A.M. Sloane, When and how to develop domain-specific languages, ACM Comput Surv, 37 (2005) 316-344.
[55]
Pretschner A, Prenninger W, Wagner S, Kühnel C, Baumgartner M, Sostawa B, Zölch R, Stauner T. One evaluation of model-based testing and its automation. In: The 27th international conference on software engineering. New York, NY, USA: ACM; 2005. p. 392-401.
[56]
C. Cadar, K. Sen, Symbolic execution for software testing, Commun ACM, 56 (2013) 82-90.
[57]
Li N. The structured test automation language framework. {http://cs.gmu.edu/~nli1/stale/ last seen May 2013.
[58]
P. Ammann, J. Offutt, Introduction to software testing, Cambridge University Press, New York, NY, USA, 2008.
[59]
Groce A, Zhang C, Eide E, Chen Y, Regehr J. Swarm testing. In: Proceedings of the 2012 international symposium on software testing and analysis. New York, NY, USA: ACM; 2012. p. 78-88.

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computer Languages, Systems and Structures
Computer Languages, Systems and Structures  Volume 45, Issue C
April 2016
219 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 April 2016

Author Tags

  1. Denotational semantics
  2. Software testing
  3. Test case generation
  4. Test oracle

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 24 Nov 2024

Other Metrics

Citations

View Options

View options

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media