research-article

Search-based software testing and test data generation for a dynamic programming language

Authors:

Stefan Mairhofer,

Richard TorkarAuthors Info & Claims

GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computation

Pages 1859 - 1866

https://doi.org/10.1145/2001576.2001826

Published: 12 July 2011 Publication History

Abstract

Manually creating test cases is time consuming and error prone. Search-based software testing can help automate this process and thus reduce time and effort and increase quality by automatically generating relevant test cases. Previous research has mainly focused on static programming languages and simple test data inputs such as numbers. This is not practical for dynamic programming languages that are increasingly used by software developers. Here we present an approach for search-based software testing for dynamically typed programming languages that can generate test scenarios and both simple and more complex test data. The approach is implemented as a tool, RuTeG, in and for the dynamic programming language Ruby. It combines an evolutionary search for test cases that give structural code coverage with a learning component to restrict the space of possible types of inputs. The latter is called for in dynamic languages since we cannot always know statically which types of objects are valid inputs. Experiments on 14 cases taken from real-world Ruby projects show that RuTeG achieves full or higher statement coverage on more cases and does so faster than randomly generated test cases.

References

[1]

M. Alshraideh and L. Bottaci. Search-based software test data generation for string data using program-specific search operators. Software Testing, Verification & Reliability, 16(3):175--203, 2006.

Digital Library

[2]

A. Arcuri and X. Yao. Search based software testing of object-oriented containers. Information Sciences, 178(15):3075--3095, 2008.

Digital Library

[3]

A. Bertolino. Software testing research: Achievements, challenges, dreams. In 2007 Future of Software Engineering, FOSE '07, pages 85--103, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[4]

M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 69:35--45, December 2007.

Digital Library

[5]

R. Feldt. Biomimetic Software Engineering Techniques for Dependability. PhD thesis, "Dept. of Computer Engineering, Chalmers University of Technology", 2002.

[6]

M. Harman and P. McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, pages 226--247, 2009.

Digital Library

[7]

B. Jones, H.-H. Sthamer, and D. Eyres. Automatic structural testing using genetic algorithms. The Software Engineering Journal, 11:299--306, 1996.

[8]

K. Lakhotia, M. Harman, and P. McMinn. Handling dynamic data structures in search based testing. In Proceedings of the 10th annual conference on Genetic and evolutionary computation, GECCO '08, pages 1759--1766, New York, NY, USA, 2008. ACM.

Digital Library

[9]

R. Lefticaru and F. Ipate. Automatic state-based test generation using genetic algorithms. In Proceedings of the Ninth International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pages 188--195, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[10]

D. Marinov. Automatic testing of software with structurally complex inputs. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 2005. AAI0807757.

Digital Library

[11]

P. McMinn. Search-based software test data generation: A survey. Software Testing, Verification & Reliability, 14:105--156, June 2004.

Digital Library

[12]

T. Mikkonen and A. Taivalsaari. Using JavaScript as a real programming language. Sun Microsystems Laboratories Technical Report TR-2007, 168, 2007.

Digital Library

[13]

C. Pacheco, S. K. Lahiri, and T. Ball. Finding errors in .NET with feedback-directed random testing. In Proceedings of the 2008 international symposium on Software testing and analysis, ISSTA '08, pages 87--96, New York, NY, USA, 2008. ACM.

Digital Library

[14]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proceedings of the 29th international conference on Software Engineering, ICSE '07, pages 75--84, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[15]

P. Tonella. Evolutionary testing of classes. SIGSOFT Software Engineering Notes, 29(4):119--128, 2004.

Digital Library

[16]

T. Vos, A. Baars, F. Lindlar, P. Kruse, A. Windisch, and J. Wegener. Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool. In 2010 Third International Conference on Software Testing, Verification and Validation, pages 175--184. IEEE, 2010.

Digital Library

[17]

S. Wappler and I. Schieferdecker. Improving evolutionary class testing in the presence of non-public methods. In Proceedings of the 22nd IEEE/ACM international conference on Automated software engineering, ASE '07, pages 381--384, New York, NY, USA, 2007. ACM.

Digital Library

[18]

S. Wappler and J. Wegener. Evolutionary unit testing of object-oriented software using a hybrid evolutionary algorithm. In CEC'06: Proceedings of the 2006 IEEE Congress on Evolutionary Computation, pages 851--858. IEEE, 2006.

Digital Library

[19]

S. Wappler and J. Wegener. Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In Proceedings of the 8th annual conference on Genetic and evolutionary computation, GECCO '06, pages 1925--1932, New York, NY, USA, 2006. ACM.

Digital Library

[20]

G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, and Z. Su. Dynamic test input generation for web applications. In ISSTA '08: Proceedings of the 2008 International Symposium on Software Testing and Analysis, pages 249--260, New York, NY, USA, 2008. ACM.

Digital Library

[21]

A. Watkins and E. M. Hufnagel. Evolutionary test data generation: A comparison of fitness functions. Software Practice and Experience, 36(1):95--116, 2006.

Digital Library

[22]

J. Wegener, A. Baresel, and H. Sthamer. Evolutionary test environment for automatic structural testing. Information and Software Technology, 43(14):841--854, 2001.

[23]

R. Zhao and Q. Li. Automatic test generation for dynamic data structures. In SERA '07: Proceedings of the 5th ACIS International Conference on Software Engineering Research, Management & Applications, pages 545--549, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

Cited By

Boukhlif MHanine MKharmoum N(2023)A Decade of Intelligent Software Testing Research: A Bibliometric AnalysisElectronics10.3390/electronics1209210912:9(2109)Online publication date: 5-May-2023
https://doi.org/10.3390/electronics12092109
Verma AChoudhary ATiwari S(2023)Software Test Case Generation Tools and Techniques: A ReviewInternational Journal of Mathematical, Engineering and Management Sciences10.33889/IJMEMS.2023.8.2.0188:2(293-315)Online publication date: 1-Apr-2023
https://doi.org/10.33889/IJMEMS.2023.8.2.018
Lukasczyk SKroiß FFraser G(2023)An empirical study of automated unit test generation for PythonEmpirical Software Engineering10.1007/s10664-022-10248-w28:2Online publication date: 31-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10248-w
Show More Cited By

Index Terms

Search-based software testing and test data generation for a dynamic programming language
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Whole Test Suite Generation

Not all bugs lead to program crashes, and not always is there a formal specification to check the correctness of a software test's outcome. A common scenario in software testing is therefore that test data are generated, and a tester manually adds test ...
Fault-based test suite prioritization for specification-based testing

Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in ...
A test generation solution to automate software testing
AST '08: Proceedings of the 3rd international workshop on Automation of software test

This paper describes the LEIRIOS Smart Testing™ approach to the functional validation of a subpart of the StarUML case study. This model-based testing process for test automation is shown from start to end: from functional requirements to the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computation

July 2011

2140 pages

ISBN:9781450305570

DOI:10.1145/2001576

Editor:
Natalio Krasnogor
University of Nottingham, UK
,
General Chair:
Pier Luca Lanzi
Politecnico di Milano, Italy

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '11

Sponsor:

SIGEVO

GECCO '11: Genetic and Evolutionary Computation Conference

July 12 - 16, 2011

Dublin, Ireland

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
454
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)1

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Boukhlif MHanine MKharmoum N(2023)A Decade of Intelligent Software Testing Research: A Bibliometric AnalysisElectronics10.3390/electronics1209210912:9(2109)Online publication date: 5-May-2023
https://doi.org/10.3390/electronics12092109
Verma AChoudhary ATiwari S(2023)Software Test Case Generation Tools and Techniques: A ReviewInternational Journal of Mathematical, Engineering and Management Sciences10.33889/IJMEMS.2023.8.2.0188:2(293-315)Online publication date: 1-Apr-2023
https://doi.org/10.33889/IJMEMS.2023.8.2.018
Lukasczyk SKroiß FFraser G(2023)An empirical study of automated unit test generation for PythonEmpirical Software Engineering10.1007/s10664-022-10248-w28:2Online publication date: 31-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10248-w
Abdi MRocha HDemeyer SBergel A(2022)Small-Amp: Test amplification in a dynamically typed languageEmpirical Software Engineering10.1007/s10664-022-10169-827:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10169-8
Ferreira Vilela RChoma Neto JSantiago Costa Pinto VLopes de Souza Pdo Rocio Senger de Souza S(2022)Bio‐inspired optimization to support the test data generation of concurrent softwareConcurrency and Computation: Practice and Experience10.1002/cpe.748935:2Online publication date: 10-Nov-2022
https://doi.org/10.1002/cpe.7489
Vilela RPinto VColanzi TSouza S(2019)Bio-Inspired Optimization of Test Data Generation for Concurrent SoftwareSearch-Based Software Engineering10.1007/978-3-030-27455-9_9(121-136)Online publication date: 3-Aug-2019
https://doi.org/10.1007/978-3-030-27455-9_9
Elyasov APrasetya IHage J(2018)Search-Based Test Data Generation for JavaScript Functions that Interact with the DOM2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2018.00020(88-99)Online publication date: Oct-2018
https://doi.org/10.1109/ISSRE.2018.00020
Poulding SFeldt R(2017)Automated Random Testing in Multiple Dispatch Languages2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2017.37(333-344)Online publication date: Mar-2017
https://doi.org/10.1109/ICST.2017.37
Wibowo JHendradjaya BWidyani Y(2015)Unit test code generator for lua programming language2015 International Conference on Data and Software Engineering (ICoDSE)10.1109/ICODSE.2015.7437005(241-245)Online publication date: Nov-2015
https://doi.org/10.1109/ICODSE.2015.7437005
Aleti AGrunske L(2015)Test data generation with a Kalman filter-based adaptive genetic algorithmJournal of Systems and Software10.1016/j.jss.2014.11.035103:C(343-352)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1016/j.jss.2014.11.035
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents