research-article

Experiences of testing bioinformatics programs for detecting subtle faults

Authors:

Anders Lundgren,

Upulee KanewalaAuthors Info & Claims

SE4Science '16: Proceedings of the International Workshop on Software Engineering for Science

Pages 16 - 22

https://doi.org/10.1145/2897676.2897678

Published: 14 May 2016 Publication History

Abstract

One of the biggest challenges for conducting automated systematic testing on scientific programs is the oracle problem. This challenge is especially prevalent in the field of bioinformatics due to the inherent complexity of these programs. In this paper, we explore two approaches: pseudo-oracles and metamorphic testing for conducting automated systematic testing of bioinformatics programs. We use BBMap: an open source genome alignment tool as the system under test to evaluate the effectiveness of the above two approaches for identifying subtle faults.

Our results show that the pseudo-oracle testing does not provide a consistent basis for fault-detection. This is mainly due to not having a consistent rate of agreement between the outputs of the system under test and the pseudo-oracle. On the other hand, metamorphic testing detects a majority of subtle faults, suggesting that it is more effective in identifying these types of faults.

References

[1]

S. Altschul, B. Demchak, R. Durbin, R. Gentleman, M. Krzywinski, H. Li, A. Nekrutenko, J. Robinson, W. Rasband, J. Taylor, and C. Trapnell. The anatomy of successful computational biology software. Nature Biotechnology, 31(10):894--897, 2010.

[2]

E. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. Software Engineering, IEEE Transactions on, 41(5):507--525, May 2015.

Digital Library

[3]

T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45(1):1--9, 2003.

Digital Library

[4]

P. A. Fujita, B. Rhead, A. S. Zweig, A. S. Hinrichs, D. Karolchik, M. S. Cline, M. Goldman, G. P. Barber, H. Clawson, A. Coelho, et al. The ucsc genome browser database: update 2011. Nucleic acids research, page gkq963, 2010.

[5]

E. Giannoulatou, S.-H. Park, D. T. Humphreys, and J. W. Ho. Verification and validation of bioinformatics software without a gold standard: a case study of bwa and bowtie. BMC bioinformatics, 15(Suppl 16):S15, 2014.

[6]

Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37:649--678, 2011.

Digital Library

[7]

L. N. Joppa, G. McInerny, R. Harper, L. Salido, K. Takeda, K. O'Hara, D. Gavaghan, and S. Emmott. Troubling trends in scientific software use. Science, 340(6134):814--815, 2013.

[8]

U. Kanewala. Techniques for automatic detection of metamorphic relations. In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, pages 237--238, March 2014.

Digital Library

[9]

U. Kanewala and J. M. Bieman. Techniques for testing scientific programs without an oracle. In Proc. SE-CSE, pages 48--57. IEEE, 2013.

Digital Library

[10]

U. Kanewala and J. M. Bieman. Testing scientific software: A systematic literature review. Information and Software Technology, 56(10):1219--1232, 2014.

Digital Library

[11]

U. Kanewala, J. M. Bieman, and A. Ben-Hur. Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Software testing, verification and reliability, 2015. In press.

[12]

W. J. Kent. BLAT--the BLAST-like alignment tool. Genome Research, 12(4):656--664, Apr. 2002.

[13]

J. Offut and Y. S. Ma. Description of method-level mutation operators for java. https://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf. Accessed: Dec, 2015.

[14]

M. Papadakis and N. Malevris. An empirical evaluation of the first and second order mutation testing strategies. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 90--99, April 2010.

Digital Library

[15]

R. Sanders and D. Kelly. The challenge of testing scientific software. In In Proc. of the Conference for the Association for Software Testing (CAST), pages 30--36, Toronto, July 2008.

[16]

B. Smith and L. Williams. An empirical evaluation of the mujava mutation operators. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, 2007. TAICPART-MUTATION 2007, pages 193--202, Sept 2007.

Digital Library

[17]

A. Sung, B. Choi, W. E. Wong, and V. Debroy. Mutant generation for embedded systems using kernel-based software and hardware fault simulation. Information and Software Technology, 53(10):1153--1164, 2011.

[18]

M. Umarji, C. Seaman, A. Koru, and H. Liu. Software engineering education for bioinformatics. In Software Engineering Education and Training, 2009. CSEET '09. 22nd Conference on, pages 216--223, Feb 2009.

Digital Library

[19]

E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--470, 1982.

[20]

L. L. Wu and G. G. Kaiser. Constructing subtle concurrency bugs using synchronization-centric second-order mutation operators. Columbia University Academic Commons, 2011.

Cited By

Peng ZKanewala UNiu NLanubile F(2021)Contextual Understanding and Improvement of Metamorphic Testing in Scientific Software DevelopmentProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484188(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484188
Gerten MLathrop JCohen MKlinge TGrundy JLe Goues CLo D(2020)ChemTestProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416638(548-560)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416638
Shinn M(2020)Refinement Type Contracts for Verification of Scientific Investigative SoftwareVerified Software. Theories, Tools, and Experiments10.1007/978-3-030-41600-3_10(143-160)Online publication date: 14-Mar-2020
https://doi.org/10.1007/978-3-030-41600-3_10
Show More Cited By

Recommendations

Conformance testing of boolean programs with multiple faults
FMOODS'12/FORTE'12: Proceedings of the 14th joint IFIP WG 6.1 international conference and Proceedings of the 32nd IFIP WG 6.1 international conference on Formal Techniques for Distributed Systems

Conformance testing is the problem of constructing a complete test suite of inputs based on a specification S such that any implementation I (of size less than a given bound) that is not equivalent to S gives a different output on the test suite than S. ...
Detecting associative shift faults in predicate testing

Fault-based predicate testing is a promising strategy for generating effective software test data. The test target is faults that may appear in predicates found in the specification or implementation of a computer program. A hierarchy of fault classes ...
Pseudo-random pattern testing of bridging faults
ICCD '97: Proceedings of the 1997 International Conference on Computer Design (ICCD '97)

While previous research has focused on deterministic testing of bridging faults, this paper studies pseudo-random testing of bridging faults and describes a means for achieving high fault coverage in a built-in self-test (BIST) environment. Bridging ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SE4Science '16: Proceedings of the International Workshop on Software Engineering for Science

May 2016

41 pages

ISBN:9781450341677

DOI:10.1145/2897676

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
146
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Peng ZKanewala UNiu NLanubile F(2021)Contextual Understanding and Improvement of Metamorphic Testing in Scientific Software DevelopmentProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484188(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484188
Gerten MLathrop JCohen MKlinge TGrundy JLe Goues CLo D(2020)ChemTestProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416638(548-560)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416638
Shinn M(2020)Refinement Type Contracts for Verification of Scientific Investigative SoftwareVerified Software. Theories, Tools, and Experiments10.1007/978-3-030-41600-3_10(143-160)Online publication date: 14-Mar-2020
https://doi.org/10.1007/978-3-030-41600-3_10
Pourreza Shahri MSrinivasan MReynolds GBimczok DKahanda IKanewala U(2019)Metamorphic Testing for Quality Assurance of Protein Function Prediction Tools2019 IEEE International Conference On Artificial Intelligence Testing (AITest)10.1109/AITest.2019.00017(140-148)Online publication date: Apr-2019
https://doi.org/10.1109/AITest.2019.00017
Cashman MCohen MRanjan PCottingham RHuchard MKästner CFraser G(2018)Navigating the maze: the impact of configurability in bioinformatics softwareProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering10.1145/3238147.3240466(757-767)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3238147.3240466
Srinivasan MShahri MKahanda IKanewala UXie XPullum LPoon P(2018)Quality assurance of bioinformatics softwareProceedings of the 3rd International Workshop on Metamorphic Testing10.1145/3193977.3193981(26-33)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3193977.3193981
Storer T(2017)Bridging the ChasmACM Computing Surveys10.1145/308422550:4(1-32)Online publication date: 25-Aug-2017
https://dl.acm.org/doi/10.1145/3084225

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents