short-paper

Flake It 'Till You Make It: Using Automated Repair to Induce and Fix Latent Test Flakiness

Authors:

Gregory M. Kapfhammer,

Michael Hilton,

Phil McMinnAuthors Info & Claims

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Pages 11 - 12

https://doi.org/10.1145/3387940.3392177

Published: 25 September 2020 Publication History

Abstract

Since flaky tests pass or fail nondeterministically, without any code changes, they are an unreliable indicator of program quality. Developers may quarantine or delete flaky tests because it is often too time consuming to repair them. Yet, since decommissioning too many tests may ultimately degrade a test suite's effectiveness, developers may eventually want to fix them, a process that is challenging because the nondeterminism may have been introduced previously. We contend that the best time to discover and repair a flaky test is when a developer first creates and best understands it. We refer to tests that are not currently flaky, but that could become so, as having latent flakiness. We further argue that efforts to expose and repair latent flakiness are valuable in ensuring the future-reliability of the test suite, and that the testing cost is greater if latent flakiness is left to manifest itself later. Using concrete examples from a real-world program, this paper posits that automated program repair techniques will prove useful for surfacing latent flakiness.

References

[1]

J. Bell, G. Kaiser, E. Melski, and M. Dattatreya. 2015. Efficient Dependency Detection for Safe Java Test Acceleration. In ESEC/FSE 2015.

[2]

J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov. 2018. DeFlaker: Automatically Detecting Flaky Tests. In ICSE 2018.

[3]

M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli. 2019. Understanding flaky tests: the developer's perspective. In ESEC/FSE 2019.

[4]

Facebook. 2016. Python in production engineering, https://code.fb.com/production-engineering/python-in-production-engineering.

[5]

A. Gambi, J. Bell, and A. Zeller. 2018. Practical Test Dependency Detection. In ICST 2018.

[6]

Z. Gao, Y. Liang, M. B. Cohen, A. M. Memon, and Z. Wang. 2015. Making System User Interactive Tests Repeatable: When and What Should we Control?. In ICSE 2015.

[7]

A. Gyori, A. Shi, F. Hariri, and D. Marinov. 2015. Reliable Testing: Detecting State-Polluting Tests to Prevent Test Dependency. In ISSTA 2015.

Digital Library

[8]

M. Hilton, J. Bell, and D. Marinov. 2018. A Large-Scale Study of Test Coverage Evolution. In ASE 2018.

[9]

P. S. Kochhar, X. Xia, and D. Lo. 2019. Practitioners' Views on Good Software Testing Practices. In ICSE-SEIP 2019.

[10]

W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie. 2019. IDFlakies: A framework for detecting and partially classifying flaky tests. In ICST 2019.

[11]

C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. 2012. A Systematic Study of Automated Program Repair: Fixing 55 out of 105 Bugs for $8 Each. In ICSE 2012.

[12]

K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé. 2019. TBar: Revisiting Template-based Automated Program Repair. In ISSTA 2019.

[13]

P. Marinescu, P. Hosek, and C. Cadar. 2014. Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software. In ISSTA 2014.

[14]

P. McMinn. 2004. Search-Based Software Test Data Generation: A Survey. Software Testing, Verification and Reliability (2004).

[15]

A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov. 2019. iFixFlakies: A Framework for Automatically Fixing Order-dependent Flaky Tests. In ESEC/FSE 2019.

[16]

StackOverflow. 2019. Developer survey results, https://insights.stackoverflow.com/survey/2019.

[17]

H. Zhai, C. Casalnuovo, and P. Devanbu. 2019. Test Coverage in Python Programs. In MSR 2019.

[18]

S. Zhang, D. Jalali, J. Wuttke, K. Muşlu, W. Lam, M. D. Ernst, and D. Notkin. 2014. Empirically revisiting the test independence assumption. In ISSTA 2014.

Cited By

Salim Abdulla OAbdul Qader Thabit BAhmed Al-Zaidi G(2023)Software Testing Based on Research: A Road MapEdison Journal for electrical and electronics engineering10.62909/ejeee.2023.0051(24-31)Online publication date: 10-Dec-2023
https://doi.org/10.62909/ejeee.2023.005
Parry OKapfhammer GHilton MMcMinn P(2023)Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning modelsEmpirical Software Engineering10.1007/s10664-023-10307-w28:3Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1007/s10664-023-10307-w
Cordy MRwemalika RFranci APapadakis MHarman MDwyer MDamian DZeller A(2022)FlakiMeProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510194(982-994)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510194
Show More Cited By

Index Terms

Flake It 'Till You Make It: Using Automated Repair to Induce and Fix Latent Test Flakiness
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program verification

Index terms have been assigned to the content through auto-classification.

Recommendations

Make It Possible: Multilingual Sentiment Analysis without Much Prior Knowledge
WI-IAT '14: Proceedings of the 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 02

Sentiment analysis is a hard problem, while multilingual sentiment analysis is even harder due to the different expression styles in different languages. Although many methods for multilingual sentiment analysis have been developed in the open ...
make test-zesti: a symbolic execution solution for improving regression testing
ICSE '12: Proceedings of the 34th International Conference on Software Engineering

Software testing is an expensive and time consuming process, often involving the manual creation of comprehensive regression test suites. However, current testing methodologies do not take full advantage of these tests. In this paper, we present a ...
Do Automatically Generated Test Cases Make Debugging Easier? An Experimental Assessment of Debugging Effectiveness and Efficiency

Several techniques and tools have been proposed for the automatic generation of test cases. Usually, these tools are evaluated in terms of fault-revealing or coverage capability, but their impact on the manual debugging activity is not considered. The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

June 2020

831 pages

ISBN:9781450379632

DOI:10.1145/3387940

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper
Research
Refereed limited

Conference

ICSE '20

Sponsor:

SIGSOFT
KIISE

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
168
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Salim Abdulla OAbdul Qader Thabit BAhmed Al-Zaidi G(2023)Software Testing Based on Research: A Road MapEdison Journal for electrical and electronics engineering10.62909/ejeee.2023.0051(24-31)Online publication date: 10-Dec-2023
https://doi.org/10.62909/ejeee.2023.005
Parry OKapfhammer GHilton MMcMinn P(2023)Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning modelsEmpirical Software Engineering10.1007/s10664-023-10307-w28:3Online publication date: 28-Apr-2023
https://dl.acm.org/doi/10.1007/s10664-023-10307-w
Cordy MRwemalika RFranci APapadakis MHarman MDwyer MDamian DZeller A(2022)FlakiMeProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510194(982-994)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510194
Parry OKapfhammer GHilton MMcMinn P(2021)A Survey of Flaky TestsACM Transactions on Software Engineering and Methodology10.1145/347610531:1(1-74)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3476105
Lampel JJust SApel SZeller ASpinellis DGousios GChechik MDi Penta M(2021)When life gives you oranges: detecting and diagnosing intermittent job failures at MozillaProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3473931(1381-1392)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3473931
Verdecchia RCruciani EMiranda BBertolino A(2021)Know You Neighbor: Fast Static Prediction of Test FlakinessIEEE Access10.1109/ACCESS.2021.30824249(76119-76134)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3082424
Barboni MBertolino ADe Angelis G(2021)What We Talk About When We Talk About Software Test FlakinessQuality of Information and Communications Technology10.1007/978-3-030-85347-1_3(29-39)Online publication date: 25-Aug-2021
https://doi.org/10.1007/978-3-030-85347-1_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents