JUniVerse: large-scale JUnit-test analysis in the wild
Pages 1768 - 1775
Abstract
Current approaches for analyzing large number of open-source projects mainly focus on data mining or on static analysis techniques. On the contrary, research applying dynamic analyses such as Runtime Verification (RV) to open-source projects is scarce. This is due to lack of automatic means for executing arbitrary pieces of software that rely on complex dependencies and input parameters. In this paper, we present a fully automated infrastructure, JUniVerse, to conduct large-scale studies on unit tests in open-source projects in the wild. The proposed infrastructure runs on a cluster for parallel execution. We demonstrate the effectiveness of JUniVerse by conducting a large-scale study on Java projects hosted on GitHub. We apply a selection criteria based on static analysis to select 3 490 active projects. To show the feasibility of JUniVerse, we choose RV as a case study, and investigate the applicability of 182 publicly available JavaMOP specifications to the code exercised by unit tests. Our study reveals that 37 (out of 182) specifications (i.e., 20%) are not applicable to the code exercised by unit tests of real-world projects. Finally, with JUniVerse, we are able to identify a set of specs and projects for future RV studies.
References
[1]
Jeff Anderson, Saeed Salem, and Hyunsook Do. 2014. Improving the effectiveness of test suite through mining historical data. In MSR. 142--151.
[2]
Ezio Bartocci, Yliès Falcone, Borzoo Bonakdarpour, Christian Colombo, Normann Decker, Klaus Havelund, Yogi Joshi, Felix Klaedtke, Reed Milewicz, Giles Reger, Grigore Rosu, Julien Signoles, Daniel Thoma, Eugen Zalinescu, and Yi Zhang. 2017. First international competition on runtime verification: Rules, benchmarks, tools, and final results of CRV 2014. In IJSTTT. 1--40.
[3]
Eric Bodden, Laurie Hendren, Patrick Lam, Ondřej Lhoták, and Nomair A. Naeem. 2007. Collaborative runtime verification with tracematches. In RV. 22--37.
[4]
Raymond P.L. Buse and Westley R. Weimer. 2010. Automatically documenting program changes. In ASE. 33--42.
[5]
Valerio Cosentino, Javier Luis, and Jordi Cabot. 2016. Findings from GitHub: Methods, datasets and limitations. In MSR. 137--141.
[6]
Normann Decker, Jannis Harder, Torben Scheffel, Malte Schmitz, and Daniel Thoma. 2016. Runtime monitoring with union-find structures. In TACAS. 868--884.
[7]
Erik Hilsdale and Jim Hugunin. 2004. Advice weaving in AspectJ. In AOSD. 26--35.
[8]
Jeff Huang, Patrick Meredith, and Grigore Rosu. 2014. Maximal sound predictive Race retection with control flow abstraction. In PLDI. 337--348.
[9]
D. Jin, P. O. Meredith, C. Lee, and G. Roşu. 2012. JavaMOP: Efficient parametric runtime monitoring framework. In ICSE. 1427--1430.
[10]
Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2014. The promises and perils of mining GitHub. In MSR. 92--101.
[11]
Shuhei Kimura, Yoshiki Higo, Hiroshi Igaki, and Shinji Kusumoto. 2012. Move code refactoring with dynamic analysis. In ICSM. 575--578.
[12]
Owolabi Legunsen, Wajih Ul Hassan, Xinyue Xu, Grigore Roşu, and Darko Marinov. 2016. How good are the specs? A study of the bug-finding effectiveness of existing Java API specifications. In ASE. 602--613.
[13]
Cristina V. Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: A map of code duplicates on GitHub. In OOPSLA. 1--28.
[14]
Rafael Lotufo, Zeeshan Malik, and Krzysztof Czarnecki. 2015. Modelling the 'hurried' bug report reading process to summarize bug reports. In ESE. 516--548.
[15]
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In FSE. 643--653.
[16]
Paul Dan Marinescu, Petr Hosek, and Cristian Cadar. 2014. Covrig: A framework for the analysis of code, test, and coverage evolution in real software. In ISSTA. 93--104.
[17]
Hoan Anh Nguyen, Robert Dyer, Tien N. Nguyen, and Hridesh Rajan. 2014. Mining preconditions of APIs in large-scale code corpus. In FSE. 166--177.
[18]
Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are mutation scores correlated with real fault detection? A large scale empirical study on the relationship between mutants and real faults. In ICSE. 537--548.
[19]
R. Ramler, M. Moser, and J. Pichler. 2016. Automated static analysis of unit test code. In SANER. 25--28.
[20]
Sarah Rastkar, Gail C. Murphy, and Gabriel Murray. 2010. Summarizing software artifacts: A case study of bug reports. In ICSE. 505--514.
[21]
Baishakhi Ray, Daryl Posnett, Vladimir Filkov, and Premkumar Devanbu. 2014. A large scale study of programming languages and code quality in Github. In FSE. 155--165.
[22]
Giles Reger, Helena Cuenca Cruz, and David Rydeheard. 2015. MarQ: Monitoring at runtime with QEA. In TACAS. 596--610.
[23]
Martin P. Robillard, Eric Bodden, David Kawrykow, Mira Mezini, and Tristan Ratchford. 2013. Automated API property inference techniques. In TSE. 613--637.
[24]
Andrea Rosà, Eduardo Rosales, and Walter Binder. 2018. Analyzing and optimizing task granularity on the JVM. In CGO. 27--37.
[25]
Riccardo Roveda, Francesca Arcelli Fontana, Claudia Raibulet, Marco Zanoni, and Federico Rampazzo. 2017. Does the migration to GitHub relate to internal software quality?. In ENASE. 293--300.
[26]
Aibek Sarimbekov, Andreas Sewe, Walter Binder, Philippe Moret, and Mira Mezini. 2014. JP2: Call-site aware calling context profiling for the JVM. In SCP. 146 -- 157.
[27]
Xiaobing Sun, Bixin Li, Hareton Leung, Bin Li, and Yun Li. 2015. MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks. In INFSOF. 1 -- 12.
[28]
Walter Tichy. 2010. An Interview with Prof. Andreas Zeller: Mining your way to software reliability. In Ubiquity.
[29]
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An empirical investigation into the nature of test smells. In ASE. 4--15.
[30]
Yudi Zheng, Andrea Rosà, Luca Salucci, Yao Li, Haiyang Sun, Omar Javed, Lubomír Bulej, Lydia Y. Chen, Zhengwei Qi, and Walter Binder. 2016. Auto-Bench: Finding workloads that you need using pluggable hybrid analyses. In SANER. 639--643.
Index Terms
- JUniVerse: large-scale JUnit-test analysis in the wild
Recommendations
The CSI Framework for Compiler-Inserted Program Instrumentation
SIGMETRICS '18The CSI framework provides comprehensive static instrumentation that a compiler can insert into a program-under-test so that dynamic-analysis tools - memory checkers, race detectors, cache simulators, performance profilers, code-coverage analyzers, etc. ...
Decoupled lifeguards: enabling path optimizations for dynamic correctness checking tools
PLDI '10: Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and ImplementationDynamic correctness checking tools (a.k.a. lifeguards) can detect a wide array of correctness issues, such as memory, security, and concurrency misbehavior, in unmodified executables at run time. However, lifeguards that are implemented using dynamic ...
Decoupled lifeguards: enabling path optimizations for dynamic correctness checking tools
PLDI '10Dynamic correctness checking tools (a.k.a. lifeguards) can detect a wide array of correctness issues, such as memory, security, and concurrency misbehavior, in unmodified executables at run time. However, lifeguards that are implemented using dynamic ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2019
2682 pages
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 April 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
SAC '19
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,650 of 6,669 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 97Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)1
Reflects downloads up to 22 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in