research-article

Open access

DéjàVu: a map of code duplicates on GitHub

Authors:

Cristina V. Lopes,

Hitesh Sajnani,

Jan VitekAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 1, Issue OOPSLA

Article No.: 84, Pages 1 - 28

https://doi.org/10.1145/3133908

Published: 12 October 2017 Publication History

Abstract

Previous studies have shown that there is a non-trivial amount of duplication in source code. This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing over 428 million files written in Java, C++, Python, and JavaScript. We found that this corpus has a mere 85 million unique files. In other words, 70% of the code on GitHub consists of clones of previously created files. There is considerable variation between language ecosystems. JavaScript has the highest rate of file duplication, only 6% of the files are distinct. Java, on the other hand, has the least duplication, 60% of files are distinct. Lastly, a project-level analysis shows that between 9% and 31% of the projects contain at least 80% of files that can be found elsewhere. These rates of duplication have implications for systems built on open source software as well as for researchers interested in analyzing large code bases. As a concrete artifact of this study, we have created DéjàVu, a publicly available map of code duplicates in GitHub repositories.

Supplementary Material

Auxiliary Archive (oopsla17-oopsla176-aux.zip)

Download
8778.16 MB

References

[1]

T. F. Bissyande, F. Thung, D. Lo, L. Jiang, and L. Reveillere. 2013. Orion: A Sotware Project Search Engine with Integrated Diverse Sotware Artifacts. In International Conference on Engineering of Complex Computer Systems.

Digital Library

[2]

Stephen M. Blackburn, Robin Garner, Chris Hofmann, Asjad M. Khan, Kathryn S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony L. Hosking, Maria Jump, Han Bok Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanovic, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo benchmarks: Java benchmarking development and analysis. In Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA).

Digital Library

[3]

Hudson Borges, André C. Hora, and Marco Tulio Valente. 2016. Understanding the Factors that Impact the Popularity of GitHub Repositories. (2016). http://arxiv.org/abs/1606.04984

[4]

Casey Casalnuovo, Prem Devanbu, Abilio Oliveira, Vladimir Filkov, and Baishakhi Ray. 2015. Assert Use in GitHub Projects. In International Conference on Sotware Engineering (ICSE). http://dl.acm.org/citation.cfm?id=2818754.2818846

[5]

James R. Cordy, Thomas R. Dean, and Nikita Synytskyy. 2004. Practical Language-independent Detection of Near-miss Clones. In Conference of the Centre for Advanced Studies on Collaborative Research (CASCON). http://dl.acm.org/citation. cfm?id=1034914.1034915

[6]

V. Cosentino, J. L. C. Izquierdo, and J. Cabot. 2016. Findings from GitHub: Methods, Datasets and Limitations. In Working Conference on Mining Sotware Repositories (MSR).

Digital Library

[7]

John W. Creswell. 2014. Research Design: ualitative, uantitative, and Mixed Methods Approaches. SAGE.

[8]

Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N. Nguyen. 2013. Boa: A Language and Infrastructure for Analyzing Ultra-large-scale Sotware Repositories. In International Conference on Sotware Engineering (ICSE). http: //dl.acm.org/citation.cfm?id=2486788.2486844

[9]

Jesus M. Gonzalez-Barahona, Gregorio Robles, and Santiago Dueñas. 2010. Collecting Data About FLOSS Development: The FLOSSMetrics Experience. In International Workshop on Emerging Trends in Free/Libre/Open Source Sotware Research and Development (FLOSS).

Digital Library

[10]

Georgios Gousios. 2013. The GHTorrent dataset and tool suite. In Working Conference on Mining Sotware Repositories (MSR).

[11]

Lars Heinemann, Florian Deissenboeck, Mario Gleirscher, Benjamin Hummel, and Maximilian Irlbeck. 2011. On the Extent and Nature of Sotware Reuse in Open Source Java Projects. Berlin, Heidelberg.

[12]

Felipe Hofa. 2016. 400,000 GitHub repositories, 1 billion files, 14 terabytes of code: Spaces or Tabs? (2016). https: //medium.com/@hofa/400-000-github-repositories-1-billion-iles-14-terabytes-of-code-spaces-or-tabs-7cfe0b5dd7fd

[13]

Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2014. The Promises and Perils of Mining GitHub. In Working Conference on Mining Sotware Repositories (MSR).

Digital Library

[14]

Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002. CCFinder: A Multilinguistic Token-based Code Clone Detection System for Large Scale Source Code. IEEE Trans. Sotw. Eng. 28, 7 (2002).

Digital Library

[15]

P. S. Kochhar, T. F. BissyandÃľ, D. Lo, and L. Jiang. 2013. Adoption of Sotware Testing in Open Source ProjectsśA Preliminary Study on 50,000 Projects. In European Conference on Sotware Maintenance and Reengineering.

Digital Library

[16]

R. Koschke. 2007. Survey of research on sotware clones. In Duplication, Redundancy, and Similarity in Sotware (Dagstuhl Seminar Proceedings 06301).

[17]

A. Mockus. 2007. Large-Scale Code Reuse in Open Source Sotware. In First International Workshop on Emerging Trends in FLOSS Research and Development.

Digital Library

[18]

A. Mockus. 2009. Amassing and Indexing a Large Sample of Version Control Systems: Towards the Census of Public Source Code History. In Working Conference on Mining Sotware Repositories (MSR).

Digital Library

[19]

Meiyappan Nagappan, Thomas Zimmermann, and Christian Bird. 2013. Diversity in Sotware Engineering Research. In Foundations of Sotware Engineering (FSE).

Digital Library

[20]

J. Ossher, Sushil Bajracharya, E. Linstead, P. Baldi, and Crista Lopes. 2009. SourcererDB: An aggregated repository of statically analyzed and cross-linked open source Java projects. In Working Conference on Mining Sotware Repositories (MSR).

Digital Library

[21]

Joel Ossher, Hitesh Sajnani, and Cristina Lopes. 2011. File Cloning in Open Source Java Projects: The Good, the Bad, and the Ugly. In International Conference on Sotware Maintenance (ICSM).

Digital Library

[22]

Baishakhi Ray, Daryl Posnet, Vladimir Filkov, and Premkumar Devanbu. 2014. A Large Scale Study of Programming Languages and Code uality in Github. In International Symposium on Foundations of Sotware Engineering (FSE).

Digital Library

[23]

Gregor Richards, Andreas Gal, Brendan Eich, and Jan Vitek. 2011. Automated Construction of JavaScript Benchmarks. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA).

Digital Library

[24]

C. K. Roy and J. R. Cordy. 2007. A survey on sotware clone detection research. Technical Report 541. ueens University.

[25]

Chanchal K. Roy and James R. Cordy. 2009. A Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools. In International Conference on Sotware Testing, Verification, and Validation.

Digital Library

[26]

C. K. Roy and J. R. Cordy. 2010. Near-miss Function Clones in Open Source Sotware: An Empirical Study. J. Sotw. Maint. Evol. 22, 3 (2010).

[27]

Hitesh Sajnani. 2016. Large-Scale Code Clone Detection. Ph.D. Dissertation. University of California, Irvine.

[28]

Hitesh Sajnani, Vaibhav Saini, Jefrey Svajlenko, Chanchal K. Roy, and Cristina V. Lopes. 2016. SourcererCC: Scaling Code Clone Detection to Big-code. In International Conference on Sotware Engineering (ICSE).

Digital Library

[29]

Johnny Saldaña. 2009. The Coding Manual for ualitative Researchers. SAGE.

[30]

SPEC. 1998. SPECjvm98 benchmarks. (1998).

[31]

J. Svajlenko and C. K. Roy. 2015. Evaluating clone detection tools with BigCloneBench. In International Conference on Sotware Maintenance and Evolution (ICSME).

Digital Library

[32]

Christopher Vendome, Gabriele Bavota, Massimiliano Di Penta, Mario Linares-Vásquez, Daniel German, and Denys Poshyvanyk. 2016. License usage and changes: a large-scale study on GitHub. Empirical Sotware Engineering (2016).

Digital Library

Cited By

Zhang XXu ZYang SLi ZShi ZSun L(2024)Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-task LearningProceedings of the ACM on Software Engineering10.1145/36607821:FSE(1679-1702)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660782
Lachance AMosser SCombemale BWimmer MChechik MEgyed A(2024)Building deduplicated model repositories to assess domain-specific languages evolutionProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688338(1025-1034)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3688338
Jahanshahi MMockus ASpinellis DConstantinou EBacchelli A(2024)Dataset: Copy-based Reuse in Open Source SoftwareProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644868(42-47)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644868
Show More Cited By

Index Terms

DéjàVu: a map of code duplicates on GitHub
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Near-duplicate and plagiarism detection
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Ultra-large-scale systems

Recommendations

Cross-project code clones in GitHub

Code reuse has well-known benefits on code quality, coding efficiency, and maintenance. Open Source Software (OSS) programmers gladly share their own code and they happily reuse others'. Social programming platforms like GitHub have normalized code ...
On the naturalness of auto-generated code: can we identify auto-generated code automatically?
ICPC '18: Proceedings of the 26th Conference on Program Comprehension

Recently, a variety of studies have been conducted on source code analysis. If auto-generated code is included in the target source code, it is usually removed in a preprocessing phase because the presence of auto-generated code may have negative effects ...
Is cloned code older than non-cloned code?
IWSC '11: Proceedings of the 5th International Workshop on Software Clones

It is still a debated question whether cloned code causes increased maintenance efforts. If cloned code is more stable than non-cloned code, i.e. it is changed less often, it will require less maintenance efforts. The more stable cloned code is, the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 1, Issue OOPSLA

October 2017

1786 pages

EISSN:2475-1421

DOI:10.1145/3152284

Issue’s Table of Contents

Copyright © 2017 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2017

Published in PACMPL Volume 1, Issue OOPSLA

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

117
Total Citations
View Citations
7,975
Total Downloads

Downloads (Last 12 months)591
Downloads (Last 6 weeks)78

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XXu ZYang SLi ZShi ZSun L(2024)Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-task LearningProceedings of the ACM on Software Engineering10.1145/36607821:FSE(1679-1702)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660782
Lachance AMosser SCombemale BWimmer MChechik MEgyed A(2024)Building deduplicated model repositories to assess domain-specific languages evolutionProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3652620.3688338(1025-1034)Online publication date: 22-Sep-2024
https://dl.acm.org/doi/10.1145/3652620.3688338
Jahanshahi MMockus ASpinellis DConstantinou EBacchelli A(2024)Dataset: Copy-based Reuse in Open Source SoftwareProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644868(42-47)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644868
AlOmar EKnobloch BKain TKalish CMkaouer MOuni ARoychoudhury APaiva AAbreu RStorey M(2024)AntiCopyPaster 2.0: Whitebox just-in-time code duplicates extractionProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3640035(84-88)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3640035
Misu MAchar RLopes C(2024)SourcererJBF: A Java Build Framework For Large-Scale CompilationACM Transactions on Software Engineering and Methodology10.1145/363571033:3(1-35)Online publication date: 15-Mar-2024
https://dl.acm.org/doi/10.1145/3635710
OBrien DDyer RNguyen TRajan HRoychoudhury APaiva AAbreu RStorey M(2024)Data-Driven Evidence-Based Syntactic Sugar DesignProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639580(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639580
Na YWoo SLee JLee HRoychoudhury APaiva AAbreu RStorey M(2024)CNEPS: A Precise Approach for Examining Dependencies among Third-Party C/C++ Open-Source ComponentsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639209(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639209
Ahmed TPai KDevanbu PBarr ERoychoudhury APaiva AAbreu RStorey M(2024)Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)Proceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639183(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639183
Jiang LAn JHuang HTang QNie SWu SZhang YRoychoudhury APaiva AAbreu RStorey M(2024)BinaryAI: Binary Software Composition Analysis via Intelligent Binary Source Code MatchingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639100(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639100
He YWang YZhu SWang WZhang YLi QYu A(2024)Automatically Identifying CVE Affected Versions With Patches and Developer LogsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2023.326456721:2(905-919)Online publication date: Mar-2024
https://doi.org/10.1109/TDSC.2023.3264567
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents