Cross-project code clones in GitHub

Code reuse has well-known benefits on code quality, coding efficiency, and maintenance. Open Source Software (OSS) programmers gladly share their own code and they happily reuse others’. Social programming platforms like GitHub have normalized code foraging via their common platforms, enabling code search and reuse across different projects. Removing project borders may facilitate more efficient code foraging, and consequently faster programming. But looking for code across projects takes longer and, once found, may be more challenging to tailor to one’s needs. Learning how much code reuse goes on across projects, and identifying emerging patterns in past cross-project search behavior may help future foraging efforts. Our contribution is two fold. First, to understand cross-project code reuse, here we present an in-depth empirical study of cloning in GitHub. Using Deckard, a popular clone finding tool, we identified copies of code fragments across projects, and investigate their prevalence and characteristics using statistical and network science approaches, and with multiple case studies. By triangulating findings from different analysis methods, we find that cross-project cloning is prevalent in GitHub, ranging from cloning few lines of code to whole project repositories. Some of the projects serve as popular sources of clones, and others seem to contain more clones than their fair share. Moreover, we find that ecosystem cloning follows an onion model: most clones come from the same project, then from projects in the same application domain, and finally from projects in different domains. Second, we utilized these results to develop a novel tool named CLONE-HUNTRESS that streamlines finding and tracking code clones in GitHub. The tool is GitHub integrated, built around a user-friendly interface and runs efficiently over a modern database system. We describe the tool and make it publicly available at http://clone-det.ictic.sharif.edu/.

1. See StackExchange question: http://progr-ammers.stack-exchange.com/-questions-/-193415/best-prac-tices-for-sharing-tiny-snippets-of-code-across-projects

2. https://searchcode.com

3. Since shorter exact clones can capture, to some extent, more variability during longer code evolution.

4. All these operations are done using MySQL Server and SQL queries.

5. The project domain identification process was implemented with Python.

6. https://github.com/cloudera/kitten

7. https://github.com/mongodb/mongo-hadoop

8. https://github.com/Netflix/servo

9. https://github.com/ideaconsult/apps-ambit

10. https://github.com/vivo-project/Vitro

11. https://github.com/scottfrazer/hermes

12. https://github.com/mendhak/gpslogger

13. https://github.com/nifty-gui/nifty-gui

14. https://github.com/hoegertn/restdoc-java-server

15. We used R and the “igraph” package for all graph constructions, comparisons and analyses.

16. The graphs are created using Gephi.

17. We also investigated the correlation between clone density and project size. The results were similar to those in Fig. 2.

18. This number is among the set of the projects that had any clones at all. So the total sum of all domain sizes adds up to the first row numbers of Table 3.

19. This number is derived from the implementation of queries described in Section 4.1 and using GHtorrent’s 2018-04-01 dump of GitHub projects.

We thank Prof. Prem Devanbu and members of the DECAL lab at UC Davis for valuable discussions. We also thank Mr. Seyed Mohammad Masoud Sadrnezhaad for his help in updating CLONE-HUNTRESS’s database.

Authors and Affiliations

1. Department of Computer Science, University of California, Davis, CA, 95616, USA

   Mohammad Gharehyazie & Vladimir Filkov

2. AICT Innovation Center, Sharif University of Technology, Tehran, Iran

   Mohammad Gharehyazie

3. Columbia University, 500 West 120 Street, MC0401, New York, NY, 10027, USA

   Baishakhi Ray

4. Sharif University of Technology, Tehran, Iran

   Mehdi Keshani & Masoumeh Soleimani Zavosht

5. Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

   Abbas Heydarnoori

Corresponding author

Correspondence to Mohammad Gharehyazie.

Communicated by: Abram Hindle and Lin Tan

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: CLONE-HUNTRESS tool description and use

Here we describe CLONE-HUNTRESS, our online tool for (1) identifying clones between a user selected source project and a target list of Java-based GitHub projects, and (2) tracking changes to the clones over time. Our design goal was to provide a GitHub integrated, comprehensive, and efficient tool that users can interact with transparently, without the need to experience the mechanics of the clone search process. We wanted users to be able to come back to the tool over time and be able to monitor the changes to the cloned code. The tool is available at http://clone-det.ictic.sharif.edu/.

Finding clones among the many projects that exist in GitHub is very time consuming and computationally infeasible, specially when constrained by a reasonable response time limit. Also, as per our findings in the main text of this paper, clones are often found in pairs of projects in the same domain. Hence, to speed up the search among projects, CLONE-HUNTRESS allows users to search and track clones between projects in the same domain.

We selected a list of projects consisting of the 39422 Java-based GitHub projects, as an initial preset list that will grow over time through automatic addition of users’ projects. This number is derived from the implementation of the queries described in Section 4.1 and applying them to the April^st2018 GHTorrent MySQL dump. In other words we selected Java projects that had at least 2 developers, were at least 1 year old, and had more than 10 commits. We also eliminated projects that were forked.

The front page of the tool is shown in Fig. 12.

Users of the tool will encounter the front page when they access the tool link

Full size image

1.1 A.1 Login, registration and settings

CLONE-HUNTRESS is GitHub integrated. To use CLONE-HUNTRESS a user must first get authenticated through GitHub. Once authenticated, CLONE-HUNTRESS automatically pulls the list of the user’s publicly available projects and adds them to their profile within the tool. Users can chose one from these projects, or add other projects manually, as described later, as the source project for clone detection.

By clicking on the user’s GitHub name, email, or avatar on the dashboard, the Profile page is shown, where users can change the tool’s tracking frequency settings. As shown in Fig. 13, there are two options that govern CLONE-HUNTRESS’s behavior. The first one is the update frequency of the tracked clones. This frequency determines how often the tool should update the changes that are taking place on the tracked clone code. The second one is the frequency at which clone detection is executed from scratch. This option exists because after a sufficiently long time, many of the tracked clones may change via commits, and thus may not be similar anymore to the original clone in the user’s project.

Code clone and project update tracking frequencies can be changed on this page. It is accessible from the user’s dashboard, by clicking on the user’s GitHub name

Full size image

1.2 A.2 Detecting and tracking clones

The main functionality of CLONE-HUNTRESS i.e., tacking clones, is accessible through the ”Add project” button on the top right corner of the dashboard (Fig. 14) which redirects the user to the corresponding page (Fig. 15, top) where users can select a project from their list of GitHub projects. In addition to the list of user’s GitHub projects, any other project of interest can be selected as the source by providing its URL directly, as illustrated in Fig. 15 (top). Once a project is specified, the tool will ask for the project’s application domain, and once it is specified and ”Get projects” is pressed, it will present a list of all projects (within its current project list) in that application domain (Fig. 15, bottom).

The main tool dashboard. The results of all clone detections are shown here as a list which allows navigation to all of the tracked clone instances and change reports

Full size image

Top: Users can select from their own GitHub projects or any other random GitHub project by providing its url. Bottom: The tool proposes a list of target projects to the user

Full size image

Users can select up to 20 target projects from the given list, to detect clones between them and the source project. These limitations are imposed for two reasons: 1) Hardware resource limitations and response time limits and 2) The fact that tracking a large number of projects eventually leads to confusion rather than providing benefits. Users are also able to add any other GitHub project to the target list by specifying the project link directly, using the “Add other project” button below the list, as illustrated in Fig. 16. The target list can be reset to its original form using the “Reset project list” button at the bottom of the list.

Users can add projects to the list directly via their URLs

Full size image

With the source and target projects chosen, clone detection is initiated by pressing the “Detect-Clones” button at the bottom of the page. It could take the tool a few minutes to show the results of clone detection. When done, CLONE-HUNTRESS will redirect the user to the result page, which will resemble Fig. 17. If any clones are found, the results will show the clone instances from the source project and those from the target projects.

Results of clone detections are some traceable clone instances

Full size image

Users can choose to track any clone instance they want by selecting them and clicking on the “Save and track” button, and over time see the changes that occur on these selected instances. There is a limitation on the number of traceable clones. Users can track up to 20 clone instances due to the aforementioned reasons. After choosing some clone instances to track, users are returned to their dashboard. Every clone detection that the user has done will be displayed as a row in a table placed in the dashboard page, as shown in Fig. 14.

1.3 A.3 Tracking reports

CLONE-HUNTRESS provides View, Edit, and Delete functions in each row of the clone detection table (see the buttons in the ACTION column in Fig. 14). The View buttons report the tracking of changes made to the respective clone instances. Our tool checks at pre-specified intervals whether or not the clone instances have changed, and if so, the number of changes will be displayed as a notification on the View button. The intervals are identified by the update frequencies of tracked clones, found under the Profile page, as mentioned before. Clicking on the View button will redirect users to an “Alerts and Reports” page for that clone, similar to Fig. 18. There, clones from the user’s source project will be shown, and below each there will be the tracked clone instances, and links to the actual code. Changed clone instances are marked and users can visit the changed files. It is also possible for users to untrack any clones or clone instances from this page.

The Alerts And Reports page

Full size image

Edit directs users to a page similar to the first page of the process (Fig. 19), where users can repeat the steps of clone detection. The tool shows them all the steps they have already taken, and they can change anything they want and re-run clone detection again. Through the Delete button, the corresponding entry be deleted, and so the results of clone detection for that specific project will disappear.

Clone detections that have already been completed can be edited by the user

Full size image

1.4 A.4 Future improvements

While we have tried our best to provide a polished and useful product, there are many ways in which our tool can be improved. The first and foremost thing is to improve its hardware resource so that clone detection and checking for updates does not take as much time and users would be able to check for clones across more projects. The second area of improvement is to provide documentation and access to CLONE-HUNTRESS’s web services so other developers may integrate its functionalities within other tools and environments such as Eclipse.

Reprints and permissions