research-article

Open-source code repository attributes predict impact of computer science research

Authors:

Prajjwal Bhattarai,

Mohammed Ghassemi,

Tuka AlhanaiAuthors Info & Claims

JCDL '22: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

Article No.: 16, Pages 1 - 7

https://doi.org/10.1145/3529372.3530927

Published: 20 June 2022 Publication History

Abstract

With an increased importance of transparency and reproducibility in computer science research, it has become common to publicly release open-source repositories that contain the code, data, and documentation alongside a publication. We study the relationship between transparency of a publication (as represented by the attributes of its open-source repository) and its scientific impact (as represented by paper citations). Using the Mann-Whitney test and Cliff's delta, we observed a statistically significant difference in citations between papers with and without an associated open-source repository. We also observed a statistically significant correlation (p < 0.01) between citations and several repository interaction features: Stars, Forks, Subscribers and Issues. Finally, using time-series features of repository growth (Stars), we trained a classifier to predict whether a paper would be highly cited (top 10%) with cross-validated AUROC of 0.8 and AUPRC of 0.65. Our results provide evidence that those who make sustained efforts in making their works transparent also tend to have a higher scientific impact.

References

[1]

2020. Research, reuse, repeat. Nature Machine Intelligence 2, 12 (Dec. 2020), 729--729.

[2]

Xiaomei Bai, Fuli Zhang, and Ivan Lee. 2019. Predicting the citations of scholarly paper. Journal of Informetrics 13, 1 (2019), 407--418.

[3]

Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (01 May 2016), 452--454.

[4]

Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Predicting the Popularity of GitHub Repositories. In Proceedings of the The 12th International Conference on Predictive Models and Data Analytics in Software Engineering (Ciudad Real, Spain) (PROMISE 2016). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages.

Digital Library

[5]

Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Understanding the Factors That Impact the Popularity of GitHub Repositories. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). 334--344.

[6]

Lutz Bornmann, Felix de Moya Anegón, and Loet Leydesdorff. 2012. The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. Journal of Informetrics 6, 2 (2012), 333--335.

[7]

L. Bornmann, L. Leydesdorff, and R. Mutz. 2013. The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics 7, 1 (2013), 158--165. cited By 99.

[8]

Jorge Ramón Fonseca Cacho and Kazem Taghva. 2020. The State of Reproducible Research in Computer Science. In 17th International Conference on Information Technology-New Generations (ITNG 2020), Shahram Latifi (Ed.). Springer International Publishing, Cham, 519--524.

[9]

Tanmoy Chakraborty, Suhansanu Kumar, Pawan Goyal, Niloy Ganguly, and Animesh Mukherjee. 2015. On the Categorization of Scientific Citation Profiles in Computer Science. Commun. ACM 58, 9 (Aug. 2015), 82--90.

Digital Library

[10]

Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114, 3 (1993), 494--509.

[11]

Laura Dabbish, Colleen Stuart, Jason Tsay, and James Herbsleb. 2013. Leveraging Transparency. IEEE Software 30, 1 (2013), 37--43.

Digital Library

[12]

NA Diamantidis, Dimitris Karlis, and Emmanouel A Giakoumakis. 2000. Unsupervised stratification of cross-validation for accuracy estimation. Artificial Intelligence 116, 1--2 (2000), 1--16.

Digital Library

[13]

Mikhail G. Dozmorov. 2018. GitHub Statistics as a Measure of the Impact of Open-Source Bioinformatics Software. Frontiers in Bioengineering and Biotechnology 6 (2018), 198.

[14]

Gunther Eysenbach. 2011. Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact. J Med Internet Res 13, 4 (16 Dec 2011), e123.

[15]

Yuanrui Fan, Xin Xia, David Lo, Ahmed E. Hassan, and Shanping Li. 2021. What makes a popular academic AI repository? Empirical Software Engineering 26, 1 (05 Jan 2021), 2.

Digital Library

[16]

Michael Färber. 2020. Analyzing the GitHub Repositories of Research Papers. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (Virtual Event, China) (JCDL '20). Association for Computing Machinery, New York, NY, USA, 491--492.

Digital Library

[17]

Jorge Ramón Fonseca Cacho and Kazem Taghva. 2018. Reproducible Research in Document Analysis and Recognition. In Information Technology - New Generations, Shahram Latifi (Ed.). Springer International Publishing, Cham, 389--395.

[18]

Santo Fortunato, Carl T. Bergstrom, Katy Börner, James A. Evans, Dirk Helbing, Staša Milojević, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert-László Barabási. 2018. Science of science. Science 359, 6379 (March 2018), eaao0185.

[19]

Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas, et al. 2006. Handling imbalanced datasets: A review. GESTS International Transactions on Computer Science and Engineering 30, 1 (2006), 25--36.

[20]

Michael Ley. 2009. DBLP: some lessons learned. Proceedings of the VLDB Endowment 2, 2 (2009), 1493--1500.

Digital Library

[21]

Alberto Martín-Martín, Enrique Orduna-Malea, Mike Thelwall, and Emilio Delgado López-Cózar. 2018. Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics 12, 4 (2018), 1160--1177.

[22]

Robert K Merton. 1968. The Matthew effect in science: The reward and communication systems of science are considered. Science 159, 3810 (1968), 56--63.

[23]

Mert Ozer, Anna Sapienza, Andrés Abeliuk, Goran Muric, and Emilio Ferrara. 2020. Discovering patterns of online popularity from time series. Expert Systems with Applications 151 (2020), 113337.

[24]

John Paparrizos and Luis Gravano. 2016. K-Shape: Efficient and Accurate Clustering of Time Series. SIGMOD Rec. 45, 1 (June 2016), 69--76.

Digital Library

[25]

Roger D. Peng. 2011. Reproducible Research in Computational Science. Science 334, 6060 (2011), 1226--1227. arXiv:https://science.sciencemag.org/content/334/6060/1226.full.pdf

[26]

Brandon K. Peoples, Stephen R. Midway, Dana Sackett, Abigail Lynch, and Patrick B. Cooney. 2016. Twitter Predicts Citation Rates of Ecological Research. PLOS ONE 11, 11 (11 2016), 1--11.

[27]

S. Redner. 1998. How popular is your paper? An empirical study of the citation distribution. The European Physical Journal B - Condensed Matter and Complex Systems 4, 2 (01 Jul 1998), 131--134.

[28]

Takaya Saito and Marc Rehmsmeier. 2015. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE 10, 3 (03 2015), 1--21.

[29]

Robert Tijssen and T Van Leeuwen. 2006. Centres of research excellence and science indicators. Can 'excellence' be captured in numbers. In Ninth international conference on science and technology indicators. Katholieke Universiteit Leuven Leuven, Belgium, 146--147.

[30]

Chih-Fong Tsai and Ming-Lun Chen. 2010. Credit rating by hybrid machine learning techniques. Applied Soft Computing 10, 2 (2010), 374--380.

Digital Library

[31]

András Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132. arXiv:https://doi.org/10.3102/10769986025002101

[32]

E.S. Vieira and J.A.N.F. Gomes. 2010. Citations to scientific articles: Its distribution and dependence on the article features. Journal of Informetrics 4, 1 (2010), 1--13.

[33]

Gary M. Weiss. 2004. Mining with Rarity: A Unifying Framework. SIGKDD Explor. Newsl. 6, 1 (June 2004), 7--19.

Digital Library

[34]

Yue Yu, Gang Yin, Huaimin Wang, and Tao Wang. 2014. Exploring the patterns of social behavior in GitHub. In Proceedings of the 1st international workshop on crowd-based software development methods and technologies. 31--36.

Digital Library

Cited By

Brown ESchwartz LHuang RWeber NKlein MBen-David AJäschke RKelly M(2024)Soft-Search: Two Datasets to Study the Identification and Production of Research SoftwareProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00040(228-231)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00040
Kang DKang TJang J(2023)Papers with code or without code? Impact of GitHub repository usability on the diffusion of machine learning researchInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10347760:6Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ipm.2023.103477
Di Nunzio G(2023)Focused Issue on Digital Library Challenges to Support the Open Science ProcessInternational Journal on Digital Libraries10.1007/s00799-023-00388-924:4(185-189)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1007/s00799-023-00388-9

Index Terms

Open-source code repository attributes predict impact of computer science research
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
    2. Publishing

Recommendations

Do interdisciplinary research teams deliver higher gains to science?

The present paper takes its place in the stream of studies that analyze the effect of interdisciplinarity on the impact of research output. Unlike previous studies, in this study the interdisciplinarity of the publications is not inferred through their ...
Authorship, patents, citations, acknowledgments, tweets, reader counts and the multifaceted reward system of science
ASIST '15: Proceedings of the 78th ASIS&T Annual Meeting: Information Science with Impact: Research in and for the Community

Building upon well-established paradigms brought forth by such theorists as Robert K. Merton, Pierre Bourdieu, and Blaise Cronin, the panel will span the full cycle of academic production to show, through various bibliometric measures and other ...
Do “interdisciplinary” disciplines have an interdisciplinary impact?: Examining citations between educational technology and library and information science journals
Abstract
This study examines the exchange of citations between two disciplines – Educational Technology (ET) and Library and Information Science (LIS) – as an indication of their interdisciplinary impact. The study was conducted in three phases. The first ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

JCDL '22: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries

June 2022

392 pages

ISBN:9781450393454

DOI:10.1145/3529372

General Chairs:
Akiko Aizawa
National Institute of Informatics, Japan
,
Thomas Mandl
University of Hildesheim, Germany
,
Zeljko Carevic
GESIS - Leibniz Institute for the Social Sciences, Germany
,
Program Chairs:
Annika Hinze
University of Waikato, New Zealand
,
Philipp Mayr
GESIS - Leibniz Institute for the Social Sciences, Germany
,
Philipp Schaer
TH Köln (University of Applied Sciences), Germany

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

IEEE Technical Committee on Digital Libraries (TC DL)

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

JCDL '22

Sponsor:

JCDL '22: The ACM/IEEE Joint Conference on Digital Libraries in 2022

June 20 - 24, 2022

Cologne, Germany

Acceptance Rates

JCDL '22 Paper Acceptance Rate 35 of 132 submissions, 27%;

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Upcoming Conference

JCDL '24

Sponsor:
sigir
sigir

The 2024 ACM/IEEE Joint Conference on Digital Libraries

December 16 - 20, 2024

Hong Kong , China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
196
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)11

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Brown ESchwartz LHuang RWeber NKlein MBen-David AJäschke RKelly M(2024)Soft-Search: Two Datasets to Study the Identification and Production of Research SoftwareProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00040(228-231)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00040
Kang DKang TJang J(2023)Papers with code or without code? Impact of GitHub repository usability on the diffusion of machine learning researchInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10347760:6Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.ipm.2023.103477
Di Nunzio G(2023)Focused Issue on Digital Library Challenges to Support the Open Science ProcessInternational Journal on Digital Libraries10.1007/s00799-023-00388-924:4(185-189)Online publication date: 29-Nov-2023
https://dl.acm.org/doi/10.1007/s00799-023-00388-9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents