research-article

Citations and Readership are Poor Indicators of Research Excellence: Introducing TrueImpactDataset, a New Dataset for Validating Research Evaluation Metrics

Authors:

Drahomira Herrmannova,

Robert M. Patton,

Christopher G. StahlAuthors Info & Claims

SWM '17: Proceedings of the 1st Workshop on Scholarly Web Mining

Pages 41 - 48

https://doi.org/10.1145/3057148.3057154

Published: 10 February 2017 Publication History

Abstract

In this paper we show that citation counts and Mendeley readership are poor indicators of research excellence. Our experimental design builds on the assumption that a good evaluation metric should be able to distinguish publications that have changed a research field from those that have not. The experiment has been conducted on a new dataset for bibliometric research which we call TrueImpactDataset. TrueImpactDataset is a collection of research publications of two types -- research papers which are considered seminal work in their area and papers which provide a survey (a literature review) of a research area. The dataset also contains related metadata, which include DOIs, titles, authors and abstracts. We describe how the dataset was built and provide overview statistics of the dataset. We propose to use the dataset for validating research evaluation metrics. By using this data, we show that widely used research metrics only poorly distinguish excellent research.

References

[1]

Research Excellence Framework (REF) 2014 Units of Assessment. http://www.ref.ac.uk/panels/unitsofassessment/. Accessed: 2016-11-11.

[2]

D. W. Aksnes. Characteristics of highly cited papers. Research Evaluation, 12(3):159--170, 2003.

[3]

T. C. Almind and P. Ingwersen. Informetric analyses on the world wide web: Methodological approaches to 'webometrics'. Journal of documentation, 53(4):404--426, 1997.

[4]

B. M. Althouse, J. D. West, C. T. Bergstrom, and T. Bergstrom. Differences in Impact Factor Across Fields and Over Time. Journal of the American Society for Information Science and Technology, 60(1):27--34, 2009.

Digital Library

[5]

C. A. D'Angelo and G. Abramo. Publication Rates in 192 Research Fields of the Hard Sciences. In Proceedings of the 15th ISSI Conference, pages 915--925, 2015.

[6]

F. Galligan and S. Dyas-Correia. Altmetrics: Rethinking the way we measure. Serials review, 39(1):56--61, 2013.

[7]

E. Garfield. Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159):108--111, 1955.

[8]

A.-W. Harzing. Microsoft Academic (Search): a Phoenix arisen from the ashes?, 2016.

[9]

A.-W. Harzing and S. Alakangas. Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2):787--804, 2016.

Digital Library

[10]

D. Herrmannova and P. Knoth. Semantometrics in coauthorship networks: Fulltext-based approach for analysing patterns of research collaboration. D-Lib Magazine, 21(11/12), 2015.

[11]

D. Herrmannova and P. Knoth. Simple yet effective methods for large-scale scholarly publication ranking. In WSDM Cup 2016 -- Entity Ranking Challenge Workshop at International Conference on Web Search and Data Mining (WSDM), San Francisco, CA, USA, Feb 2016.

[12]

J. E. Hirsch. An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, pages 16569--16572, 2005.

[13]

P. Ingwersen. The calculation of web impact factors. Journal of documentation, 54(2):236--243, 1998.

[14]

A. E. Jinha. Article 50 million: an estimate of the number of scholarly articles in existence. Learned Publishing, 23(3):258--263, 2010.

[15]

P. Knoth and D. Herrmannova. Towards semantometrics: A new semantic similarity based measure for assessing a research publication's contribution. D-Lib Magazine, 20(11):8, 2014.

[16]

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012.

Digital Library

[17]

M. Laakso and B.-C. Björk. Anatomy of open access publishing: a study of longitudinal development and internal structure. BMC medicine, 10(1):1, 2012.

[18]

N. Maflahi and M. Thelwall. When are readership counts as useful as citation counts? scopus versus mendeley for lis journals. Journal of the Association for Information Science and Technology, 67(1):191--199, 2016.

Digital Library

[19]

R. M. Patton, C. G. Stahl, and J. C. Wells. Measuring Scientific Impact Beyond Citation Counts. D-Lib Magazine, 22(9/10):5, 2016.

[20]

M. Peplow. Peer review -- reviewed. Nature News, Dec 2014.

[21]

H. Piwowar and J. Priem. The power of altmetrics on a cv. Bulletin of the American Society for Information Science and Technology, 39(4):10--13, 2013.

Digital Library

[22]

J. Priem. Altmetrics. In B. Cronin and C. R. Sugimoto, editors, Beyond bibliometrics: harnessing multidimensional indicators of scholarly impact, chapter 14, pages 263--288. MIT Press, Cambridge, MA, 2014.

[23]

J. Priem, D. Taraborelli, P. Groth, and C. Neylon. Altmetrics: A manifesto. 2010. Accessed: 2016-11-07.

[24]

REF 2014. Panel criteria and working methods. Technical Report January 2012, 2012.

[25]

P. O. Seglen. Why the impact factor of journals should not be used for evaluating research. BMJ: British Medical Journal, 314(February):498--502, 1997.

[26]

A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-j. P. Hsu, and K. Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th International Conference on World Wide Web, pages 243--246. ACM, 2015.

Digital Library

[27]

R. Smith. Peer review: a flawed process at the heart of science and journals. Journal of the royal society of medicine, 99(4):178--182, 2006.

[28]

M. Valenzuela, V. Ha, and O. Etzioni. Identifying meaningful citations. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.

[29]

A. D. Wade, K. Wang, Y. Sun, and A. Gulli. Wsdm cup 2016: Entity ranking challenge. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pages 593--594. ACM, 2016.

Digital Library

[30]

R. Whalen, Y. Huang, A. Sawant, B. Uzzi, and N. Contractor. Natural Language Processing, Article Content & Bibliometrics: Predicting High Impact Science. ASCW'15 Workshop at Web Science 2015, pages 6--8, 2015.

[31]

J. Wilsdon, L. Allen, E. Belfiore, P. Campbell, S. Curry, S. Hill, R. Jones, R. Kain, S. Kerridge, M. Thelwall, J. Tinkler, I. Viney, P. Wouters, J. Hill, and B. Johnson. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. 2015.

[32]

R. Yan, C. Huang, J. Tang, Y. Zhang, and X. Li. To Better Stand on the Shoulder of Giants. In Proceedings of the 12th Joint Conference on Digital Libraries, pages 51--60, Washington, DC, 2012. ACM.

Digital Library

Cited By

Herrmannova DKnoth PStahl CPatton RWells J(2023)Research Collaboration Analysis Using Text and Graph FeaturesComputational Linguistics and Intelligent Text Processing10.1007/978-3-031-23804-8_33(431-441)Online publication date: 26-Feb-2023
https://doi.org/10.1007/978-3-031-23804-8_33
Kreutz CSahitaj PSchenkel R(2020)Evaluating semantometrics from computer science publicationsScientometrics10.1007/s11192-020-03409-5Online publication date: 18-Mar-2020
https://doi.org/10.1007/s11192-020-03409-5
Bertin MAtanassova I(2018)Recommending Scientific PapersProceedings of the 1st International Conference on Digital Tools & Uses Congress10.1145/3240117.3240123(1-4)Online publication date: 3-Oct-2018
https://dl.acm.org/doi/10.1145/3240117.3240123

Recommendations

Do citations and readership identify seminal publications?

This work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to ...
Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows

Although there is evidence that counting the readers of an article in the social reference site, Mendeley, may help to capture its research impact, the extent to which this is true for different scientific fields is unknown. In this study, we compare ...
A review of the characteristics of 108 author-level bibliometric indicators

An increasing demand for bibliometric assessment of individuals has led to a growth of new bibliometric indicators as well as new variants or combinations of established ones. The aim of this review is to contribute with objective facts about the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SWM '17: Proceedings of the 1st Workshop on Scholarly Web Mining

February 2017

65 pages

ISBN:9781450352406

DOI:10.1145/3057148

Copyright © 2017 ACM.

© 2017 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

In-Cooperation

Oak Ridge National Laboratory
OU: The Open University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 February 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SWM '17

SWM '17: 1st Workshop on Scholarly Web Mining

February 10, 2017

Cambridge, United Kingdom

Acceptance Rates

SWM '17 Paper Acceptance Rate 8 of 17 submissions, 47%;

Overall Acceptance Rate 8 of 17 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
128
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Herrmannova DKnoth PStahl CPatton RWells J(2023)Research Collaboration Analysis Using Text and Graph FeaturesComputational Linguistics and Intelligent Text Processing10.1007/978-3-031-23804-8_33(431-441)Online publication date: 26-Feb-2023
https://doi.org/10.1007/978-3-031-23804-8_33
Kreutz CSahitaj PSchenkel R(2020)Evaluating semantometrics from computer science publicationsScientometrics10.1007/s11192-020-03409-5Online publication date: 18-Mar-2020
https://doi.org/10.1007/s11192-020-03409-5
Bertin MAtanassova I(2018)Recommending Scientific PapersProceedings of the 1st International Conference on Digital Tools & Uses Congress10.1145/3240117.3240123(1-4)Online publication date: 3-Oct-2018
https://dl.acm.org/doi/10.1145/3240117.3240123

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents