research-article

Towards a Scientific Impact Measuring Framework for Large Computing Facilities - a Case Study on XSEDE

Authors:

Gregor von Laszewski,

Geoffrey C. Fox,

Thomas R. Furlani,

Robert L. DeLeon,

Steven M. GalloAuthors Info & Claims

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

Article No.: 25, Pages 1 - 8

https://doi.org/10.1145/2616498.2616507

Published: 13 July 2014 Publication History

Abstract

We present a framework that (a) integrates publication and citation data retrieval, (b) allows scientific impact metrics generation at different aggregation levels, and (c) provides correlation analysis of impact metrics based on publication and citation data with resource allocation for a computing facility. Furthermore, we use this framework to conduct a scientific impact metrics evaluation of XSEDE. We carry out an extensive statistical analysis correlating XSEDE allocation size to the impact metrics aggregated by project and field of science. This analysis not only helps to provide an indication of XSEDE's scientific impact, but also provides insight regarding maximizing the return on investment in terms of allocation by taking into account the field of science or project based impact metrics. The findings from this analysis can be utilized by the XSEDE resource allocation committee to help assess and identify projects with higher scientific impact. It can also help provide metrics regarding the return on investment for XSEDE resources, or other institutional or campus resources for which an analysis of impact based on publications is important.

References

[1]

altmetrics. Web Page. URL: http://altmetrics.org/manifesto/.

[2]

Google Scholar. Web Page. URL: http://scholar.google.com/.

[3]

i-10 index | google scholar citations open to all. URL: http://googlescholar.blogspot.com/2011/11/google-scholar-citations-open-to-all.html.

[4]

ISI Web of Science. Web Page. URL: http://wokinfo.com/.

[5]

Mendeley. Web Page. URL: http://www.mendeley.com/.

[6]

Microsoft Academic Search. Web Page. URL: http://academic.research.microsoft.com/.

[7]

nanoHUB.org - Citations. Web Page. URL: https://nanohub.org/citations.

[8]

Publish or Perish. Web Page. URL: http://www.harzing.com/pop.htm.

[9]

Tas scientific impact metrics and analysis dev/testing portal. URL: http://fgdev.pti.indiana.edu:8088/xdportalpub/.

[10]

XSEDE. Web Page. URL: https://www.xsede.org/.

[11]

J. Bollen, G. Fox, and P. R. Singhal. How and where the TeraGrid supercomputing infrastructure benefits science. Journal of Informetrics, 5(1):114--121, 2011.

[12]

J. Bollen, M. A. Rodriguez, and H. Van de Sompel. MESUR: Usage-based Metrics of Scholarly Impact. In Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL '07, pages 474--474, New York, NY, USA, 2007. ACM. URL: http://doi.acm.org/10.1145/1255175.1255273.

Digital Library

[13]

J. Bollen, H. Van de Sompel, A. Hagberg, and R. Chute. A principal component analysis of 39 scientific impact measures. PloS one, 4(6):e6022, 2009.

[14]

J. Bollen, H. Van de Sompel, and M. A. Rodriguez. Towards Usage-based Impact Metrics: First Results from the Mesur Project. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL '08, pages 231--240, New York, NY, USA, 2008. ACM. URL: http://doi.acm.org/10.1145/1378889.1378928.

Digital Library

[15]

L. Egghe. Theory and practise of the g-index. Scientometrics, 69(1):131--152, 2006.

[16]

T. R. Furlani, B. L. Schneider, M. D. Jones, J. Towns, D. L. Hart, S. M. Gallo, R. L. DeLeon, C.-D. Lu, A. Ghadersohi, R. J. Gentner, A. K. Patra, G. von Laszewski, F. Wang, J. T. Palmer, and N. Simakov. Using xdmod to facilitate xsede operations, planning and analysis. In Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery, XSEDE '13, pages 46:1--46:8, New York, NY, USA, 2013. ACM. URL: http://doi.acm.org/10.1145/2484762.2484763.

Digital Library

[17]

J. E. Hirsch. An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences of the United States of America, 102(46):16569--16572, 2005.

[18]

J. Kaur, D. T. Hoang, X. Sun, L. Possamai, M. JafariAsbagh, S. Patil, and F. Menczer. Scholarometer: A social framework for analyzing impact across disciplines. PloS one, 7(9):e43235, 2012.

[19]

P. Thomas and D. Watkins. Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications. Scientometrics, 41(3):335--355, 1998.

[20]

K. Yang and L. I. Meho. Citation analysis: a comparison of Google Scholar, Scopus, and Web of Science. Proceedings of the American Society for Information Science and Technology, 43(1):1--15, 2006.

Cited By

von Laszewski GWang FFox G(2021)Comprehensive Evaluation of XSEDE’s Scientific Impact using Semantic Scholar DataPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465601(1-4)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465601
Wang FLaszewski GWhitson TFox GFurlani TDeLeon RGallo SSanielevici S(2018)Evaluating the Scientific Impact of XSEDEProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3219124(1-8)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3219124
Mayernik MHart DMaull KWeber N(2017)Assessing and tracing the outcomes and impact of research infrastructuresJournal of the Association for Information Science and Technology10.1002/asi.2372168:6(1341-1359)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1002/asi.23721
Show More Cited By

Index Terms

Towards a Scientific Impact Measuring Framework for Large Computing Facilities - a Case Study on XSEDE
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Information systems
  1. Information systems applications

Recommendations

Evaluating the Scientific Impact of XSEDE
PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity

We use the bibliometrics approach to evaluate the scientific impact of XSEDE. By utilizing publication data from various sources, e.g., ISI Web of Science and Microsoft Academic Graph, we calculate the impact metrics of XSEDE publications and show how ...
Comprehensive Evaluation of XSEDE’s Scientific Impact using Semantic Scholar Data
PEARC '21: Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions

The United States science and engineering community faces multiple challenges related to funding and funding policies for science and engineering. A framework is needed to evaluate the impact of scientific facilities and instruments. In this paper, we ...
$h_{u}$ -index: a unified index to quantify individuals across disciplines
Abstract
Nowadays scientific evaluation is becoming increasingly important and necessary in many cases, such as faculty hiring, funding and promotion. Among existing evaluation metrics for individual performance, h-index is the most famous indicator and ... $_{}$ $_{}$ $_{}$ $_{}$

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

July 2014

445 pages

ISBN:9781450328937

DOI:10.1145/2616498

General Chair:
Scott Lathrop
National Center for Supercomputing Applications
,
Program Chair:
Jay Alameda
National Center for Supercomputing Applications

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

NSF: National Science Foundation
Drexel University
Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

XSEDE '14

XSEDE '14: Annual Conference of the Extreme Science and Engineering Discovery Environment

July 13 - 18, 2014

GA, Atlanta, USA

Acceptance Rates

XSEDE '14 Paper Acceptance Rate 80 of 120 submissions, 67%;

Overall Acceptance Rate 129 of 190 submissions, 68%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
122
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

von Laszewski GWang FFox G(2021)Comprehensive Evaluation of XSEDE’s Scientific Impact using Semantic Scholar DataPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465601(1-4)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465601
Wang FLaszewski GWhitson TFox GFurlani TDeLeon RGallo SSanielevici S(2018)Evaluating the Scientific Impact of XSEDEProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity10.1145/3219104.3219124(1-8)Online publication date: 22-Jul-2018
https://dl.acm.org/doi/10.1145/3219104.3219124
Mayernik MHart DMaull KWeber N(2017)Assessing and tracing the outcomes and impact of research infrastructuresJournal of the Association for Information Science and Technology10.1002/asi.2372168:6(1341-1359)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1002/asi.23721
Knepper RBörner K(2016)Comparing the Consumption of CPU Hours with Scientific Output for the Extreme Science and Engineering Discovery Environment (XSEDE)PLOS ONE10.1371/journal.pone.015762811:6(e0157628)Online publication date: 16-Jun-2016
https://doi.org/10.1371/journal.pone.0157628
Hart DRishel MNychka DGaither K(2016)Estimating the Accuracy of User Surveys for Assessing the Impact of HPC SystemsProceedings of the XSEDE16 Conference on Diversity, Big Data, and Science at Scale10.1145/2949550.2949583(1-7)Online publication date: 17-Jul-2016
https://dl.acm.org/doi/10.1145/2949550.2949583
DeLeon RFurlani TGallo SWhite JJones MPatra AInnus MYearke TPalmer JSperhac JRathsam RSimakov Nvon Laszewski GWang FPeterson G(2015)TAS view of XSEDE users and usageProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792766(1-8)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792766
von Laszewski GWang FFox GHart DFurlani TDeLeon RGallo S(2015)Peer Comparison of XSEDE and NCAR Publication DataProceedings of the 2015 IEEE International Conference on Cluster Computing10.1109/CLUSTER.2015.98(531-532)Online publication date: 8-Sep-2015
https://dl.acm.org/doi/10.1109/CLUSTER.2015.98

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents