Abstract
We begin with a short history of measuring science and discuss how the Science Citation Index has revolutionized the quantitative study of science and created a strong application potential. After reviewing the rationale of bibliometric analysis, we present the basic principle of the bibliometric methodology, with complex citation networks as a starting point. We show that the two main pillars of advanced bibliometric methods, citation-based analysis and science mapping, are both reducible to one and the same principle. From this basic principle we deduce a set of main indicators, particularly for the assessment of research output and international impact. Important elements include new approaches for identifying fields and research themes on the basis of a publication-level rather than a journal-level network; publication and citation counting; normalization of citation measures; the use of indicators based on averages versus those based on citation distributions; and weighting procedures and statistical reliability. In this account of the state of the art of advanced bibliometrics , we highlight in particular the developments in our Leiden institute, given its long-standing, extensive, and broad experience.
The next part of this chapter deals with practical applications of indicators, particularly real-life examples of evaluation studies. We further discuss several crucial issues such as the use of journal impact factors and h-index; the relation between peer review judgment and bibliometric findings; definition and delimitation of fields; assignment of publications; the influence of open access; webometrics and altmetrics; ranking of universities; and general objections to bibliometric analysis.
The second main pillar of the advanced bibliometric methodology is the development of science maps. We discuss the basic elements and the construction of both citation-relation and word-relation science maps. Further, we present a method to combine the two main pillars: the integration of citation analysis in science maps. This combined citation analysis and science mapping can be used to explore research related to socioeconomic problems. Recently developed bibliometric instruments enable tunable mapping, which opens up new analytical opportunities in monitoring scientific research. Finally, we contend that bibliometric indicators and maps are not just evaluation tools for science policymakers, research managers, and individual researchers, but also powerful instruments in the study of science.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
A.F.J. van Raan: Measuring science. Capita selecta of current main issues. In: Handbook of Quantitative Science and Technology Research, ed. by H.F. Moed, W. Glänzel, U. Schmoch (Springer, Dordrecht 2004) pp. 19–50
A.F.J. van Raan: Citations, h-index, journal impact and rankings: Not all sorrow and misery. CWTS: A short history of measuring science. In: Perspectives on the Past: 50 Years of FSW, ed. by J. van Holsteyn, R. Mom, I. Smit, H. Tromp, G. Wolters (Biblioscope, Utrecht 2013) pp. 86–103
A.F.J. van Raan: Advances in bibliometric analysis: research performance assessment and science mapping. In: Bibliometrics: Use and Abuse in the Review of Research Performance, Wenner-Gren International Series, Vol. 87, ed. by W. Blockmans, L. Engwall, D. Weaire (London, Portland 2014) pp. 17–28
F. Narin: Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity (National Science Foundation, Washington D.C. 1976)
A. de Candolle: Histoire des sciences et des savants depuis deux siècles (Fayard, Paris 1987), 1st edn. 1873, 2nd edn. 1885 (Genève/Basel, H. Georg)
A.J. Lotka: The frequency distribution of scientific productivity, J. Wash. Acad. Sci. 16, 317–323 (1926)
S.C. Bradford: Documentation (Crosby, London 1948)
R.S. Daniel, C.M. Louttit: Professional Problems in Psychology (Prentice-Hall, New York 1953)
E. Garfield: Citation indexes for science: A new dimension in documentation through association of ideas, Science 122(3159), 108–111 (1955)
P.F. Wouters: The Citation Culture, Ph.D. Thesis (University of Amsterdam, Amsterdam 1999)
D.J. de Solla Price: Little Science, Big Science (Yale Univ. Press, New Haven 1963)
D.J. de Solla Price: Networks of scientific papers, Science 149, 510–515 (1965)
R.K. Merton: The Matthew effect in science. The reward and communication systems of science are considered, Science 159(3810), 56–63 (1968)
D.J. de Solla Price: Science Since Babylon (Yale Univ. Press, New Haven 1975)
D.J. de Solla Price: A general theory of bibliometric and other cumulative advantage processes, J. Am. Soc. Inf. Sci. 27(5/6), 292–306 (1976)
M.M. Kessler: Bibliographic coupling between scientific papers, Am. Doc. 14, 10–25 (1963)
H. Small: Co-citation in the scientific literature: a new measure of the relationship between two documents, J. Am. Soc. Inf. Sci. Technol. 24, 265–269 (1973)
S. Cole, J.R. Cole: Scientific output and recognition, Am. Sociol. Rev. 62, 377–390 (1967)
J.R. Cole, S. Cole: The Ortega hypothesis, Science 178, 368 (1972)
National Science Board: Science Indicators 1972 (Government Printing Office, Washington DC 1973)
OECD: The Measurement of Scientific and Technological Activities. Frascati Manual (Organization for Economic Co-operation and Development (OECD), Paris 1963)
E. Garfield: Citation analysis as a tool in journal evaluation, Science 178(4060), 471–479 (1972)
E. Garfield: Citation Indexing: Its Theory and Application in Science, Technology and Humanities (Wiley, New York 1979)
G. Pinski, F. Narin: Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics, Inf. Process. Manag. 12(5), 297–312 (1976)
M.P. Carpenter, M. Cooper, F. Narin: Linkage between basic research literature and patents, Res. Manag. 23(2), 30–35 (1980)
K.H. Chang: Evaluation and Survey of a Subfield of Physics: Magnetic Resonance and Relaxation Studies in the Netherlands, FOM-Report, Vol. 37175 (FOM, Utrecht 1975)
C.J.G. Bakker: Elektronenmicroscopie in Nederland, FOM-Report, Vol. 43105 (FOM, Utrecht 1977)
Y. Elkana, J. Lederberg, R.K. Merton, A. Thackray, H. Zuckerman (Eds.): Toward a Metric of Science: The Advent of Science Indicators (Wiley, New York 1978)
D.D. Beaver, R. Rosen: Studies in scientific collaboration, 1: Professional origins of scientific co-authorship, Scientometrics 1, 65–84 (1978)
G.N. Gilbert: Measuring the growth of science- review of indicators of scientific growth, Scientometrics 1, 9–34 (1978)
M.J. Moravcsik: Phenomenology and models of growth of science, Res. Policy 4, 80–86 (1975)
J. Vláchy: Mobility in science. Bibliography of scientific career migration, field mobility, international academic circulation and brain drain, Scientometrics 1, 201–228 (1979)
H.F. Moed, W.J.M. Burger, J.G. Frankfort, A.F.J. van Raan: The use of bibliometric data for the measurement of university-research performance, Res. Policy 14(3), 131–149 (1985)
B.R. Martin, J. Irvine: Assessing basic research: Some partial indicators of scientific progress in radio astronomy, Res. Policy 12, 61–90 (1983)
D. Sullivan, D. Koester, D.H. White, R. Kern: Understanding rapid theoretical change in particle physics- a month-by-month co-citation analysis, Scientometrics 2, 309–319 (1980)
H.D. White, B.C. Griffith: Author cocitation – a literature measure of intellectual structure, J. Am. Soc. Inf. Sci. 32, 163–171 (1981)
K.W. McCain: Longitudinal author cocitation mapping – the changing structure of macroeconomics, J. Am. Soc. Inf. Sci. 35, 351–359 (1984)
S.D. Haitun: Stationary scientometric distributions. 1: Different approximations, Scientometrics 4, 89–104 (1982)
A. Schubert, W. Glänzel: Statistical reliability of comparisons based on the citation impact of scientometric publications, Scientometrics 5, 59–74 (1983)
B.C. Peritz: A classification of citation roles for the social sciences and related fields, Scientometrics 5, 303–312 (1983)
M.E.D. Koenig: Bibliometric indicators versus expert opinion in assessing research performance, J. Am. Soc. Inf. Sci. 34, 136–145 (1983)
A.L. Porter, D.E. Chubin: An indicator of cross-disciplinary research, Scientometrics 8, 161–176 (1985)
A.G. Heffner: Funded research, multiple authorship, and sub-authorship collaboration in 4 disciplines, Scientometrics 3(1), 5–12 (1981)
C. Fenichel: Online searching – measures that discriminate among users with different types of experiences, J. Am. Soc. Inf. Sci. 32(1), 23–32 (1981)
K. Pavitt: Patent statistics as indicators of innovative activities—possibilities and problems, Scientometrics 7(1/2), 77–99 (1985)
L.G. Soete, S.M.E. Wyatt: The use of foreign patenting as an internationally comparable science and technology output indicator, Scientometrics 5(1), 31–54 (1983)
M. Callon, S. Bauin, J.P. Courtial, W. Turner: From translation to problematic networks: An introduction to co-word analysis, Soc. Sci. Inf. 22, 191–235 (1983)
A. Rip, J.P. Courtial: Co-word maps of biotechnology—an example of cognitive scientometrics, Scientometrics 6, 381–400 (1984)
A.F.J. van Raan (Ed.): Handbook of Quantitative Studies of Science and Technology (North Holland, Amsterdam 1988)
A. Schubert, T. Braun: Relative indicators and relational charts for comparative-assessment of publication output and citation impact, Scientometrics 9(5-6), 281–291 (1986)
T. Braun, W. Glänzel, A. Schubert: World flash on basic research—the newest version of the facts and figures on publication output and relative citation impact of 100 countries 1981–1985, Scientometrics 13(5-6), 181–188 (1988)
R.R. Braam, H.F. Moed, A.F.J. Van Raan: Mapping of science by combined co-citation and word analysis, I: Structural aspects, J. Am. Soc. Inf. Sci. 42, 233–251 (1991)
R.R. Braam, H.F. Moed, A.F.J. Van Raan: Mapping of science by combined co-citation and word analysis, II: Dynamical aspects, J. Am. Soc. Inf. Sci. 42, 252–266 (1991)
P.O. Seglen: The skewness of science, J. Am. Soc. Inf. Sci. 43, 628–638 (1992)
S. Arunachalam, R. Srinivasan, V. Raman: International collaboration in science—participation by the Asian giants, Scientometrics 30, 7–22 (1994)
F. Narin: Patent bibliometrics, Scientometrics 30, 147–155 (1994)
F. Narin, K.S. Hamilton, D. Olivastro: The increasing linkage between US technology and public science, Res. Policy 26, 317–330 (1997)
U. Schmoch: Tracing the knowledge transfer from science to technology as reflected in patent indicators, Scientometrics 26, 193–211 (1993)
A.F.J. van Raan: Fractal dimension of co-citations, Nature 347, 626 (1990)
P.O. Seglen: Causal relationship between article citedness and journal impact, J. Am. Soc. Inf. Sci. 45, 1–11 (1994)
H.F. Moed, T.N. van Leeuwen: Improving the accuracy of the Institute for Scientific Information's journal impact factors, J. Am. Soc. Inf. Sci. 46, 461–467 (1995)
H.F. Moed, T.N. van Leeuwen: Impact factors can mislead, Nature 381, 186 (1996)
A.F.J. van Raan, T.N. van Leeuwen: Identifying the Fields for Mapping RTD Excellence in Life Sciences. First Approach (European Commission, Brussels 2001), Contract COPO-CT-2001-00001
Academic Ranking of World Universities: http://www.shanghairanking.com
Times Higher Education World University Rankings: https://www.timeshighereducation.com/world-university-rankings
L. Waltman, C. Calero-Medina, J. Kosten, E.C.M. Noyons, R.J.W. Tijssen, N.J. van Eck, T.N. van Leeuwen, A.F.J. van Raan, M.S. Visser, P. Wouters: The Leiden Ranking 2011/2012: data collection, indicators, and interpretation, J. Am. Soc. Inf. Sci. Technol. 63(12), 2419–2432 (2012)
Leiden Ranking 2016: http://www.leidenranking.com/information/indicators
QS Top Universities: https://www.topuniversities.com/university-rankings
Scimago Institutions Rankings: http://www.scimagoir.com
U-Multirank: http://www.umultirank.org
A.F.J. van Raan: Fatal Attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods, Scientometrics 62(1), 133–143 (2005)
J.E. Hirsch: An index to quantify an individual's scientific research output, Proc. Natl. Acad. Sci. USA 102(467), 16569–16572 (2005)
A.W. Harzing: The Publish or Perish Book. Your Guide to Effective and Responsible Citation Analysis (Tarma Software Research, Melbourne 2010)
A.W. Harzing: Publish or Perish, http://harzing.com/resources/publish-or-perish (2007)
H. Etzkowitz, L. Leydesdorff: The dynamics of innovation: From national systems and “Mode 2” to a triple helix of university-industry-government relations, Res. Policy 29, 109–123 (2000)
T.A. Brooks: Evidence of complex citer motivations, J. Am. Soc. Inf. Sci. 37, 34–36 (1986)
M.H. MacRoberts, B.R. MacRoberts: Author motivation for not giving citing influences—a methodological note, J. Am. Soc. Inf. Sci. 39, 432–433 (1988)
M.H. MacRoberts, B.R. MacRoberts: Problems of citation analysis, Scientometrics 36, 435–444 (1996)
P. Vinkler: Comparative investigation of frequency and strength of motives toward referencing, the reference threshold model—comments on theories of citation?, Scientometrics 43, 107–127 (1998)
L. Bornmann, H.D. Daniel: What do citation counts measure? A review of studies on citing behavior, J. Documentation 64(1), 45–80 (2008)
A.F.J. van Raan: In matters of quantitative studies of science the fault of theorists is offering too little and asking too much, Scientometrics 43, 129–139 (1998)
T.F. Frandsen, J. Nicolaisen: Citation behavior: A large-scale test of the persuasion by name-dropping hypothesis, J. Assoc. Inf. Sci. Technol. 65(2), 1278–1284 (2017)
A.J. Nederhof, A.F.J. van Raan: Peer-review and bibliometric indicators of scientific performance—A comparison of cum laude doctorates with ordinary doctorates in physics, Scientometrics 11(5-6), 333–350 (1987)
A.J. Nederhof, A.F.J. van Raan: A validation-study of bibliometric indicators—The comparative performance of cum laude doctorates in chemistry, Scientometrics 17(5-6), 427–435 (1989)
D.F. Horrobin: The philosophical basis of peer review and the suppression of innovation, J. Am. Med. Assoc. 263, 1438–1441 (1990)
C. Wennerås, A. Wold: Nepotism and sexism in peer-review, Nature 387, 341–343 (1997)
The Wellcome Trust: Women and Peer Review. An Audit of the Wellcome Trust Decision-Making on Grants (The Wellcome Trust/PRISM, London 1997)
H.W. Marsh, L. Bornmann, R. Mutz, H.D. Daniel, A. O'Mara: Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches, Rev. Educ. Res. 79(3), 1290–1326 (2009)
A.J. Nederhof: The validity and reliability of evaluation of scholarly performance. In: Handbook of Quantitative Studies of Science and Technology, ed. by A.F.J. van Raan (Elsevier/North-Holland, Amsterdam 1988) pp. 193–228
W. Glänzel: A bibliometric approach to social sciences, national research performances in 6 selected social science areas, 1990–1992, Scientometrics 35, 291–307 (1996)
D. Hicks: The difficulty of achieving full coverage of international social science literature and the bibliometric consequences, Scientometrics 44, 193–215 (1999)
H.F. Moed, M. Luwel, A.J. Nederhof: Towards research performance measurement in the humanities, Libr. Trends 50, 498–520 (2002)
L. Butler, M.S. Visser: Extending citation analysis to non-source items, Scientometrics 66(2), 327–343 (2006)
A.J. Nederhof: Bibliometric monitoring of research performance in the social sciences and the humanities: A review, Scientometrics 66(1), 81–100 (2006)
A.J. Nederhof, T.N. van Leeuwen, A.F.J. van Raan: Highly cited non-journal publications in political science, economics and psychology: A first exploration, Scientometrics 83(2), 363–374 (2010)
A. Linmans: Why with bibliometrics the humanities does not need to be the weakest link. Indicators for research evaluation based on citations, library holdings, and productivity measures, Scientometrics 83(2), 337–354 (2010)
L. Egghe, R. Rousseau: The influence of publication delays on the observed aging distribution of scientific literature, J. Am. Soc. Inf. Sci. 51, 158–165 (2000)
E. Garfield: Premature discovery or delayed recognition –why? In: Essays of an Information Scientist, Vol. 4 (1980) pp. 488–493
W. Glänzel, B. Schlemmer, B. Thijs: Better late than never? On the chance to become highly cited only beyond the standard time horizon, Scientometrics 58(3), 571–586 (2003)
H. Small, E. Greenlee: Citation context analysis of a co-citation cluster-recombinant DNA, Scientometrics 2(4), 277–301 (1980)
H. Small, E. Sweeney: Clustering the science citation index using co-citations, I: A comparison of methods, Scientometrics 7, 393–404 (1985)
H. Small, E. Sweeney, E. Greenlee: Clustering the science citation index using co-citations, II: Mapping science, Scientometrics 8, 321–340 (1985)
H. Small: Visualizing science by citation mapping, J. Am. Soc. Inf. Sci. 50, 799–813 (1999)
E. Noma: An improved method for analysing square scientometric transaction matrices, Scientometrics 4, 297–316 (1982)
Q. Zhou, L. Leydesdorff: The normalization of occurrence and co-occurrence matrices in bibliometrics using Cosine similarities and Ochiai coefficients, J. Assoc. Inf. Sci. Technol. 67(11), 2805–2814 (2016)
H.P.F. Peters, A.F.J. van Raan: Co-word-based science maps of chemical engineering, 1. Representations by direct multidimensional-scaling, Res. Policy 22(1), 23–45 (1993)
H.P.F. Peters, A.F.J. van Raan: Co-word-based science maps of chemical engineering, 2. Representations by combined clustering and multidimensional-scaling, Res. Policy 22(1), 47–71 (1993)
L. Waltman, N.J. van Eck: A new methodology for constructing a publication-level classification system of science, J. Am. Soc. Inf. Sci. Technol. 63(12), 2378–2392 (2012)
L. Waltman, N.J. van Eck: A smart local moving algorithm for large-scale modularity-based community detection, European Physical Journal B 86(11), 471 (2013)
N.J. van Eck, L. Waltman: CitNetExplorer: A new software tool for analyzing and visualizing citation networks, J. Informetrics 8(4), 802–823 (2014)
K.W. Boyack, R. Klavans: Creation of a highly detailed, dynamic, global model and map of science, J. Am. Soc. Inf. Sci. Technol. 65(4), 670–685 (2014)
CitNetExplorer: http://www.citnetexplorer.nl/home
N.J. van Eck, L. Waltman: Software survey: VOSviewer, a computer program for bibliometric mapping, Scientometrics 84(2), 523–538 (2010)
VOSviewer: http://vosviewer.com
L. Egghe: Theory and practise of the g-index, Scientometrics 69(1), 131–152 (2006)
K.W. Boyack, R. Klavans: Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?, J. Am. Soc. Inf. Sci. Technol. 61, 2389–2404 (2010)
M. Rosvall, D. Axelsson, C.T. Bergstrom: The map equation, Eur. Phys. J. ST 178, 13–23 (2009)
M. Rosvall, C.T. Bergstrom: Multilevel compression of random walks on networks reveals hierarchical organization in large integrated systems, PLoS ONE 6(4), e18209 (2011)
L. Subelj, N.J. van Eck, L. Waltman: Clustering scientific publications based on citation relations: A systematic comparison of different methods, PLoS ONE 11(4), e0154404 (2016)
D.J. de Solla Price: Toward a model for science indicators. In: Toward a Metric of Science: The Advent of Science Indicators, ed. by Y. Elkana, J. Lederberg, R.K. Merton, A. Thackray, H. Zuckerman (John Wiley, New York 1978)
J. Ziman: From parameters to portents—and back. In: Toward a Metric of Science: The Advent of Science Indicators, ed. by Y. Elkana, J. Lederberg, R.K. Merton, A. Thackray, H. Zuckerman (John Wiley, New York 1978)
VSNU: Assessment of Research Quality: Chemistry and Chemical Engineering (VSNU, Utrecht 2002)
Uppsala University: Quality and Renewal 2007: An Overall Evaluation of Research at Uppsala University (Uppsala University, Uppsala 2007)
M. Olensky, M. Schmidt, N.J. van Eck: Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to the Web of Science, J. Assoc. Inf. Sci. Technol. 67(10), 2550–2564 (2016)
L. Bornmann, L. Leydesdorff, R. Mutz: The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits, J. Informetr. 7(1), 158–165 (2013)
L. Waltman, N.J. van Eck: A systematic empirical comparison of different approaches for normalizing citation impact indicators, J. Informetr. 7(4), 833–849 (2013)
L. Waltman, N.J. van Eck, T.N. van Leeuwen, M.S. Visser, A.F.J. van Raan: Towards a new crown indicator: An empirical analysis, Scientometrics 87(3), 467–481 (2011)
L. Waltman, N.J. van Eck, T.N. van Leeuwen, M.S. Visser, A.F.J. van Raan: Towards a new crown indicator: Some theoretical considerations, J. Informetr. 5(1), 37–47 (2011)
T. Opthof, L. Leydesdorff: Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance, J. Informetr. 4(3), 423–430 (2010)
N.J. van Eck, L. Waltman, A.F.J. van Raan, R.J.M. Klautz, W.C. Peul: Citation analysis may severely underestimate the impact of clinical research as compared to basic research, PLoS ONE 8(4), e62395 (2013)
Q. Wang, L. Waltman: Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus, J. Informetr. 10, 347–364 (2016)
H.F. Moed: Measuring contextual citation impact of scientific journals, J. Informetr. 4(3), 265–27726a (2010)
L. Leydesdorff, T. Opthof: Scopus's Source Normalized Impact per Paper (SNIP) versus a journal impact factor based on fractional counting of citations, J. Am. Soc. Inf. Sci. Technol. 61(11), 2365–2369 (2010)
L. Leydesdorff: The revised SNIP indicator of Elsevier's Scopus, J. Informetr. 7(4), 859–860 (2013)
L. Waltman, N.J. van Eck: Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison, Scientometrics 96(3), 699–716 (2013)
L. Waltman: A review of the literature on citation impact indicators, J. Informetr. 10(2), 365–391 (2016)
G. Abramo, C.A. D'Angelo: A farewell to the MNCS and like size-independent indicators, J. Informetr. 10(2), 646–651 (2016)
G. Abramo, C.A. D'Angelo: A farewell to the MNCS and like size-independent indicators: Rejoinder, J. Informetr. 10(2), 679–683 (2016)
W. Glänzel, B. Thijs, K. Debackere: Productivity, performance, efficiency, impact—What do we measure anyway? Some comments on the paper “A farewell to the MNCS and like size-independent indicators” by Abramo and D'Angelo, J. Informetr. 10(2), 658–660 (2016)
G. Sivertsen: A welcome to methodological pragmatism, J. Informetr. 10(2), 664–666 (2016)
L. Bornmann, R. Haunschild: Efficiency of research performance and the glass researcher, J. Informetr. 10(2), 652–654 (2016)
R. Danell: Evaluating research organizations' contribution to science is not the same task as evaluating the performance of their scientists, J. Informetr. 10(2), 655–657 (2016)
M. Thelwall: Not dead, just resting: The practical value of per publication citation indicators, J. Informetr. 10(2), 667–670 (2016)
M. Zitt: Paving the way or pushing at open doors? A comment on Abramo and D'Angelo “Farewell to size-independent indicators”, J. Informetr. 10(2), 675–678 (2016)
J. Ruiz-Castillo: Research output indicators are not productivity indicators, J. Informetr. 10(2), 661–663 (2016)
L. Waltman, N.J. van Eck, M. Visser, P. Wouters: The elephant in the room: The problem of quantifying productivity in evaluative scientometrics, J. Informetr. 10(2), 671–674 (2016)
A.F.J. van Raan: Performance-related differences of bibliometric statistical properties of research groups: cumulative advantages and hierarchically layered networks, J. Am. Soc. Inf. Sci. Technol. 57(14), 1919–1935 (2006)
A.F.J. van Raan: Scaling rules in the science system: Influence of field-specific citation characteristics on the impact of research groups, J. Am. Soc. Inf. Sci. Technol. 59(4), 565–576 (2008)
L. Leydesdorff, T. Opthof: Normalization at the field level: Fractional counting of citations, J. Informetr. 4(4), 644–646 (2010)
L. Leydesdorff, L. Bornmann: How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science, J. Am. Soc. Inf. Sci. Technol. 62(2), 217–229 (2011)
J. Ruiz-Castillo, L. Waltman: Field-normalized citation impact indicators using algorithmically constructed classification systems of science, J. Informetr. 9(1), 102–117 (2015)
L. Waltman, N.J. van Eck: Field-normalized citation impact indicators and the choice of an appropriate counting method, J. Informetr. 9(4), 872–894 (2015)
R. van Noorden: Love thy lab neighbour, Nature 468, 1011 (2010)
K. Lee, J.S. Brownstein, R.G. Mills, I.S. Kohane: Does collocation inform the impact of collaboration?, PLoS ONE 5(12), e14279 (2010)
A. Gazni, C.R. Sugimoto, F. Didegah: Mapping world scientific collaboration: Authors, institutions, and countries, J. Am. Soc. Inf. Sci. Technol. 63(2), 323–335 (2012)
P. Zhou, R.J.W. Tijssen, L. Leydesdorff: University-industry collaboration in China and the USA: A bibliometric comparison, PLoS ONE 11(11), e0165277 (2016)
A.F.J. van Raan: On growth, ageing, and fractal differentiation of science, Scientometrics 47(2), 347–362 (2000)
D. Hicks, P. Wouters, L. Waltman, S. de Rijcke, I. Rafols: The Leiden Manifesto for research metrics, Nature 520(7548), 429–431 (2015)
A.F.J. van Raan: Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises, Scientometrics 36, 397–420 (1996)
A.F.J. van Raan: Scientometrics: State-of-the-Art, Scientometrics 38, 205–218 (1997)
Wikipedia: Retraction Watch, https://en.wikipedia.org/wiki/Retraction_Watch
J.A. Byrne, C. Labbé: Striking similarities between publications from China describing single gene knockdown experiments in human cancer cell lines, Scientometrics 110(3), 1471–1493 (2017)
T.N. van Leeuwen, M. Luwel: An in-depth analysis of papers retracted in the Web of Science. In: Proc. 19th Int. Conf. Sci. Technol. Indic., Leiden (2014) pp. 337–344
A.F.J. van Raan, T.N. van Leeuwen, M.S. Visser: Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings, Scientometrics 88(2), 495–498 (2011)
A.F.J. van Raan, T.N. van Leeuwen, M.S. Visser: Non-English papers decrease rankings, Nature 469(7328), 34 (2011)
A.F.J. van Raan: Sleeping beauties in science, Scientometrics 59(3), 461–466 (2004)
T. Braun, W. Glänzel, A. Schubert: On sleeping beauties, princes and other tales of citation distributions, Res. Eval. 19(3), 195–202 (2010)
C. Lachance, V. Larivière: On the citation lifecycle of papers with delayed recognition, J. Informetr. 8, 863–872 (2014)
A.F.J. van Raan: Dormitory of physical and engineering sciences: Sleeping beauties may be sleeping innovations, PLoS ONE 10(10), e0139786 (2015)
P. Gorry, P. Ragouet: “Sleeping beauty” and her restless sleep: Charles Dotter and the birth of interventional radiology, Scientometrics 107(2), 773–784 (2016)
A.F.J. van Raan: Sleeping beauties cited in patents: Is there also a dormitory of inventions?, Scientometrics 110(3), 1123–1156 (2017)
Q.L. Burrell: Are “sleeping beauties” to be expected?, Scientometrics 65(3), 381–389 (2005)
J. Wang, F. Ma, M. Chen, Y. Rao: Why and how can sleeping beauties be awakened?, Electron. Libr. 30(1), 5–18 (2012)
B. van Calster: It takes time: a remarkable example of delayed recognition, J. Am. Soc. Inf. Sci. Technol. 63(11), 2341–2344 (2012)
J. Li, F.Y. Ye: The phenomenon of all-elements-sleeping-beauties in scientific literature, Scientometrics 92(3), 795–799 (2012)
J. Li, D. Shi, S.X. Zhao, F.Y. Ye: A study of the “heartbeat spectra” for “sleeping beauties”, J. Informetr. 8, 493–502 (2014)
J. Li: Citation curves of “all-elements-sleeping-beauties”: “flash in the pan” first and then “delayed recognition”, Scientometrics 100(2), 595–601 (2014)
G.A. Ronda-Pupo, J.S. Katz: The scaling relationship between citation-based performance and coauthorship patterns in natural sciences, J. Assoc. Inf. Sci. Technol. 68(5), 1257–1265 (2017)
T.N. van Leeuwen: Discussing some basic critique on journal impact factors: Revision of earlier comments, Scientometrics 92(2), 443–455 (2012)
A.F.J. van Raan: Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups, Scientometrics 67(3), 491–502 (2006)
L. Bornmann, R. Mutz, H.D. Daniel: Are there better indices for evaluation purposes than the h-index? A comparison of nine different variants of the h-index using data from biomedicine, J. Am. Soc. Inf. Sci. Technol. 59(5), 830–837 (2008)
L. Waltman, N.J. Van Eck: The inconsistency of the h-index, J. Am. Soc. Inf. Sci. Technol. 63(2), 406–415 (2012)
E.J. Rinia, T.N. Van Leeuwen, H.G. van Vuren, A.F.J. van Raan: Comparative analysis of a set of bibliometric indicators and central peer review criteria. Evaluation of condensed matter physics in the Netherlands, Res. Policy 27, 95–107 (1998)
E.J. Rinia, T.N. van Leeuwen, H.G. van Vuren, A.F.J. van Raan: Influence of interdisciplinarity on peer-review and bibliometric evaluations, Res. Policy 30, 357–361 (2001)
H.F. Moed: Citation Analysis in Research Evaluation (Springer, Dordrecht 2005) pp. 229–257
L. Bornmann, H.D. Daniel: Selecting scientific excellence through committee peer review—A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants, Scientometrics 68(3), 427–440 (2006)
Y. Gargouri, C. Hajjem, V. Larivière, Y. Gingras, L. Carr, T. Brody, S. Harnad: Self-selected or mandated, open access increases citation impact for higher quality research, PLoS ONE 5(10), e13636 (2010)
Dimensions database: https://app.dimensions.ai/discover/publication
H.F. Moed: The effect of “Open access” on citation impact: An analysis of ArXiv's condensed matter section, J. Am. Soc. Inf. Sci. Technol. 58(13), 2047–2054 (2007)
A. Kastberg: Is open access really good for science?, Europhys. News 45(2), 32 (2014)
L. Björneborn, P. Ingwersen: Perspectives of webometrics, Scientometrics 50, 65–82 (2001)
M. Thelwall, A. Smith: Interlinking between Asia-Pacific University Web sites, Scientometrics 55, 363–376 (2002)
M. Thelwall, G. Harries: The connection between the research of a university and counts of links to its web pages: An investigation based upon a classification of the relationships of pages to the research of the host university, J. Am. Soc. Inf. Sci. 54, 594–602 (2003)
I.F. Aguillo: Building web indicators for the EU OA repository. In: Workshop on New Research Lines in Informetrics, IPP-CCHS (CSIC), Madrid (2011), http://digital.csic.es/bitstream/10261/40279/1/OpenAIRE%20Webometrics.pdf
J. Priem, D. Taraborelli, P. Groth, C. Neylon: Altmetrics: A manifesto, http://altmetrics.org/manifesto/ (2010)
J. Priem, P. Groth, D. Taraborelli: The altmetrics collection, PLoS ONE 7(11), e48753 (2012)
J. Bar-Ilan: Data collection methods on the web for informetric purposes—A review and analysis, Scientometrics 50, 7–32 (2001)
C. Schlögl, J. Gorraiz: Global usage versus global citation metrics: The case of pharmacology journals, J. Am. Soc. Inf. Sci. Technol. 62(1), 161–170 (2011)
C. Schlögl, J. Gorraiz, C. Gumpenberger, K. Jack, P. Kraker: Comparison of downloads, citations and readership data for two information systems journals, Scientometrics 101(2), 1113–1128 (2014)
J. Bar-Ilan, S. Haustein, I. Peters, J. Priem, H. Shema, J. Terliesner: Beyond citations: Scholars' visibility on the social web. In: Proc. 17th Int. Conf. Sci. Technol. Indic., Montréal (2012) pp. 98–109
H. Shema, J. Bar-Ilan, M. Thelwall: Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics, J. Assoc. Inf. Sci. Technol. 65(5), 1018–1027 (2014)
M. Thelwall, S. Haustein, V. Larivière, C. Sugimoto: Do altmetrics work? Twitter and ten other candidates, PLoS ONE 8(5), e64841 (2013)
Research Trends: Special issue on alternative metrics, Issue 37, June (2014)
R. Costas, Z. Zahedi, P. Wouters: Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective, J. Assoc. Inf. Sci. Technol. 66(10), 2003–2019 (2014)
P. Sud, M. Thelwall: Evaluating altmetrics, Scientometrics 98(2), 1131–1143 (2014)
C. Chorus, L. Waltman: A large-scale analysis of impact factor biased journal self-citations, PLoS ONE 11(8), e0161021 (2016)
Wikipedia: San Francisco Declaration on Research Assessment, https://en.wikipedia.org/wiki/San_Francisco_Declaration_on_Research_Assessment
L. Waltman, V.A. Traag: Use of the journal impact factor for assessing individual articles need not be wrong, https://arxiv.org/abs/1703.02334 (2017)
M. Zitt, H. Small: Modifying the journal impact factor by fractional citation weighting: The audience factor, J. Am. Soc. Inf. Sci. Technol. 59(11), 1856–1860 (2008)
L. Waltman, N.J. van Eck: The relation between Eigenfactor, audience factor, and influence weight, J. Am. Soc. Inf. Sci. Technol. 61(7), 1476–1486 (2010)
CWTS Journal Indicators: http://www.journalindicators.com/
V.P. Guerrero-Botea, F. Moya-Anegón: A further step forward in measuring journals' scientific prestige: The SJR2 indicator, J. Informetr. 6, 674–688 (2012)
Scimago journal & country rank: http://www.scimagojr.com/journalrank.php
Elsevier: CiteScore, https://www.elsevier.com/editors-update/story/journal-metrics/citescore-a-new-metric-to-help-you-choose-the-right-journal
F. Franceschini, D. Maisano, L. Mastrogiacomo: The museum of errors/horrors in Scopus, J. Informetr. 10(1), 174–182 (2016)
W.J.N. Meester, L. Colledge, E.E. Dyas: A response to “The museum of errors/horrors in Scopus” by Franceschini et al, J. Informetr. 10(1), 569–570 (2016)
S. Brin, L. Page: The anatomy of a large-scale hypertextual web search engine, Comput. Netw. ISDN Syst. 30(1–7), 107–117 (1998)
C.T. Bergstrom: Eigenfactor: Measuring the value and prestige of scholarly journals, Coll. Res. Libr. News 68(5), 314–316 (2007)
B. Zogala-Siudem, G. Siudem, A. Cena, M. Gagolewski: Agent-based model for the h-index—exact solution, Eur. Phys. J. B 89, 21 (2016)
Research Policy Team HEFCE: REF Independent review of the role of metrics in research assessment. Summary of responses submitted to the call for evidence, http://dera.ioe.ac.uk/21370/2/Summary%20of%20responses%20submitted%20to%20the%20call%20for%20evidence.pdf (Oct. 2014)
H.F. Moed, J. Bar-Ilan, G. Halevi: A new methodology for comparing Google Scholar and Scopus, J. Informetr. 10(2), 533–551 (2016)
R. van. Noorden: Science research engine links papers to grants and patents. The Dimensions database promises a financial perspective on scholarly literature, https://www.nature.com/articles/d41586-018-00688-0 (16 Jan. 2018)
D.M. Nichols, M.B. Twidale: Metrics for openness, J. Assoc. Inf. Sci. Technol. 68(4), 1048–1060 (2016)
Webometrics Ranking of World Universities: http://www.webometrics.info/en
M. Thelwall, K. Kousha: ResearchGate articles: Age, discipline, audience size, and impact, J. Assoc. Inf. Sci. Technol. 68(2), 468–479 (2017)
PLoS ONE: Altmetrics, http://journals.plos.org/plosone/browse/altmetrics
Oxford University Press: https://academic.oup.com/journals/pages/authors/article_level_metrics
Altmetrics: http://altmetrics.org
Impactstory: https://profiles.impactstory.org/
R. Haunschild, L. Bornmann: Normalization of Mendeley reader counts for impact assessment, J. Informetr. 10(1), 62–73 (2016)
M. Thelwall, P. Sud: Mendeley readership counts: An investigation of temporal and disciplinary differences, J. Assoc. Inf. Sci. Technol. 67(12), 3036–3050 (2016)
National Information Standards Organization (NISO): NISO Alternative Assessment Metrics (Altmetrics) Initiative, https://www.niso.org/standards-committees/altmetrics
The Scientist: Top 10 Retractions of 2016, http://www.the-scientist.com/?articles.view/articleNo/47813/title/Top-10-Retractions-of-2016/
H.F. Moed: A critical comparative analysis of five world university rankings, Scientometrics 110(2), 967–990 (2017)
M. Dobrota, M. Bulajic, L. Bornmann, V. Jeremic: A new approach to the QS university ranking using the composite i-distance indicator: Uncertainty and sensitivity analyses, J. Assoc. Inf. Sci. Technol. 67(1), 200–211 (2016)
A. Perianes-Rodriguez, J. Ruiz-Castillo: University citation distributions, J. Assoc. Inf. Sci. Technol. 67(11), 2790–2804 (2016)
C. Daraio, A. Bonaccorsi: Beyond university rankings? Generating new indicators on universities by linking data in open platforms, J. Assoc. Inf. Sci. Technol. 68(2), 508–529 (2017)
Committee on Publication Ethics (COPE): Promoting integrity in research and its publication, http://publicationethics.org/
E. Delgado Lopez-Cozar, N. Robinson-Garcia, D. Torres-Salinas: The Google Scholar experiment: How to index false papers and manipulate bibliometric indicators, J. Assoc. Inf. Sci. Technol. 65(3), 446–454 (2014)
L. Butler: Modifying publication practices in response to funding formulas, Res. Eval. 17, 39–46 (2003)
S. De Rijcke, P.F. Wouters, A.D. Rushforth, T.P. Franssen, B. Hammarfelt: Evaluation practices and effects of indicator use—A literature review, Res. Eval. 25(2), 161–169 (2016)
V. Larivière, R. Costas: How many is too many? On the relationship between research productivity and impact, PLoS ONE 11(9), e0162709 (2016)
N. Caplar, S. Tacchella, S. Birrer: Quantitative evaluation of gender bias in astronomical publications from citation counts, Nat. Astron. 1(6), 0141 (2017)
J. Wilsdon, L. Allen, E. Belfiore, P. Campbell, S. Curry, S. Hill, R. Jones, R. Kain, S. Kerridge, M. Thelwall, J. Tinkler, I. Viney, P. Wouters, J. Hill, B. Johnson: The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management (HEFCE, London 2015)
A. Ancaiani, A.F. Anfossi, A. Barbara, S. Benedetto, B. Blasi, V. Carletti, T. Cicero, A. Ciolfi, F. Costa, G. Colizza, M. Costantini, F. di Cristina, A. Ferrara, R.M. Lacatena, M. Malgarini, I. Mazzotta, C.A. Nappi, S. Romagnosi, S. Sileoni: Evaluating scientific research in Italy: The 2004–10 research evaluation exercise, Res. Eval. 24, 242–255 (2015)
A. Molinié, G. Bodenhausen: Bibliometrics as weapons of mass citation, Chimia 64, 78–89 (2010)
R.R. Ernst: The follies of citation indices and academic ranking lists. A brief commentray to ‘Bibliometrics as weapons of mass citation', Chimia 64, 90 (2010)
Wikipedia: Ludolph van Ceulen, https://en.wikipedia.org/wiki/Ludolph_van_Ceulen
E. Noyons, C. Calero Medina: Results of the bibliometric study on the Faculty of Veterinary Medicine of the Utrecht University 2001-2011, CWTS Report April 2012, Center for Science Technol. Studies (CWTS) (Leiden University, Leiden 2012), https://www.uu.nl/en/organisation/faculty-of-veterinary-medicine/veterinary-research/organisation/assessment-of-research-quality
R. Costas, T.N. van Leeuwen, A.F.J. van Raan: Is scientific literature subject to a 'sell-by-date'? A general methodology to analyze the 'durability' of scientific documents, J. Am. Soc. Inf. Sci. Technol. 61(2), 329–339 (2010)
R. Costas, T.N. van Leeuwen, A.F.J. van Raan: The “Mendel syndrome” in science: Durability of scientific literature and its effects on bibliometric analysis of individual scientists, Scientometrics 89(1), 177–205 (2011)
R. Costas, T.N. van Leeuwen, A.F.J. van Raan: Effects of the durability of scientific literature at the group level: Case study of chemistry research groups in the Netherlands, Res. Policy 42(4), 886–894 (2013)
A.F.J. van Raan, T.N. van Leeuwen: Assessment of the scientific basis of interdisciplinary, applied research: Application of bibliometric methods in nutrition and food research, Res. Policy 31, 611–632 (2002)
CWTS B.V.: Monitoring & Evaluation, Leiden University, http://www.cwtsbv.nl/Monitoring-Evaluation
M. Thelwall, P. Sud: National, disciplinary and temporal variations in the extent to which articles with more authors have more impact: Evidence from a geometric field normalised citation indicator, J. Informetr. 10(1), 48–61 (2016)
W. Glänzel, M. Meyer: Patents cited in the scientific literature: An exploratory study of ‘reverse' citation relations, Scientometrics 58, 415–428 (2003)
A.F.J. van Raan: Patent citations analysis and its value in research evaluation: A review and a new approach to map technology-relevant research, J. Data Inf. Sci. 2(1), 13–50 (2017)
B. Uzzi, S. Mukherjee, M. Stringer, B. Jones: Atypical combinations and scientific impact, Science 342, 468–471 (2013)
A. Thor, W. Marx, L. Leydesdorff, L. Bornmann: Introducing CitedReferencesExplorer (CRExplorer): A program for reference publication year spectroscopy with cited references standardization, J. Informetr. 10(2), 503–515 (2016)
G. Chen, L. Xiao: Selecting publication keywords for domain analysis in bibliometrics: A comparison of three methods, J. Informetr. 10(1), 212–213 (2016)
K.H. Lee, S.Y. Kim, H.J. Kim, M. Song: Comparative evaluation of bibliometric content networks by tomographic content analysis: An application to Parkinson's disease, J. Assoc. Inf. Sci. Technol. (2017), https://doi.org/10.1002/asi.23752
L. Leydesdorff, A. Nerghes: Co-word maps and topic modeling: A comparison using small and medium-sized corpora (N < 1000), J. Assoc. Inf. Sci. Technol. 68(4), 1024–1035 (2017)
A. Suominen, H. Toivanen: Map of science with topic modeling: Comparison of unsupervised learning and human-assigned subject classification, J. Assoc. Inf. Sci. Technol. 67(10), 2464–2476 (2016)
B. Wen, E. Horlings, M. van der Zouwen, P. van den Besselaar: Mapping science through bibliometric triangulation: An experimental approach applied to water research, J. Assoc. Inf. Sci. Technol. 68(3), 724–738 (2017)
E.C.M. Noyons, A.F.J. van Raan: Monitoring scientific developments from a dynamic perspective: Self-organized structuring to map neural network research, J. Am. Soc. Inf. Sci. 49(1), 68–81 (1998)
L. Waltman, A.F.J. van Raan, S. Smart: Exploring the relationship between the engineering and physical sciences and the health and life sciences by advanced bibliometric methods, PLoS ONE 9(10), e111530 (2014)
L. Leydesdorff, I. Rafols: A global map of science based on the ISI subject categories, J. Am. Soc. Inf. Sci. Technol. 60(2), 348–362 (2009)
L. Leydesdorff, S. Carley, I. Rafols: Global maps of science based on the new Web-of-Science categories, Scientometrics 94(2), 589–593 (2013)
H. Small, K.W. Boyack, R. Klavans: Identifying emerging topics in science and technology, Res. Policy 43, 1450–1467 (2014)
D. Zhao, A. Strotmann: Can citation analysis of web publications better detect research fronts?, J. Am. Soc. Inf. Sci. Technol. 58(9), 1285–1302 (2007)
S. Jensen, X. Liub, Y. Yuc, S. Milojevic: Generation of topic evolution trees from heterogeneous bibliographic networks, J. Informetr. 10, 606–621 (2016)
G. Holton: Can science be measured? In: Toward a metric of science: the advent of science indicators, ed. by Y. Elkana, J. Lederberg, R.K. Merton, A. Thackray, H. Zuckerman (John Wiley, New York 1978)
Acknowledgements
The author thanks Nees-Jan van Eck for the construction of the publication-level citation-based network map and the citation density neurology map. The text parts on the methodology of the Leiden Ranking, the CitNetExplorer, and the VOSviewer are largely based on the descriptions in the relevant CWTS webpages by Ludo Waltman and Nees-Jan van Eck.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
van Raan, A. (2019). Measuring Science: Basic Principles and Application of Advanced Bibliometrics. In: Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M. (eds) Springer Handbook of Science and Technology Indicators. Springer Handbooks. Springer, Cham. https://doi.org/10.1007/978-3-030-02511-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-02511-3_10
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-02510-6
Online ISBN: 978-3-030-02511-3
eBook Packages: Economics and FinanceEconomics and Finance (R0)