research-article

Quantifying Engagement with Citations on Wikipedia

Authors:

Tiziano Piccardi,

Giovanni Colavizza,

Robert WestAuthors Info & Claims

WWW '20: Proceedings of The Web Conference 2020

Pages 2365 - 2376

https://doi.org/10.1145/3366423.3380300

Published: 20 April 2020 Publication History

Abstract

Wikipedia is one of the most visited sites on the Web and a common source of information for many users. As an encyclopedia, Wikipedia was not conceived as a source of original information, but as a gateway to secondary sources: according to Wikipedia’s guidelines, facts must be backed up by reliable sources that reflect the full spectrum of views on the topic. Although citations lie at the heart of Wikipedia, little is known about how users interact with them. To close this gap, we built client-side instrumentation for logging all interactions with links leading from English Wikipedia articles to cited references during one month, and conducted the first analysis of readers’ interactions with citations. We find that overall engagement with citations is low: about one in 300 page views results in a reference click (0.29% overall; 0.56% on desktop; 0.13% on mobile). Matched observational studies of the factors associated with reference clicking reveal that clicks occur more frequently on shorter pages and on pages of lower quality, suggesting that references are consulted more commonly when Wikipedia itself does not contain the information sought by the user. Moreover, we observe that recent content, open access sources, and references about life events (births, deaths, marriages, etc.) are particularly popular. Taken together, our findings deepen our understanding of Wikipedia’s role in a global information economy where reliability is ever less certain, and source attribution ever more vital.

References

[1]

B. Thomas Adler, Krishnendu Chatterjee, Luca de Alfaro, Marco Faella, Ian Pye, and Vishwanath Raman. 2008. Assigning trust to Wikipedia content. In Proc. 4th International Symposium on Wikis.

Digital Library

[2]

Ofer Arazy, Hila Liifshitz-Assaf, Oded Nov, Johannes Daxenberger, Martina Balestra, and Coye Cheshire. 2017. On the ”How” and ”Why” of emergent role behaviors in Wikipedia. In Proc. ACM Conference on Computer Supported Cooperative Work and Social Computing.

[3]

Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proc. ACM Conference on Computer Supported Cooperative Work and Social Computing.

Digital Library

[4]

Peter C Austin. 2011. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research 46, 3 (2011), 399–424.

[5]

Saeideh Bakhshi, David A Shamma, and Eric Gilbert. 2014. Faces engage us: Photos with faces attract more likes and comments on instagram. In Proc. SIGCHI conference on human factors in computing systems.

Digital Library

[6]

Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. 2016. Improving post-click user engagement on native ads via survival analysis. In Proc. International Conference on World Wide Web.

Digital Library

[7]

Ivan Beschastnikh. 2008. Wikipedian Self-Governance in action: Motivating the policy lens. In Proc. International AAAI Conference on Web and Social Media.

[8]

Xiaoxi Chelsy Xie, Isaac Johnson, and Anne Gomez. 2019. Detecting and gauging impact on Wikipedia page views. In Proc. International Conference on World Wide Web.

Digital Library

[9]

Chih-Chun Chen and Camille Roth. 2012. {{Citation needed}}: the dynamics of referencing in Wikipedia. In Proc. Annual International Symposium on Wikis and Open Collaboration.

Digital Library

[10]

Jörg Claussen, Tobias Kretschmer, and Philip Mayrhofer. 2013. The effects of rewarding user engagement: The case of Facebook apps. Information Systems Research 24, 1 (2013), 186–200.

Digital Library

[11]

William Cronon. 2012. Scholarly authority in a Wikified world. Perspectives in History(2012).

[12]

Alexander Dallmann, Thomas Niebler, Florian Lemmerich, and Andreas Hotho. 2016. Extracting semantics from random walks on Wikipedia: comparing learning and counting methods. In Proc. The Workshops of the Tenth International AAAI Conference on Web and Social Media.

[13]

Dimitar Dimitrov and Florian Lemmerich. 2019. Different topic, different traffic: How search and navigation interplay on Wikipedia. The Journal of Web Science 6 (2019).

[14]

Ethan Fast, Binbin Chen, and Michael S. Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proc. CHI Conference on Human Factors in Computing Systems.

Digital Library

[15]

Besnik Fetahu, Katja Markert, Wolfgang Nejdl, and Avishek Anand. 2016. Finding news citations for Wikipedia. In Proc. Conference on Information and Knowledge Management.

Digital Library

[16]

Andrea Forte, Vanesa Larco, and Amy Bruckman. 2009. Decentralization in Wikipedia governance. Journal of Management Information Systems 26, 1 (2009), 49–72.

Digital Library

[17]

R. Stuart Geiger and Aaron Halfaker. 2013. When the levee breaks: without bots, what happens to Wikipedia’s quality control processes?. In Proc. International Symposium on Open Collaboration.

[18]

Patrick Gildersleve and Taha Yasseri. 2018. Inspiration, captivation, and misdirection: Emergent properties in networks of online navigation. In Complex Networks IX. 271–282.

[19]

Casper Grathwohl. 2011. Wikipedia comes of age. Chronicle of Higher Education 57 (2011).

[20]

Aaron Halfaker. 2017. Interpolating quality dynamics in Wikipedia and demonstrating the Keilana effect. In Proc. International Symposium on Open Collaboration.

Digital Library

[21]

Denis Helic, Markus Strohmaier, Michael Granitzer, and Reinhold Scherer. 2013. Models of human navigation in information networks based on decentralized search. In Proc. ACM Conference on Hypertext and Social Media.

Digital Library

[22]

Yuheng Hu, Shelly Farnham, and Kartik Talamadupula. 2015. Predicting user engagement on twitter with real-world events. In Proc. International AAAI Conference on Web and Social Media.

[23]

Dariusz Jemielniak and Eduard Aibar. 2016. Bridging the gap between Wikipedia and academia. Journal of the Association for Information Science and Technology 67, 7(2016), 1773–1776.

Digital Library

[24]

Yushi Jing, David Liu, Dmitry Kislyuk, Andrew Zhai, Jiajing Xu, Jeff Donahue, and Sarah Tavel. 2015. Visual search at Pinterest. In Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Digital Library

[25]

Brian Keegan, Darren Gergle, and Noshir Contractor. 2011. Hot off the wiki: dynamics, practices, and structures in Wikipedia’s coverage of the Tōhoku catastrophes. In Proc. WikiSym - International Symposium on Wikis and Open Collaboration.

Digital Library

[26]

Tobias Koopmann, Alexander Dallmann, Lena Hettinger, Thomas Niebler, and Andreas Hotho. 2019. On the right track! Analysing and predicting navigation success in Wikipedia. In Proc. ACM Conference on Hypertext and Social Media.

Digital Library

[27]

Kayvan Kousha and Mike Thelwall. 2017. Are Wikipedia citations important evidence of the impact of scholarly articles and books?Journal of the Association for Information Science and Technology 68, 3(2017), 762–779.

[28]

Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the Web: Impact, characteristics, and detection of Wikipedia hoaxes. In Proc. International Conference on World Wide Web.

Digital Library

[29]

Daniel Lamprecht, Dimitar Dimitrov, Denis Helic, and Markus Strohmaier. 2016. Evaluating and improving navigability of Wikipedia: a comparative study of eight language editions. In Proc. International Symposium on Open Collaboration.

Digital Library

[30]

Daniel Lamprecht, Kristina Lerman, Denis Helic, and Markus Strohmaier. 2017. How the structure of Wikipedia articles influences user navigation. New Review of Hypermedia and Multimedia 23, 1 (2017), 29–50.

Digital Library

[31]

Janette Lehmann, Claudia Müller-Birn, David Laniado, Mounia Lalmas, and Andreas Kaltenbrunner. 2014. Reader preferences and behavior on Wikipedia. In Proceedings of the 25th ACM conference on Hypertext and social media.

Digital Library

[32]

Florian Lemmerich, Diego Sáez-Trumper, Robert West, and Leila Zia. 2019. Why the world reads Wikipedia: beyond English speakers. In Proc. ACM International Conference on Web Search and Data Mining.

Digital Library

[33]

Włodzimierz Lewoniewski, Krzysztof Węcel, and Witold Abramowicz. 2017. Analysis of references across Wikipedia languages. Information and Software Technologies 756 (2017), 561–573.

[34]

Lauren A Maggio, John M Willinsky, Ryan M Steinberg, Daniel Mietchen, Joseph L Wass, and Ting Dong. 2019. Wikipedia as a gateway to biomedical research: The relative distribution and use of citations in the English Wikipedia. PLOS ONE 12, 12 (2019).

[35]

Mostafa Mesgari, Chitu Okoli, Mohamad Mehdi, Finn Nielsen, and Arto Lanamäki. 2015. “The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia. Journal of the Association for Information Science and Technology 66, 2(2015), 219–245.

Digital Library

[36]

Marc Miquel-Ribé. 2015. User engagement on Wikipedia, a review of studies of readers and rditors. In Proc. International AAAI Conference on Web and Social Media.

[37]

Helen Susannah Moat, Chester Curme, Adam Avakian, Dror Y. Kenett, H. Eugene Stanley, and Tobias Preis. 2013. Quantifying Wikipedia usage patterns before stock market moves. Scientific Reports 3, 1 (2013).

[38]

Finn Årup Nielsen. 2007. Scientific Citations in Wikipedia. First Monday 12(2007).

[39]

Finn Årup Nielsen. 2012. Wikipedia Research and Tools: Review and Comments. Social Science Research Network (SSRN) - Electronic Journal (2012).

[40]

Finn Årup Nielsen, Daniel Mietchen, and Egon Willighagen. 2017. Scholia, Scientometrics and Wikidata. In The Semantic Web: ESWC 2017 Satellite Events. Vol. 10577. 237–259.

[41]

Nov Oded. 2007. What motivates Wikipedians?Commun. ACM 50, 11 (2007), 60–64.

Digital Library

[42]

Ashwin Paranjape, Robert West, Leila Zia, and Jure Leskovec. 2016. Improving website hyperlink structure using server logs. In Proc. International Conference on Web Search and Data Mining.

Digital Library

[43]

Svitlana Petrasova, Nina Khairova, Włodzimierz Lewoniewski, Orken Mamyrbayev, and Kuralay Mukhsina. 2018. Similar text fragments extraction for identifying common Wikipedia Communities. Data 3, 4 (2018).

[44]

Tiziano Piccardi, Michele Catasta, Leila Zia, and Robert West. 2018. Structuring Wikipedia articles with section recommendations. In Proc. ACM SIGIR Conference on Research & Development in Information Retrieval.

Digital Library

[45]

Alessandro Piscopo and Elena Simperl. 2019. What we talk about when we talk about Wikidata quality: a literature survey. In Proc. International Symposium on Open Collaboration.

Digital Library

[46]

Reid Priedhorsky, Jilin Chen, Shyong Tony K. Lam, Katherine Panciera, Loren Terveen, and John Riedl. 2007. Creating, destroying, and restoring value in wikipedia. In Proc. ACM Conference on supporting group work.

Digital Library

[47]

Jacob Ratkiewicz, Santo Fortunato, Alessandro Flammini, Filippo Menczer, and Alessandro Vespignani. 2010. Characterizing and Modeling the Dynamics of Online Popularity. Physical Review Letters 105, 15 (2010).

[48]

Miriam Redi, Besnik Fetahu, Jonathan Morgan, and Dario Taraborelli. 2019. Citation Needed: A Taxonomy and algorithmic assessment of Wikipedia’s verifiability. In Proc. International Conference on World Wide Web.

Digital Library

[49]

Flavia Salutari, Diego Da Hora, Gilles Dubuc, and Dario Rossi. 2019. A Large-scale study of Wikipedia users’ quality of experience. In Proc. International Conference on World Wide Web.

Digital Library

[50]

Aju Thalappillil Scaria, Rose Marie Philip, Robert West, and Jure Leskovec. 2014. The last click: why users give up information network navigation. In Proc. International Conference on Web Search and Data Mining.

Digital Library

[51]

Thomas Shafee, Gwinyai Masukume, Lisa Kipersztok, Diptanshu Das, Mikael Häggström, and James Heilman. 2017. Evolution of Wikipedia’s medical content: past, present and future. Journal of Epidemiology and Community Health (Aug. 2017), jech–2016–208601.

[52]

Philipp Singer, Denis Helic, Behnam Taraghi, and Markus Strohmaier. 2014. Detecting memory and structure in human navigation patterns using Markov chain models of varying order. PLoS ONE 9, 7 (2014), e102070.

[53]

Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, Markus Strohmaier, and Jure Leskovec. 2017. Why we read Wikipedia. In Proc. International Conference on World Wide Web.

Digital Library

[54]

Philipp Singer, Thomas Niebler, Markus Strohmaier, and Andreas Hotho. 2013. Computing semantic relatedness from human navigational paths: A case study on Wikipedia. International Journal on Semantic Web and Information Systems 9, 4(2013), 41–70.

Digital Library

[55]

Yang Song, Xiaolin Shi, and Xin Fu. 2013. Evaluating and predicting user engagement change with degraded search relevance. In Proc. International Conference on World Wide Web.

Digital Library

[56]

A. Spoerri. 2007. What is popular on Wikipedia and why?First Monday 12, 4 (2007). https://firstmonday.org/ojs/index.php/fm/article/view/1765/1645

[57]

Nathan TeBlunthuis, Tilman Bayer, and Olga Vasileva. 2019. Dwelling on Wikipedia: investigating time spent by global encyclopedia readers. In Proc. International Symposium on Open Collaboration.

Digital Library

[58]

Misha Teplitskiy, Grace Lu, and Eamon Duede. 2017. Amplifying the impact of open access: Wikipedia and the diffusion of science. Journal of the Association for Information Science and Technology 68, 9(2017), 2116–2127.

Digital Library

[59]

Neil Thompson and Douglas Hanley. 2018. Science is shaped by Wikipedia: evidence from a randomized control trial. MIT Sloan Research Paper 5238, 17 (2018).

[60]

Robert Tomaszewski and Karen I. MacDonald. 2016. A Study of citations to Wikipedia in scholarly publications. Science & Technology Libraries 35, 3 (2016), 246–261.

[61]

Daniel Torres-Salinas, Esteban Romero-Frías, and Wenceslao Arroyo-Machado. 2019. Mapping the backbone of the humanities through the eyes of Wikipedia. Journal of Informetrics 13, 3 (2019), 793–803.

[62]

Christoph Trattner, Denis Helic, Philipp Singer, and Markus Strohmaier. 2012. Exploring the differences and similarities between hierarchical decentralized search and human navigation in information networks. In Proc. International Conference on Knowledge Management and Knowledge Technologies.

Digital Library

[63]

Vivienne Waller. 2011. The search queries that took Australian Internet users to Wikipedia. Information Research 16, 3 (2011).

[64]

Robert West and Jure Leskovec. 2012. Automatic versus human navigation in information networks. In International AAAI Conference on Web and Social Media.

[65]

Robert West and Jure Leskovec. 2012. Human wayfinding in information networks. In Proc. International Conference on World Wide Web. Lyon, France.

Digital Library

[66]

Robert West, Ashwin Paranjape, and Jure Leskovec. 2015. Mining missing hyperlinks from human navigation traces: A case study of Wikipedia. In Proc. International Conference on World Wide Web.

Digital Library

[67]

Ellery Wulczyn, Robert West, Leila Zia, and Jure Leskovec. 2016. Growing Wikipedia across languages via recommendation. In Proc. International Conference on World Wide Web.

Digital Library

[68]

Taha Yasseri, Robert Sumi, András Rung, András Kornai, and János Kertész. 2012. Dynamics of conflicts in Wikipedia. PLoS ONE 7, 6 (June 2012), e38869.

[69]

Ramtin Yazdanian, Leila Zia, Jonathan Morgan, Bahodir Mansurov, and Robert West. 2019. Eliciting new Wikipedia users’ interests via automatically mined questionnaires: For a warm welcome, not a cold start. International AAAI Conference on Web and Social Media.

[70]

Xing Yi. 2015. Dwell time based advertising in a scrollable content stream. US Patent App. 13/975,157.

Cited By

Lewoniewski W(2024)The Most Cited Scientific Information Sources in Wikipedia Articles Across Various LanguagesBiblioteka10.14746/b.2023.27.12(269-294)Online publication date: 7-Mar-2024
https://doi.org/10.14746/b.2023.27.12
Krenca KTaylor EDeacon S(2024)Scrolling and hyperlinks: The effects of two prevalent digital features on children's digital reading comprehensionJournal of Research in Reading10.1111/1467-9817.12468Online publication date: 7-Aug-2024
https://doi.org/10.1111/1467-9817.12468
Lewoniewski WWęcel KAbramowicz W(2024)Understanding the Use of Scientific References in Multilingual Wikipedia across Various TopicsProcedia Computer Science10.1016/j.procs.2023.10.393225:C(3977-3986)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.10.393
Show More Cited By

Index Terms

Quantifying Engagement with Citations on Wikipedia

Index terms have been assigned to the content through auto-classification.

Recommendations

Methodological issues in measuring citations in Wikipedia: a case study in Library and Information Science

Wikipedia citations have been suggested as a metric that partially captures the impact of research, providing an indication of the transfer of scholarly output to a wider audience beyond the academic community. In this article, we explore the coverage ...
Finding News Citations for Wikipedia
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

An important editing policy in Wikipedia is to provide citations for added statements in Wikipedia pages, where statements can be arbitrary pieces of text, ranging from a sentence to a paragraph. In many cases citations are either outdated or missing ...
Named entity recognition in Wikipedia
People's Web '09: Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources

Named entity recognition (NER) is used in many domains beyond the newswire text that comprises current gold-standard corpora. Recent work has used Wikipedia's link structure to automatically generate near gold-standard annotations. Until now, these ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '20: Proceedings of The Web Conference 2020

April 2020

3143 pages

ISBN:9781450370233

DOI:10.1145/3366423

Editors:
Yennun Huang
Acadmica sinica, Taiwan
,
Irwin King
The Chinese University of Hong Kong, Hong Kong
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
539
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lewoniewski W(2024)The Most Cited Scientific Information Sources in Wikipedia Articles Across Various LanguagesBiblioteka10.14746/b.2023.27.12(269-294)Online publication date: 7-Mar-2024
https://doi.org/10.14746/b.2023.27.12
Krenca KTaylor EDeacon S(2024)Scrolling and hyperlinks: The effects of two prevalent digital features on children's digital reading comprehensionJournal of Research in Reading10.1111/1467-9817.12468Online publication date: 7-Aug-2024
https://doi.org/10.1111/1467-9817.12468
Lewoniewski WWęcel KAbramowicz W(2024)Understanding the Use of Scientific References in Multilingual Wikipedia across Various TopicsProcedia Computer Science10.1016/j.procs.2023.10.393225:C(3977-3986)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.procs.2023.10.393
Yang PShoaib AWest RColavizza G(2024)Open access improves the dissemination of science: insights from WikipediaScientometrics10.1007/s11192-024-05163-4129:11(7083-7106)Online publication date: 15-Oct-2024
https://doi.org/10.1007/s11192-024-05163-4
Li HVincent NChancellor SHecht B(2023)The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data ProducersProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594070(1151-1161)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594070
Piccardi TGerlach MArora AWest R(2023)A Large-Scale Characterization of How Readers Browse WikipediaACM Transactions on the Web10.1145/358031817:2(1-22)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3580318
Baigutanova AMyung JSaez-Trumper DChou ARedi MJung CCha M(2023)Longitudinal Assessment of Reference Quality on WikipediaProceedings of the ACM Web Conference 202310.1145/3543507.3583218(2831-2839)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583218
Zou JAliannejadi MKanoulas EPera MLiu Y(2023)Users Meet Clarifying Questions: Toward a Better Understanding of User Interactions for Search ClarificationACM Transactions on Information Systems10.1145/352411041:1(1-25)Online publication date: 9-Jan-2023
https://dl.acm.org/doi/10.1145/3524110
Petroni FBroscheit SPiktus ALewis PIzacard GHosseini LDwivedi-Yu JLomeli MSchick TBevilacqua MMazaré PJoulin AGrave ERiedel S(2023)Improving Wikipedia verifiability with AINature Machine Intelligence10.1038/s42256-023-00726-15:10(1142-1148)Online publication date: 19-Oct-2023
https://doi.org/10.1038/s42256-023-00726-1
Schmidt MKircheis WSimons APotthast MStein B(2023)A diachronic perspective on citation latency in Wikipedia articles on CRISPR/Cas-9: an exploratory case studyScientometrics10.1007/s11192-023-04703-8128:6(3649-3673)Online publication date: 14-May-2023
https://doi.org/10.1007/s11192-023-04703-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents