Nothing Special   »   [go: up one dir, main page]

Skip to main content

RepLab: An Evaluation Campaign for Online Monitoring Systems

  • Chapter
  • First Online:
Information Retrieval Evaluation in a Changing World

Part of the book series: The Information Retrieval Series ((INRE,volume 41))

Abstract

Over a period of 3 years, RepLab was a CLEF initiative where computer scientists and online reputation experts worked together to identify and formalize the computational challenges in the area of online reputation monitoring. Two main results emerged from RepLab: a community of researchers engaged in the problem, and an extensive Twitter test collection comprising more than half a million expert annotations, which cover many relevant tasks in the field of online reputation: named entity resolution, topic detection and tracking, reputational alerts identification, reputational polarity, author profiling, opinion makers identification and reputational dimension classification. It has probably been one of the CLEF labs with a larger set of expert annotations provided to participants in a single year, and one of the labs where the target user community has been more actively engaged in the evaluation campaign. Here we summarize the design and results of the Replab campaigns, and also report on research that has built on RepLab datasets after completion of the 3-year competition cycle.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  • Alsop RJ (2006) The 18 immutable laws of corporate reputation: creating, protecting and repairing your most valuable asset. Kogan Page, London

    Google Scholar 

  • Amigó E, Gonzalo J, Artiles J, Verdejo F (2009) A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf Retr 12(4):461–486

    Article  Google Scholar 

  • Amigó E, Gonzalo J, Artiles J, Verdejo F (2011) Combining evaluation metrics via the unanimous improvement ratio and its application to clustering tasks. J Artif Intell Res 42(1):689–718

    MathSciNet  MATH  Google Scholar 

  • Amigó E, Gonzalo J, Verdejo F (2013) A general evaluation measure for document organization tasks. In: Proceedings of the 36th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’13, pp 643–652. https://doi.org/10.1145/2484028.2484081

  • Amigó E, Carrillo-de Albornoz J, Almagro-Cádiz M, Gonzalo J, Rodríguez-Vidal J, Verdejo F (2017) Evall: open access evaluation for information access systems. In: Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’17, pp 1301–1304. https://doi.org/10.1145/3077136.3084145

  • Balahur A, Tanev H (2012) Detecting entity-related events and sentiments from tweets using multilingual resources. In: CLEF (Online Working Notes/Labs/Workshop)

    Google Scholar 

  • Berrocal A, Luis J, Figuerola CG, Zazo Rodríguez ÁF (2013) Reina at replab2013 topic detection task: community detection. In: CLEF (Working Notes)

    Google Scholar 

  • Carrillo-de-Albornoz J, Chugur I, Amigó E (2012) Using an emotion-based model and sentiment analysis techniques to classify polarity for reputation. In: CLEF (Online Working Notes/Labs/Workshop)

    Google Scholar 

  • Carrillo-de Albornoz J, Amigó E, Plaza L, Gonzalo J (2016) Tweet stream summarization for online reputation management. In: Ferro N, Crestani F, Moens MF, Mothe J, Silvestri F, Di Nunzio GM, Hauff C, Silvello G (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 378–389

    Chapter  Google Scholar 

  • Chong WN, Tan G (2010) Obtaining intangible and tangible benefits from corporate social responsibility. Int Rev Bus Res Pap 6(4):360

    Google Scholar 

  • Cossu JV, Bigot B, Bonnefoy L, Morchid M, Bost X, Senay G, Dufour R, Bouvier V, Torres-Moreno JM, El-Bèze M (2013) Lia at replab 2013. In: CLEF (Working Notes)

    Google Scholar 

  • Cossu JV, Bigot B, Bonnefoy L, Senay G (2014a) Towards the improvement of topic priority assignment using various topic detection methods for e-reputation monitoring on twitter. In: Métais E, Roche M, Teisseire M (eds) Natural language processing and information systems. Springer International Publishing, Cham, pp 154–159

    Chapter  Google Scholar 

  • Cossu JV, Janod K, Ferreira E, Gaillard J, El-Bèze M (2014b) Lia at replab 2014: 10 methods for 3 tasks. In: 4th international conference of the CLEF initiative

    Google Scholar 

  • Doorley J, Garcia HF (2011) Reputation management: the key to successful public relations and corporate communication. Routledge, New York

    Google Scholar 

  • Filgueiras J, Amir S (2013) Popstar at replab 2013: polarity for reputation classification. In: CLEF (Online Working Notes/Labs/Workshop), vol 60

    Google Scholar 

  • Gârbacea C, Tsagkias M, de Rijke M (2014) Detecting the reputation polarity of microblog posts. In: Proceedings of the twenty-first european conference on artificial intelligence. IOS Press, Amsterdam, ECAI’14, pp 339–344. https://doi.org/10.3233/978-1-61499-419-0-339

  • Giachanou A, Gonzalo J, Mele I, Crestani F (2017) Sentiment propagation for predicting reputation polarity. In: Jose JM, Hauff C, Altıngovde IS, Song D, Albakour D, Watt S, Tait J (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 226–238

    Chapter  Google Scholar 

  • Glance N, Hurst M, Nigam K, Siegler M, Stockton R, Tomokiyo T (2005) Deriving marketing intelligence from online discussion. In: Proceedings of the eleventh ACM SIGKDD international conference on knowledge discovery in data mining. ACM, New York, NY, KDD ’05, pp 419–428. https://doi.org/10.1145/1081870.1081919

  • Hangya V, Farkas R (2013) Filtering and polarity detection for reputation management on tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60

    Google Scholar 

  • Hoffman T (2008) Online reputation management is hot – but is it ethical. Computerworld (44). https://www.computerworld.com/article/2537007/networking/online-reputation-management-is-hot----but-is-it-ethical-.html

  • Jansen B, Zhang M, Sobel K, Chowdury A (2009a) Twitter power: tweets as electronic word of mouth. J Am Soc Inf Sci Technol 60(11):2169–2188

    Article  Google Scholar 

  • Jansen BJ, Zhang M, Sobel K, Chowdury A (2009b) Twitter power: tweets as electronic word of mouth. J Assoc Inf Sci Technol 60(11):2169–2188

    Article  Google Scholar 

  • Kaptein R (2012) Learning to analyze relevancy and polarity of tweets. In: CLEF (Online Working Notes/Labs/Workshop), vol 60

    Google Scholar 

  • Kreps DM, Wilson R (1982) Reputation and imperfect information. J Econ Theory 27(2):253–279

    Article  MathSciNet  Google Scholar 

  • Krishnamurthy B, Gill P, Arlitt M (2008) A few chirps about Twitter. In: Proceedings of the first workshop on online social networks (WOSP’08), pp 19–24

    Google Scholar 

  • Li YM, Li TY (2013) Deriving market intelligence from microblogs. Decis Support Syst 55(1):206–217. https://doi.org/10.1016/j.dss.2013.01.023. http://www.sciencedirect.com/science/article/pii/S0167923613000511

    Article  Google Scholar 

  • Mabrouk O, Hlaoua L, Nazih Omri M (2018) Profile categorization system based on features reduction. In: International symposium on artificial intelligence and mathematics, ISAIM 2018, Fort Lauderdale, Florida. http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Mabrouk_etal.pdf

  • Magdy W, Elsayed T (2016) Unsupervised adaptive microblog filtering for broad dynamic topics. Inf Process Manag 52(4):513–528

    Article  Google Scholar 

  • Mahalakshmi GS, Koquilamballe K, Sendhilkumar S (2017) Influential detection in twitter using tweet quality analysis. In: 2017 second international conference on recent trends and challenges in computational models (ICRTCCM), pp 315–319. https://doi.org/10.1109/ICRTCCM.2017.62

  • Martın T, Spina D, Amigó E, Gonzalo J (2012) Uned at replab 2012: monitoring task. In: CLEF 2012 Working Notes, CLEF

    Google Scholar 

  • Martín-Wanton T, Gonzalo J, Amigó E (2013) An unsupervised transfer learning approach to discover topics for online reputation management. In: Proceedings of the 22nd ACM international conference on conference on information & knowledge management, ACM, New York, NY, CIKM ’13, pp 1565–1568. https://doi.org/10.1145/2505515.2507845

  • McDonald G, Deveaud R, McCreadie R, Gollins T, Macdonald C, Ounis I (2014) University of glasgow terrier team/project abacá at replab 2014: reputation dimensions task. In: CLEF (Working Notes), pp 1500–1504

    Google Scholar 

  • McDonald G, Deveaud R, Mccreadie R, Macdonald C, Ounis I (2015) Tweet enrichment for effective dimensions classification in online reputation management. In: 9th international AAAI conference on web and social media, pp 654–657

    Google Scholar 

  • Nebot V, Rangel F, Berlanga R, Rosso P (2018) Identifying and classifying influencers in twitter only with textual information. In: Proceedings of the NLDB 2018

    Google Scholar 

  • Panem S, Bansal R, Gupta M, Varma V (2014) Entity tracking in real-time using sub-topic detection on twitter. In: de Rijke M, Kenter T, de Vries AP, Zhai C, de Jong F, Radinsky K, Hofmann K (eds) Advances in information retrieval. Springer International Publishing, Cham, pp 528–533

    Chapter  Google Scholar 

  • Peetz MH, de Rijke M, Kaptein R (2016) Estimating reputation polarity on microblog posts. Inf Process Manag 52(2):193–216. https://doi.org/10.1016/j.ipm.2015.07.003. http://www.sciencedirect.com/science/article/pii/S0306457315000874

    Article  Google Scholar 

  • Qureshi M, Younus A, O’Riordan C, Pasi G (2015) Company name disambiguation in tweets: a two-step filtering approach. In: Information retrieval technology, vol 9460

    Chapter  Google Scholar 

  • Qureshi MA, O’Riordan C, Pasi G (2012) Concept term expansion approach for monitoring reputation of companies on twitter. In: CLEF (Online Working Notes/Labs/Workshop)

    Google Scholar 

  • Qureshi MA, Younus A, O’Riordan C, Pasi G (2017) A wikipedia-based semantic relatedness framework for effective dimensions classification in online reputation management. J Ambient Intell Humaniz Comput 9:1403

    Article  Google Scholar 

  • Rodriguez J, Carrillo-de Albornoz J, Plaza L, Amigó E, Gonzalo J (2019a) Automatic generation of entity-oriented summaries for reputation management. J Ambient Intell Humaniz Comput:1–15. https://link.springer.com/article/10.1007/s12652-019-01255-9

  • Rodriguez J, Gonzalo J, Plaza L, Anaya H (2019b) Automatic detection of influencers in social networks: authority versus domain signals. J Assoc Inf Sci Technol 70:7

    Google Scholar 

  • Spina D, Carrillo-de-Albornoz J, Martín-Wanton T, Amigó E, Gonzalo J, Giner F (2013) Uned online reputation monitoring team at replab 2013. In: CLEF (Working Notes)

    Google Scholar 

  • Spina D, Gonzalo J, Amigó E (2014) Learning similarity functions for topic detection in online reputation monitoring. In: Proceedings of the 37th international ACM SIGIR conference on research & development in information retrieval. ACM, New York, NY, SIGIR ’14, pp 527–536. https://doi.org/10.1145/2600428.2609621

  • Spina D, Peetz MH, de Rijke M (2015) Active learning for entity filtering in microblog streams. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. ACM, New York, NY, SIGIR ’15, pp 975–978. https://doi.org/10.1145/2766462.2767839

  • Vilares D, Hermo M, Alonso MA, Gómez-Rodríguez C, Vilares J (2014) Lys at clef replab 2014: creating the state of the art in author influence ranking and reputation classification on twitter. In: CLEF (Working Notes), pp 1468–1478

    Google Scholar 

  • Villena-Román J, Lana-Serrano S, Moreno C, García-Morera J, Cristóbal JCG (2012) Daedalus at replab 2012: polarity classification and filtering on twitter data. In: CLEF (Online Working Notes/Labs/Workshop), vol 60

    Google Scholar 

Download references

Acknowledgement

This research was partially supported by the Spanish Ministry of Science and Innovation (Vemodalen Project, TIN2015-71785-R).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Julio Gonzalo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Carrillo-de-Albornoz, J., Gonzalo, J., Amigó, E. (2019). RepLab: An Evaluation Campaign for Online Monitoring Systems. In: Ferro, N., Peters, C. (eds) Information Retrieval Evaluation in a Changing World. The Information Retrieval Series, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-22948-1_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-22948-1_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-22947-4

  • Online ISBN: 978-3-030-22948-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics