research-article

Free access

Automatic evaluation of topic coherence

Authors:

Timothy BaldwinAuthors Info & Claims

HLT '10: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics

Pages 100 - 108

Published: 02 June 2010 Publication History

Abstract

This paper introduces the novel task of topic coherence evaluation, whereby a set of words, as generated by a topic model, is rated for coherence or interpretability. We apply a range of topic scoring models to the evaluation task, drawing on WordNet, Wikipedia and the Google search engine, and existing research on lexical similarity/relatedness. In comparison with human scores for a set of learned topics over two distinct datasets, we show a simple co-occurrence measure based on pointwise mutual information over Wikipedia data is able to achieve results for the task at or nearing the level of inter-annotator correlation, and that other Wikipedia-based lexical relatedness methods also achieve strong results. Google produces strong, if less consistent, results, while our results over WordNet are patchy at best.

References

[1]

E Agirre, E Alfonseca, K Hall, J Kravalova, M Paşca, and A Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proc. of HLT: NAACL 2009, pages 19--27, Boulder, Colorado.

Digital Library

[2]

S Banerjee and T Pedersen. 2002. An adapted Lesk algorithm for word sense disambiguation using WordNet. Proc. of CICLing'02, pages 136--145.

Digital Library

[3]

DM Blei, AY Ng, and MI Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993--1022.

Digital Library

[4]

S Brody and M Lapata. 2009. Bayesian word sense induction. In Proc. of EACL 2009, pages 103--111, Athens, Greece.

Digital Library

[5]

A Budanitsky and G Hirst. 2005. Evaluating WordNet-based Measures of Lexical Sematic Relatedness. Computational Linguistics, 32(1):13--47.

Digital Library

[6]

WL Buntine and A Jakulin. 2004. Applying discrete PCA in data analysis. In Proc. of UAI 2004, pages 59--66.

Digital Library

[7]

J Chang, J Boyd-Graber, S Gerris, C Wang, and D Blei. 2009. Reading tea leaves: How humans interpret topic models. In Proc. of NIPS 2009.

[8]

H Daume III. 2009. Non-parametric bayesian areal linguistics. In Proc. of HLT: NAACL 2009, pages 593--601, Boulder, USA.

Digital Library

[9]

Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6).

[10]

C Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, USA.

[11]

E Gabrilovich and S Markovitch. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proc. of IJCAI'07, pages 1606--1611, Hyderabad, India.

Digital Library

[12]

T Griffiths and M Steyvers. 2004. Finding scientific topics. In Proc. of the National Academy of Sciences, volume 101, pages 5228--5235.

[13]

T Griffiths and M Steyvers. 2006. Probabilistic topic models. In Latent Semantic Analysis: A Road to Meaning.

[14]

A Haghighi and L Vanderwende. 2009. Exploring content models for multi-document summarization. In Proc. of HLT: NAACL 2009, pages 362--370, Boulder, USA.

Digital Library

[15]

G Hirst and D St-Onge. 1998. Lexical chains as representations of context for the detection and correction of malapropism. In Fellbaum (Fellbaum, 1998), pages 305--332.

[16]

T Hofmann. 2001. Unsupervised learning by probabilistic latent semantic analysis. Machine Learning, 42(1):177--196.

Digital Library

[17]

JJ Jiang and DW Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proc. of COLING'97, pages 19--33, Taipei, Taiwan.

[18]

C Leacock, G A Miller, and M Chodorow. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147--65.

Digital Library

[19]

M Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In Proc. of SIGDOC'86, pages 24--26, Toronto, Canada.

Digital Library

[20]

D Lin. 1998. Automatic retrieval and clustering of similar words. In Proc. of COLING/ACL'98, pages 768--774, Montreal, Canada.

Digital Library

[21]

C-Y Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Proc. of the ACL 2004 Workshop on Text Summarization Branches Out (WAS 2004), pages 74--81, Barcelona, Spain.

[22]

Q Mei, X Shen, and CX Zhai. 2007. Automatic labeling of multinomial topic models. In Proc. of KDD 2007, pages 490--499.

Digital Library

[23]

D Milne and IH Witten. 2008. An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In Proc. of AAAI Workshop on Wikipedia and Artificial Intelligence, pages 25--30, Chicago, USA.

[24]

H Misra, O Cappe, and F Yvon. 2008. Using LDA to detect semantically incoherent documents. In Proc. of CoNLL 2008, pages 41--48, Manchester, England.

Digital Library

[25]

D Newman, S Karimi, and L Cavedon. 2009. External evaluation of topic models. In Proc. of ADCS 2009, pages 11--18, Sydney, Australia.

[26]

D Newman, T Baldwin, L Cavedon, S Karimi, D Martinez, and J Zobel. to appeara. Visualizing document collections and search results using topic mapping. Journal of Web Semantics.

[27]

D Newman, Y Noh, E Talley, S Karimi, and T Baldwin. to appearb. Evaluating topic models for digital libraries. In Proc. of JCDL/ICADL 2010, Gold Coast, Australia.

Digital Library

[28]

K Papineni, S Roukos, T Ward, and W-J Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL 2002, pages 311--318, Philadelphia, USA.

Digital Library

[29]

P Pecina. 2008. Lexical Association Measures: Collocation Extraction. Ph.D. thesis, Charles University.

[30]

P Resnik. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proc. of IJCAI'95, pages 448--453, Montreal, Canada.

Digital Library

[31]

H Schütze. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97--123.

Digital Library

[32]

M Strübe and SP Ponzetto. 2006. WikiRelate! computing semantic relateness using Wikipedia. In Proc. of AAAI'06, pages 1419--1424, Boston, USA.

Digital Library

[33]

Q Sun, R Li, D Luo, and X Wu. 2008. Text segmentation with LDA-based Fisher kernel. In Proc. of ACL-08: HLT, pages 269--272.

Digital Library

[34]

HM Wallach, I Murray, R Salakhutdinov, and DM Mimno. 2009. Evaluation methods for topic models. In Proc. of ICML 2009, page 139.

Digital Library

[35]

D Widdows and K Ferraro. 2008. Semantic Vectors: A scalable open source package and online technology management application. In Proc. of LREC 2008, Marrakech, Morocco.

[36]

Z Wu and M Palmer. 1994. Verb selection and lexical selection. In Proc. of ACL'94, pages 133--138, Las Cruces, USA.

Digital Library

Cited By

Tang YSitu JHuang YKostakos VKay JHoang T(2024)Beyond User Experience: Technical and Contextual Metrics for Large Language Models in Extended RealityCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678995(640-643)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678995
Shin HChoi JOh C(2024)Delivering the Future: Understanding User Perceptions of Delivery RobotsProceedings of the ACM on Human-Computer Interaction10.1145/36536878:CSCW1(1-24)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653687
Silva CGalster MGilson F(2024)Applying short text topic models to instant messaging communication of software developersJournal of Systems and Software10.1016/j.jss.2024.112111216:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112111
Show More Cited By

Recommendations

A Non-Parametric Topic Model for Short Texts Incorporating Word Coherence Knowledge
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

Mining topics in short texts (e.g. tweets, instant messages) can help people grasp essential information and understand key contents, and is widely used in many applications related to social media and text analysis. The sparsity and noise of short ...
An analysis of the coherence of descriptors in topic modeling

We evaluate the coherence and generality of topic descriptors found by LDA and NMF.Six new and existing corpora were specifically compiled for this evaluation.A new coherence measure using word2vec-modeled term vector similarity is proposed.NMF ...
Aggregated topic models for increasing social media topic coherence
Abstract
This research presents a novel aggregating method for constructing an aggregated topic model that is composed of the topics with greater coherence than individual models. When generating a topic model, a number of parameters have to be specified. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

HLT '10: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics

June 2010

1070 pages

ISBN:1932432655

General Chair:
Ronald M. Kaplan
Microsoft Bing

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 02 June 2010

Qualifiers

Research-article

Acceptance Rates

Overall Acceptance Rate 240 of 768 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

108
Total Citations
View Citations
3,348
Total Downloads

Downloads (Last 12 months)158
Downloads (Last 6 weeks)26

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tang YSitu JHuang YKostakos VKay JHoang T(2024)Beyond User Experience: Technical and Contextual Metrics for Large Language Models in Extended RealityCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678995(640-643)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678995
Shin HChoi JOh C(2024)Delivering the Future: Understanding User Perceptions of Delivery RobotsProceedings of the ACM on Human-Computer Interaction10.1145/36536878:CSCW1(1-24)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653687
Silva CGalster MGilson F(2024)Applying short text topic models to instant messaging communication of software developersJournal of Systems and Software10.1016/j.jss.2024.112111216:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112111
Xu YSun JSu YLiu XDuan ZChen BZhou MOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Context-guided embedding adaptation for effective topic modeling in low-resource regimesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669624(79959-79979)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669624
Hosseiny Marani ABaumer E(2023)A Review of Stability in Topic Modeling: Metrics for Assessing and Techniques for Improving StabilityACM Computing Surveys10.1145/362326956:5(1-32)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3623269
Yuan MLin PRashidi LZobel JYoshioka MKiseleva JAliannejadi M(2023)Assessment of the Quality of Topic Models for Information Retrieval ApplicationsProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605118(265-274)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3578337.3605118
Gupta AZhang Z(2023)Neural Topic Modeling via Discrete Variational InferenceACM Transactions on Intelligent Systems and Technology10.1145/357050914:2(1-33)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.1145/3570509
Khalili FMohebbi ATerragni VPezzè MMariani LHeydarnoori ARastogi ATufano RBavota GArnaoudova VHaiduc S(2022)The ineffectiveness of domain-specific word embedding models for GUI test reuseProceedings of the 30th IEEE/ACM International Conference on Program Comprehension10.1145/3524610.3527873(560-564)Online publication date: 16-May-2022
https://dl.acm.org/doi/10.1145/3524610.3527873
Peikert SKubach CAl Qundus JSandra Vu LPaschke A(2021)Objective Functions to Determine the Number of Topics for Topic ModelingThe 23rd International Conference on Information Integration and Web Intelligence10.1145/3487664.3487710(328-332)Online publication date: 29-Nov-2021
https://dl.acm.org/doi/10.1145/3487664.3487710
El Akrouchi MBenbrahim HKassou I(2021)Review on adopting concept extraction in weak signals detection in competitive intelligenceThe 7th Annual International Conference on Arab Women in Computing in Conjunction with the 2nd Forum of Women in Research10.1145/3485557.3485560(1-8)Online publication date: 25-Aug-2021
https://dl.acm.org/doi/10.1145/3485557.3485560
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents