Evaluation Methods of Hierarchical Models
Pages 455 - 464
Abstract
In this paper, we consider the problem of evaluating the quality of hierarchical models. This task arises due to the current researchers use subjective evaluation, such as a survey to test the goodness of a hierarchy discovered by their models. We propose three methods to evaluate the quality of hierarchy extracted from unstructured text. These methods are used to reflects three important characteristics of an optimal tree: (1) Coverage which reflects a topic on a high level, close to the root node, should cover a wider range of sub-concepts than those on a lower level; (2) Parent-child relentless which means the parent topic in the tree should be semantically related to its children rather than to its non-children; (3) Topic coherence that identifies all words within a topic should be semantically related to the other words. Moreover, we introduce a new metric called, Interest-based coherent to evaluate the hierarchical tree extracted from structured data like relational data. We compare different state-of-art methods and perform extensive experiments on three real datasets. The results confirm that the proposed methods can properly evaluate the quality of the hierarchy discovered by several models.
References
[1]
Ibrahim, I.A., Almars, A., Pokharel, S., Zhao, X., Li, X.: Interesting recommendations based on hierarchical visualizations of medical data (2018)
[2]
Almars A, Li X, Ibrahim IA, and Zhao X Hacid H, Cellary W, Wang H, Paik H-Y, and Zhou R Learning concept hierarchy from short texts using context coherence Web Information Systems Engineering – WISE 2018 2018 Cham Springer 319-329
[3]
Almars Abdulqader, Li Xue, Zhao Xin, Ibrahim Ibrahim A., Yuan Weiwei, and Li Bohan Structured Sentiment Analysis Advanced Data Mining and Applications 2017 Cham Springer International Publishing 695-707
[4]
Blei DM, Griffiths TL, and Jordan MI The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies J. ACM 2010 57 7
[5]
Chen, H., Yin, H., Li, X., Wang, M., Chen, W., Chen, T.: People opinion topic model: opinion based user clustering in social networks. In: Proceedings of the 26th International Conference on World Wide Web Companion, WWW 2017 Companion (2017)
[6]
Chen, P., Zhang, N.L., Liu, T., Poon, L.K.M., Chen, Z.: Latent tree models for hierarchical topic detection. CoRR abs/1605.06650 (2016)
[7]
Chen, W., et al.: EEG-based motion intention recognition via multi-task RNNs, pp. 279–287. SIAM (2018)
[8]
Church KW and Hanks P Word association norms, mutual information, and lexicography Comput. Linguist. 1990 16 22-29
[9]
Gerani, S., Carenini, G., Ng, R.T.: Modeling content and structure for abstractive review summarization. Comput. Speech Lang. (2016)
[10]
Ibrahim IA, Albarrak AM, and Li X Constrained recommendations for query visualizations Knowl. Inf. Syst. 2017 51 499-529
[11]
Kim, J.H., Kim, D., Kim, S., Oh, A.: Modeling topic hierarchies with the recursive Chinese restaurant process. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 783–792. ACM (2012)
[12]
Kim, S., Zhang, J., Chen, Z., Oh, A.H., Liu, S.: A hierarchical aspect-sentiment model for online reviews. In: AAAI (2013)
[13]
Li, W., McCallum, A.: Pachinko allocation: dag-structured mixture models of topic correlations. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 577–584. ACM, New York (2006)
[14]
Mimno, D., Li, W., McCallum, A.: Mixtures of hierarchical topics with pachinko allocation. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, pp. 633–640 (2007)
[15]
Mimno, D., Wallach, H.M., Talley, E., Leenders, M., McCallum, A.: Optimizing semantic coherence in topic models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, pp. 262–272. Association for Computational Linguistics, Stroudsburg (2011)
[16]
Wang, C., et al.: A phrase mining framework for recursive construction of a topical hierarchy. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 437–445 (2013)
[17]
Wang, S., Chen, Z., Liu, B.: Mining aspect-specific opinion using a holistic lifelong topic model. In: Proceedings of the 25th International Conference on World Wide Web, WWW 2016, pp. 167–176 (2016)
[18]
Xu Y, Yin J, Huang J, and Yin Y Hierarchical topic modeling with automatic knowledge mining Expert Syst. Appl. 2018 103 106-117
[19]
Yan, X., Guo, J., Lan, Y., Cheng, X.: A biterm topic model for short texts. In: Proceedings of the 22nd International Conference on World Wide Web, WWW 2013 (2013)
[20]
Zhao P, Li X, and Wang K Lin X, Manolopoulos Y, Srivastava D, and Huang G Feature extraction from micro-blogs for comparison of products and services Web Information Systems Engineering – WISE 2013 2013 Heidelberg Springer 82-91
Recommendations
Hierarchical summarization of large documents
Many automatic text summarization models have been developed in the last decades. Related research in information science has shown that human abstractors extract sentences for summaries based on the hierarchical structure of documents; however, the ...
Integration of Opinion into Customer Analysis Model
ICEBE '11: Proceedings of the 2011 IEEE 8th International Conference on e-Business EngineeringNowadays, Opinion Mining is getting more important than before especially in doing analysis and forecasting about customers' behavior for businesses purpose. The right decision in producing new products or services based on data about customers' ...
Leveraging integrated information to extract query subtopics for search result diversification
AbstractSearch result diversification aims to diversify search results to cover different query subtopics, i.e., pieces of relevant information. The state of the art diversification methods often explicitly model the diversity based on query subtopics, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Nov 2018
537 pages
ISBN:978-3-030-05089-4
DOI:10.1007/978-3-030-05090-0
- Editors:
- Guojun Gan,
- Bohan Li,
- Xue Li,
- Shuliang Wang
© Springer Nature Switzerland AG 2018.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 29 December 2018
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 08 Feb 2025