Abstract
We discuss the evaluation of rankings of documents that aim to summarize the overall opinion expressed in product reviews. Such a ranking can be used by e-commerce websites to represent a gist of the public opinion about a product. The inability of traditional IR metrics to reward such a representation is argued for. A matrix based labelling procedure serves as the framework for such evaluation. Three alternative metrics are adapted from previous similar works to evaluate opinion representativeness. Similarly, two metrics are adapted for exhaustiveness. Finally, we compare the robustness of these metrics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 659–666. ACM (2008)
Dang, V., Bruce Croft, W.: Diversity by proportionality: an election-based approach to search result diversification. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 65–74. ACM (2012)
Demartini, G., Siersdorfer, S.: Dear search engine: what’s your opinion about...?: sentiment analysis for semantic enrichment of web search results. In: Proceedings of the 3rd International Semantic Search Workshop, p. 4. ACM (2010)
Ding, X., Liu, B., Yu, P.S.: A holistic lexicon-based approach to opinion mining. In: Proceedings of the 2008 International Conference on Web Search and Data Mining, pp. 231–240. ACM (2008)
Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 168–177. ACM (2004)
Lappas, T., Crovella, M., Terzi, E.: Selecting a characteristic set of reviews. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 832–840. ACM (2012)
Liu, Q., Gao, Z., Liu, B., Zhang, Y.: Automated rule selection for aspect extraction in opinion mining. In: IJCAI, pp. 1291–1297 (2015)
Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., Manandhar, S.: Semeval-2014 task 4: aspect based sentiment analysis. In: Proceedings of SemEval, pp. 27–35 (2014)
Zhai, C.X., Cohen, W.W., Lafferty, J.: Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 10–17. ACM (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Singh, A.K., Thawani, A., Gupta, A., Mundotiya, R.K. (2017). Evaluating Opinion Summarization in Ranking. In: Sung, WK., et al. Information Retrieval Technology. AIRS 2017. Lecture Notes in Computer Science(), vol 10648. Springer, Cham. https://doi.org/10.1007/978-3-319-70145-5_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-70145-5_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70144-8
Online ISBN: 978-3-319-70145-5
eBook Packages: Computer ScienceComputer Science (R0)