Semantic similarity based on corpus statistics and lexical taxonomy

JJ Jiang, DW Conrath - arXiv preprint cmp-lg/9709008, 1997 - arxiv.org
JJ Jiang, DW Conrath
arXiv preprint cmp-lg/9709008, 1997arxiv.org
This paper presents a new approach for measuring semantic similarity/distance between
words and concepts. It combines a lexical taxonomy structure with corpus statistical
information so that the semantic distance between nodes in the semantic space constructed
by the taxonomy can be better quantified with the computational evidence derived from a
distributional analysis of corpus data. Specifically, the proposed measure is a combined
approach that inherits the edge-based approach of the edge counting scheme, which is then …
This paper presents a new approach for measuring semantic similarity/distance between words and concepts. It combines a lexical taxonomy structure with corpus statistical information so that the semantic distance between nodes in the semantic space constructed by the taxonomy can be better quantified with the computational evidence derived from a distributional analysis of corpus data. Specifically, the proposed measure is a combined approach that inherits the edge-based approach of the edge counting scheme, which is then enhanced by the node-based approach of the information content calculation. When tested on a common data set of word pair similarity ratings, the proposed approach outperforms other computational models. It gives the highest correlation value (r = 0.828) with a benchmark based on human similarity judgements, whereas an upper bound (r = 0.885) is observed when human subjects replicate the same task.
arxiv.org