Improving text retrieval for the routing problem using latent semantic indexing

Reviewer: Alan F. Smeaton

Routing is an information retrieval application based on static user queries or profiles matched against an incoming stream of documents. Hull applies the latent semantic indexing (LSI) technique to the routing problem. Information retrieval research is dominated by work based on the vector space model, in which index terms are represented as orthogonal base vectors and documents and queries are represented as vectors in this vector space. One of the drawbacks of this model is that, in reality, index terms are not independent but have varying degrees of term associations. LSI is a statistical technique that transforms the search space into a space with much-reduced dimensionality that builds in term associations, albeit at the expense of computational overhead. Using a small test collection, this paper reports marginal improvements in the effectiveness of routing with LSI over that of the vector space model. The author then combines LSI with statistical classification techniques based on discriminant analysis and obtains greater improvements in effectiveness. Because of the small size of the document set used, this work can be described as exploratory; the interesting work has still to come when the technique is scaled up to much larger collections.