RCV1: A New Benchmark Collection for Text Categorization Research

Reviewer: Fabrizio Sebastiani

Reuters-21578 is a familiar name to information retrieval (IR) researchers, since it denotes the standard benchmark of research in text categorization (TC), the subfield of IR concerned with the classification of textual documents into a set of predefined categories. A TC benchmark is a set of pre-classified documents, and it serves two main purposes: first, a text classifier can be automatically built by learning the semantic characteristics of the categories from a subset of the benchmark known as the training set; second, the accuracy of the classifier can be tested by comparing its classification decisions with those encoded in the remaining documents, collectively known as the test set. Throughout the 1990s, the availability of Reuters-21578, a set of 12,902 newswires pre-classified according to 118 topical classes, contributed to stimulating research in TC, by providing a common testing ground for competing TC approaches. It is now time, however, for Reuters-21578 to go into early retirement, especially because its size is now considered too small to warrant significant conclusions from experiments (in other IR subfields, standard benchmarks easily reach up to tens of millions of documents). This paper reports on a newly available test collection, Reuters Corpus Volume I (RCV1), which promises to be the new standard benchmark of TC research, and a better substitute for Reuters-21578, because of sheer size (804,414 documents), Extensible Markup Language (XML) tagging, a smaller amount of noise, and more clearly specified semantics. The authors' report on the coding practices that human coders used in preparing the data, on the nature of the data, and on the statistics about it, will be necessary background for anyone wishing to use RCV1 in their TC experiments, while the authors' report on experimental results they obtained by running several known TC techniques (ranging from example-based methods to support vector machines) on the data will provide baselines upon which new techniques will need to improve. This is an essential paper for those wishing to engage in text categorization research.