SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR)

J Arguello, F Diaz, J Lin, A Trotman - … of the 38th International ACM SIGIR …, 2015 - dl.acm.org
Proceedings of the 38th International ACM SIGIR Conference on Research and …, 2015dl.acm.org
Many, if not most, published research papers in Information Retrieval (IR) describe the
following process: the authors identify an opportunity to improve on a particular IR task,
implement an experimental system, and compare its performance against one or more
baselines (or a control condition, in the case of a user study). The quality of the research is
judged based on the magnitude of the improvement and whether the methodological
choices suggest external validity and generalizability, for example, whether the experimental …
Many, if not most, published research papers in Information Retrieval (IR) describe the following process: the authors identify an opportunity to improve on a particular IR task, implement an experimental system, and compare its performance against one or more baselines (or a control condition, in the case of a user study). The quality of the research is judged based on the magnitude of the improvement and whether the methodological choices suggest external validity and generalizability, for example, whether the experimental setup is “realistic” or whether the baseline methods reflect the state of the art. Unfortunately, research demonstrating the failure to reproduce or generalize previous results does not have a similar publication venue. This sort of result—often referred to as a ‘negative result’—serves to control the quality of published research in a scientific discipline and to better understand the limits of previously published methods. Publication venues for such research exist in fields such as ecology, 1 biomedicine, 2 pharmacy, 3, and social science. 4 The SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR) aims to provide a venue for publication and discussion of IR research that fails to reproduce a previously published result under the same or similar experimental conditions (eg, same test collection and system configuration) and research that demonstrates the failure to generalize an existing approach to a new domain. To this end, we have developed a set of categories covering different ways in which a result may
ACM Digital Library
Showing the best result for this search. See all results