Inter- and Intra-Observer Variability of Radiologists Evaluating CBIR Systems

The purpose of the study is to evaluate the inter- and intra-observer variability of the radiologists in evaluation of the similarity between the query and retrieved ulatasound images containing breast masses by the Content-Based Image Retrieval (CBIR) CADx system. Three radiologists rated the similarity between the query masses and the computer-retrieved (ED-CBIR) masses. Three CBIR systems based on each radiologist’s subjective similarity ratings (R-CBIRs) were formed and compared with the ED-CBIR. The intra-observer variability was smaller than the inter-observer variability for all three radiologists. The radiologists’ performance with the R-CBIRs produced similarity ratings results close to the radiologists’ performance with the ED-CBIR. The average difference in classification accuracy (Az) between the ED-CBIR and the R-CBIRs was slightly lower than the average difference in Az between the R-CBIRs. The relatively large intra- and inter-observer variability may make more difficult to evaluate the effect of the CBIR CADx systems on radiologists’ performance.

Authors and Affiliations

1. Department of Radiology, University of Michigan, Ann Arbor, MI, 48109, USA

   Lubomir Hadjiiski, Hyun-chong Cho, Heang-Ping Chan, Mark A. Helvie, Chintana Paramagul & Alexis V. Nees

2. US Food and Drug Administration, Center for Devices and Radiological Health, Silver Spring, MD, 20993, USA

   Berkman Sahiner

Editors and Affiliations

1. Department of Radiology, University of Pennsylvania, 3400 Spruce Street, 1 Silverstein Building, 19014, Philadelphia, PA, USA

   Andrew D. A. Maidment , Predrag R. Bakic  & Sara Gavenonis ,  & 

Reprints and permissions

© 2012 Springer-Verlag Berlin Heidelberg

Policies and ethics