Abstract
Domain shift poses significant problems to computer-aided diagnostic (CAD) systems when deployed in clinical scenarios. There’s still no definite fix nor an in-depth understanding of the exact factors driving domain shifts in medical X-rays. Here, we conduct an exploratory study on three covariate shift factors in X-ray classification by controlling for different variables. This is possible by leveraging a homogenously-relabelled mix of public and private X-ray data spanning 23 medical institutions over four continents and 17 classes of pathologies. We show that the acquisition parameter, device manufacturer and geographical shifts degrade out-of-distribution (OOD) F1 by 6%, 3.2% and 3.3%, respectively. Pneumothorax was found to be the most impaired pathology, suffering a mean F1 generalisation gap of 13.3%, despite being one of the most clinically-consequential radiological findings. To this end, we introduced LISA-topK, a multi-label adaptation of Learning Invariant Predictors with Selective Augmentation (LISA), that we showed to narrow down the OOD gap, surpassing other methods consistently. These pragmatic results shed light on some of the elements of OOD generalisation in X-ray classification, which are essential to researching, understanding and deploying CAD systems. Code is available at https://github.com/RayscapeAI/LISA-topK
B. Bercean and A. Buburuzan—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Shifts challenge 2022 - grand challenge. www.shifts.grand-challenge.org/. Accessed 10 Mar 2023
Allen, B., Agarwal, S., Coombs, L., Wald, C., Dreyer, K.: 2020 ACR data science institute artificial intelligence survey. J. Am. Coll. Radiol. 18(8), 1153–1159 (2021)
Aubreville, M., Bertram, C., Breininger, K., Jabari, S., Stathonikos, N., Veta, M.: Mitosis domain generalization challenge 2022 (2022). https://doi.org/10.5281/zenodo.6362337
Bandi, P., et al.: From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge. IEEE Trans. Med. Imaging 38(2), 550–560 (2018)
European Society of Radiology (ESR). Current practical experience with artificial intelligence in clinical radiology: a survey of the European Society of Radiology. Insights Imaging 13, 107 (2022). https://doi.org/10.1186/s13244-022-01247-y
Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: PadChest: a large chest X-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020)
Cohen, J.P., Hashir, M., Brooks, R., Bertrand, H.: On the limits of cross-domain generalization in automated X-ray prediction. In: Medical Imaging with Deep Learning, pp. 136–155. PMLR (2020)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Gichoya, J.W., et al.: AI recognition of patient race in medical imaging: a modelling study. Lancet Digital Health 4(6), e406–e414 (2022)
Gulrajani, I., Lopez-Paz, D.: In search of lost domain generalization. In: International Conference on Learning Representations
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Irvin, J., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Thirty-Third AAAI Conference on Artificial Intelligence (2019)
Johnson, A.E., et al.: Mimic-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data 6(1), 317 (2019)
Kilim, O., Olar, A., Joó, T., Palicz, T., Pollner, P., Csabai, I.: Physical imaging parameter variation drives domain shift. Sci. Rep. 12(1), 21302 (2022)
Koh, P.W., et al.: WILDS: a benchmark of in-the-wild distribution shifts. In: International Conference on Machine Learning, pp. 5637–5664. PMLR (2021)
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5542–5550 (2017)
McHugh, M.L.: Interrater reliability: the kappa statistic. Biochemia Med. 22(3), 276–282 (2012)
Pooch, E.H.P., Ballester, P., Barros, R.C.: Can we trust deep learning based diagnosis? The impact of domain shift in chest radiograph classification. In: Petersen, J., et al. (eds.) TIA 2020. LNCS, vol. 12502, pp. 74–83. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62469-9_7
Reis, E.P., et al.: Brax, Brazilian labeled chest X-ray dataset. Sci. Data 9(1), 487 (2022)
Wang, H., Xia, Y.: Domain-ensemble learning with cross-domain mixup for thoracic disease classification in unseen domains. Biomed. Sig. Process. Control 81, 104488 (2023)
Wang, J., et al.: Generalizing to unseen domains: a survey on domain generalization. IEEE Trans. Knowl. Data Eng. 35(8), 8052–8072 (2022)
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)
Wenkel, S.: Concatenated MNIST (CMNIST). making 784 pixels challenging again. (2019). www.simonwenkel.com/publications/articles/pdf/20190924_CMNIST.pdf
Yao, H., et al.: Improving out-of-distribution robustness via selective augmentation. In: International Conference on Machine Learning, pp. 25407–25437. PMLR (2022)
Zhang, H., Dullerud, N., Seyyed-Kalantari, L., Morris, Q., Joshi, S., Ghassemi, M.: An empirical framework for domain generalization in clinical settings. In: Proceedings of the Conference on Health, Inference, and Learning, pp. 279–290 (2021)
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C.: Domain generalization: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 45(4), 4396–4415 (2022)
Zunaed, M., Haque, M., Hasan, T., et al.: Learning to generalize towards unseen domains via a content-aware style invariant framework for disease detection from chest x-rays. arXiv preprint arXiv:2302.13991 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bercean, B., Buburuzan, A., Birhala, A., Avramescu, C., Tenescu, A., Marcu, M. (2023). Breaking Down Covariate Shift on Pneumothorax Chest X-Ray Classification. In: Sudre, C.H., Baumgartner, C.F., Dalca, A., Mehta, R., Qin, C., Wells, W.M. (eds) Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. UNSURE 2023. Lecture Notes in Computer Science, vol 14291. Springer, Cham. https://doi.org/10.1007/978-3-031-44336-7_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-44336-7_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44335-0
Online ISBN: 978-3-031-44336-7
eBook Packages: Computer ScienceComputer Science (R0)