Abstract
While deep learning models become more widespread, their ability to handle unseen data and generalize for any scenario is yet to be challenged. In medical imaging, there is a high heterogeneity of distributions among images based on the equipment that generates them and their parametrization. This heterogeneity triggers a common issue in machine learning called domain shift, which represents the difference between the training data distribution and the distribution of where a model is employed. A high domain shift often results in a poor generalization performance from the models. In this work, we evaluate the extent of which domain shift damages model performance on four of the largest datasets of chest radiographs. We show how training and testing with different datasets (e.g., training in ChestX-ray14 and testing in CheXpert) drastically affects model performance, posing a big question over the reliability of deep learning models trained on public datasets. We also show that models trained on CheXpert and MIMIC-CXR generalized better to other datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
AlBadawy, E.A., Saha, A., Mazurowski, M.A.: Deep learning for segmentation of brain tumors: impact of cross-institutional training and testing. Med. Phys. 45(3), 50 (2018)
Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: Padchest: a large chest x-ray image dataset with multi-label annotated reports (2019)
Chen, C., Dou, Q., Chen, H., Heng, P.A.: Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation. In: Proceedings of the International Workshop on Machine Learning in Medical Imaging. Springer (2018)
del Ciello, A., Franchi, P., Contegiacomo, A., Cicchetti, G., Bonomo, L., Larici, A.R.: Missed lung cancer: when, where, and why? Diagnostic and Interventional Radiol. 23(2), 118–126 (2017). https://doi.org/10.5152/dir.2016.16187
Dou, Q., Ouyang, C., Chen, C., Chen, H., Heng, P.A.: Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 691–697 (2018)
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 4 (2016)
Gholami, A., et al.: A novel domain adaptation framework for medical image segmentation. In: Proceedings of the International Medical Image Computing and Computer Assisted Intervention Brainlesion Workshop, pp. 289–298. Springer, Cham (2018)
Hirsch, F.R., Franklin, W.A., Gazdar, A.F., Bunn, P.A.: Early detection of lung cancer: clinical perspectives of recent advances in biology and radiology. Clin. Cancer Res. 7(1), 5–22 (2001)
Hoffman, J., et al.: Cycada: cycle consistent adversarial domain adaptation. In: Proceedings of the International Conference on Machine Learning, p. 15 (2018)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Irvin, J., et al.: Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019)
Javanmardi, M., Tasdizen, T.: Domain adaptation for biomedical image segmentation using adversarial training. In: Proceedings of the 15th International Symposium on Biomedical Imaging. IEEE (2018)
Johnson, A.E., et al.: Mimic-cxr: a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)
Madani, A., Moradi, M., Karargyris, A., Syeda-Mahmood, T.: Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). pp. 1038–1042. IEEE (2018). https://doi.org/10.1109/ISBI.2018.8363749
Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans. Med. Imag. 37, 10 (2018)
Oakden-Rayner, L.: Exploring large scale public medical image datasets. Technical report, The University of Adelaide (2019). https://arxiv.org/pdf/1907.12720.pdf
Rajpurkar, P., et al.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
Torralba, A., Efros, A.A., et al.: Unbiased look at dataset bias. In: Proceedings of the Conference on Computer Vision and Pattern Recognition. vol. 1, p. 7. Citeseer (2011)
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. CoRR abs/1705.02315 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Pooch, E.H.P., Ballester, P., Barros, R.C. (2020). Can We Trust Deep Learning Based Diagnosis? The Impact of Domain Shift in Chest Radiograph Classification. In: Petersen, J., et al. Thoracic Image Analysis. TIA 2020. Lecture Notes in Computer Science(), vol 12502. Springer, Cham. https://doi.org/10.1007/978-3-030-62469-9_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-62469-9_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62468-2
Online ISBN: 978-3-030-62469-9
eBook Packages: Computer ScienceComputer Science (R0)