Abstract
Deployment of different techniques of deep learning including Convolutional Neural Networks (CNN) in image classification systems has accomplished outstanding results. However, the advantages and potential impact of such a system can be completely negated if it does not reach a target accuracy. To achieve high classification accuracy with low variance in medical image classification system, there is needed the large size of the training data set with suitable quality score. This paper presents a study on the use of various consistency checking methods to refine the quality of annotations. It is assumed that tagging was done by volunteers (crowd-sourcing model). The aim of this work was to evaluate the fitness of this approach in the medical field and the usefulness of our innovative web tool, called MedTagger, designed to facilitate large-scale annotation of magnetic resonance (MR) images, as well as the accuracy of crowd-source assessment using this tool, comparing to expert classification. We present the methodology followed to annotate the collection of kidney MR scans. All of the 156 images were acquired from the Medical University of Gdansk. Two groups of students (with and without medical educational background) and three nephrologists were engaged. This research supports the thesis that some types of MR image annotations provided by naive individuals are comparable to expert annotation, but this process could be shortened in time. Furthermore, it is more cost-effective in the simultaneous preservation of image analysis accuracy. With pixel-wise majority voting, it was possible to create crowd-sourced organ segmentations that match the quality of those created by individual medical experts (mAP up to 94% ±3.9%).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
References
Barth, R., IJsselmuiden, J., Hemming, J., Van Henten, E.: Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation. Comput. Electron. Agric. 161, 291–304 (2019). https://doi.org/10.1016/j.compag.2017.11.040. https://www.sciencedirect.com/science/article/pii/S0168169917307664. BigData and DSS in Agriculture
Brzeski, A., Grinholc, K., Nowodworski, K., Przybyłek, A.: Evaluating performance and accuracy improvements for attention-OCR. In: Saeed, K., Chaki, R., Janev, V. (eds.) CISIM 2019. LNCS, vol. 11703, pp. 3–11. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28957-7_1
Cabezas, F., Carlier, A., Charvillat, V., Salvador, A., Giro-I-Nieto, X.: Quality control in crowdsourced object segmentation. In: Proceedings - International Conference on Image Processing, ICIP 2015-December(May), pp. 4243–4247 (2015). https://doi.org/10.1109/ICIP.2015.7351606
Cocos, A., Masino, A., Qian, T., Pavlick, E., Callison-Burch, C.: Effectively crowdsourcing radiology report annotations. In: Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis, pp. 109–114. Association for Computational Linguistics, Lisbon, September 2015. https://doi.org/10.18653/v1/W15-2614
Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD 1996, pp. 226–231. AAAI Press (1996)
Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., Greenspan, H.: GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 321, 321–331 (2018). https://doi.org/10.1016/j.neucom.2018.09.013. https://www.sciencedirect.com/science/article/pii/S0925231218310749
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020). https://doi.org/10.1145/3422622
Heim, E., et al.: Large-scale medical image annotation with crowd-powered algorithms. J. Med. Imaging 5(03), 1 (2018). https://doi.org/10.1117/1.jmi.5.3.034002. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6129178/
Kohli, M.D., Summers, R.M., Geis, J.R.: Medical image data and datasets in the era of machine learning-whitepaper from the 2016 C-MIMI meeting dataset session. J. Digit. Imaging 30(4), 392–399 (2017). https://doi.org/10.1007/s10278-017-9976-3
Montagnon, E., et al.: Deep learning workflow in radiology: a primer (2020). https://doi.org/10.1186/s13244-019-0832-5
Press, W.H., Teukolsky, S.A., Vettering, W.T., Flannery, B.P.: Numerical Recipes the Art of Scientific Computing, 3rd edn. Cambridge University Press (2007). https://doi.org/10.1017/CBO9781107415324.004
Taha, A.A., Hanbury, A.: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med. Imaging 15(1), 29 (2015). https://doi.org/10.1186/s12880-015-0068-x
Jimenez-del Toro, O., et al.: Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: Visceral anatomy benchmarks. IEEE Trans. Med. Imaging 35(11), 2459–2475 (2016). https://doi.org/10.1109/TMI.2016.2578680
Wong, S.C., Gatt, A., Stamatescu, V., McDonnell, M.D.: Understanding data augmentation for classification: when to warp? In: 2016 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2016 (2016). https://doi.org/10.1109/DICTA.2016.7797091
Acknowledgements
This work has been partially supported by Statutory Funds of Electronics, Telecommunications and Informatics Faculty, Gdansk University of Technology, and grants from National Centre for Research and Development (Internet platform for data integration and collaboration of medical research teams for stroke treatment centers, PBS2/A3/17/2013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Cychnerski, J., Dziubich, T. (2021). Segmentation Quality Refinement in Large-Scale Medical Image Dataset with Crowd-Sourced Annotations. In: Bellatreche, L., et al. New Trends in Database and Information Systems. ADBIS 2021. Communications in Computer and Information Science, vol 1450. Springer, Cham. https://doi.org/10.1007/978-3-030-85082-1_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-85082-1_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85081-4
Online ISBN: 978-3-030-85082-1
eBook Packages: Computer ScienceComputer Science (R0)