Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A real-time image-centric transfer function design based on incremental classification

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

A key issue in scientific visualization is the transfer function (TF) for direct volume rendering (DVR). The TF serves as a tool for translating data values into color and opacity, to visualize the relevant structures present in the volumetric data studied. An adequate transfer function should have a non-complicated interactive strategy for new users or even experts. Furthermore, it has to achieve high-quality and not time-consuming visualization. In this paper, we propose a novel image-centric method for the real-time generation of transfer functions. The method is based on incremental classification. This incremental classification-based approach is theoretically faster than that using batch classification. The method does not require users to manipulate complex widgets. We present a simple user interface adapted to the incremental learning process. Thus, this interface made it possible for the user to interact with a series of 2D images, precise the cluster, and identify some voxels. The whole volume is incrementally classified and the rendering result is shown to the user as selected voxels are added. The TF is generated by assigning the optical properties to clusters using harmonic colors. We further introduce a novel incremental classifier, namely incremental discriminant-based support vector machine( IDSVM), that can learn through time. The IDSVM was used in the classification stage of the proposed image-centric method. To evaluate the IDSVM, an extensive comparison of the model with other state-of-the-art incremental and batch classifiers on 12 real-world datasets and four other famous large datasets, namely MNIST-full, MNIST-test, USPS, and Fashion-MNIST, has been carried out. Using the area under curve, it has been found that the IDSVM outperforms the other classifiers. Furthermore, to evaluate the proposed image-centric method, we made use of several benchmark datasets. Qualitative results and a detailed user survey demonstrate the effectiveness of the proposed method and the positive effect of the incrementality in visual and interaction time performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Card, S.K., Mackinlay, J.D., Shneiderman, B.: Readings in information visualization: using vision to think. Academic Press (1999), ISBN 978-1-55860-533-6, pp. I–XVII, 1–686 (1999)

  2. Johnson, C.: Top scientific visualization research problems. IEEE Comput. Graph. Appl. 24(4), 13–17 (2004)

    Article  Google Scholar 

  3. Schultz, T., Kindlmann, G.L.: Open-box spectral clustering: applications to medical image analysis. IEEE Tran. Vis. Comput. Graph. 19(12), 2100–2108 (2013)

    Article  Google Scholar 

  4. Silva, S., Santos, B.S., Madeira, J.: Using color in visualization: a survey. Comput. Graph. 35(2), 320–333 (2011)

    Article  Google Scholar 

  5. Li, L., Peng, H., Chen, X., Cheng, J., Gao, D.: Visualization of boundaries in volumetric data sets through a what material you pick is what boundary you see approach. Comput. Methods Progr. Biomed. 126, 76–88 (2016)

    Article  Google Scholar 

  6. Liu, Y., Lisle, C., Collins, J.: Quick2insight: A user-friendly framework for interactive rendering of biological image volumes. In: 2011 IEEE Symposium on Biological Data Visualization (BioVis)., pp. 1–8. IEEE (2011)

  7. Kniss, J., Kindlmann, G., Hansen, C.: Interactive volume rendering using multi-dimensional transfer functions and direct manipulation widgets. In: Proceedings Visualization, 2001. VIS’01., pp. 255–562. IEEE (2001)

  8. Hong, F., Liu, C., Yuan, X.: Dnn-volvis: Interactive volume visualization supported by deep neural network. In: 2019 IEEE Pacific Visualization Symposium (PacificVis), pp. 282–291. IEEE (2019)

  9. Khan, N.M., Ksantini, R., Guan, L.: A novel image-centric approach toward direct volume rendering. ACM Trans. Intell. Syst. Technol. 9(4), 1–18 (2018)

    Article  Google Scholar 

  10. Studholme, C., Hill, D.L., Hawkes, D.J.: An overlap invariant entropy measure of 3d medical image alignment. Pattern Recogn. 32(1), 71–86 (1999)

    Article  Google Scholar 

  11. Khan, N.M., Ksantini, R., Ahmad, I.S., Guan, L.: Sn-svm: a sparse nonparametric support vector machine classifier. Signal Image Video Process. 8(8), 1625–1637 (2014)

    Article  Google Scholar 

  12. Zhang, B., Chen, X., Shan, S., Gao, W.: Nonlinear face recognition based on maximum average margin criterion. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 554–559. IEEE (2005)

  13. Shivaswamy, P.K., Jebara, T.: Maximum relative margin and data-dependent regularization. J. Mach. Learn. Res. 11(2), 747–788 (2010)

  14. Correa, C., Ma, K.L.: Size-based transfer functions: a new volume exploration technique. IEEE Trans. Vis. Comput. Graph. 14(6), 1380–1387 (2008)

    Article  Google Scholar 

  15. Kindlmann, G., Durkin, J.W.: Semi-automatic generation of transfer functions for direct volume rendering. In: IEEE Symposium on Volume Visualization (Cat. No. 989EX300), pp. 79–86. IEEE (1998)

  16. Mesquita, R., Celes, W.: Robust and effective method for automatic generation of one-dimensional transfer functions. In: 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 92–99. IEEE (2019)

  17. Kindlmann, G., Whitaker, R., Tasdizen, T., Moller, T.: Curvature-based transfer functions for direct volume rendering: Methods and applications. In: 14th IEEE Visualization Conference, IEEE Vis 2003, Seattle, WA, USA, October 19–24, 2003, Proceedings, pp. 513–520. IEEE (2003)

  18. Engel, D., Ropinski, T.: Deep volumetric ambient occlusion. arXiv preprint arXiv:2008.08345 (2020)

  19. Maciejewski, R., Jang, Y., Woo, I., Jänicke, H., Gaither, K.P., Ebert, D.S.: Abstracting attribute space for transfer function exploration and design. IEEE Trans. Vis. Comput. Graph. 19(1), 94–107 (2012)

    Article  Google Scholar 

  20. Lundstrom, C., Ljung, P., Ynnerman, A.: Local histograms for design of transfer functions in direct volume rendering. IEEE Trans. Vis. Comput. Graph. 12(6), 1570–1579 (2006)

    Article  Google Scholar 

  21. Selver, M.A., Guzelis, C.: Semiautomatic transfer function initialization for abdominal visualization using self-generating hierarchical radial basis function networks. IEEE Trans. Vis. Comput. Graph. 15(3), 395–409 (2009)

    Article  Google Scholar 

  22. Athawale, T.M., Ma, B., Sakhaee, E., Johnson, C.R., Entezari, A.: Direct volume rendering with nonparametric models of uncertainty. IEEE Trans. Vis. Comput. Graph. 27(2), 1797–1807 (2020)

    Article  Google Scholar 

  23. Sereda, P., Bartroli, A.V., Serlie, I.W., Gerritsen, F.A.: Visualization of boundaries in volumetric data sets using lh histograms. IEEE Transactions on Visualization and Computer Graphics 12(2), 208–218 (2006)

    Article  Google Scholar 

  24. Maciejewski, R., Woo, I., Chen, W., Ebert, D.: Structuring feature space: a non-parametric method for volumetric transfer function generation. IEEE Trans. Vis. Comput. Graph. 15(6), 1473–1480 (2009)

    Article  Google Scholar 

  25. Khan, N.M., Kyan, M., Guan, L.: Intuitive volume exploration through spherical self-organizing map and color harmonization. Neurocomputing 147, 160–173 (2015)

    Article  Google Scholar 

  26. Ge, F., Noël, R., Navarro, L., Courbebaisse, G.: Volume Rendering and Lattice-Boltzmann Method. GRETSI 2017, ID168, Juan Les Pins (France) (2017)

  27. Sharma, O., Arora, T., Khattar, A.: Computer Graphics Forum. Graph-based transfer function for volume rendering, pp. 76–88. Wiley Online Library (2020)

    Google Scholar 

  28. Ma, J., Muad, Y.A., Chen, J.: Visualization of medical volume data based on improved k-means clustering and segmentation rules. IEEE Access 9, 100498–100512 (2021)

  29. Yang, F., Meng, X., Lang, J., Lu, W., Liu, L.: Region space guided transfer function design for nonlinear neural network augmented image visualization. Adv. Multimedia 2018, 7479316:1–7479316:8 (2018)

  30. Weiss, J., Navab, N.: Deep direct volume rendering: Learning visual feature mappings from exemplary images. arXiv preprint arXiv:2106.05429 (2021)

  31. Torayev, A., Schultz, T.: Interactive classification of multi-shell diffusion MRI with features from a dual-branch CNN autoencoder. In: EG Workshop on Visual Computing for Biology and Medicine, pp. 1–11 (2020)

  32. Fang, S., Biddlecome, T., Tuceryan, M.: Image-based transfer function design for data exploration in volume visualization. In: Proceedings Visualization’98 (Cat. No. 98CB36276), pp. 319–326. IEEE (1998)

  33. Marks, J., Andalman, B., Beardsley, P.A., Freeman, W., Gibson, S., Hodgins, J., Kang, T., Mirtich, B., Pfister, H., Ruml, W., et al.: Design galleries: A general approach to setting parameters for computer graphics and animation. In: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pp. 389–400 (1997)

  34. Ropinski, T., Praßni, J.S., Steinicke, F., Hinrichs, K.H.: Stroke-based transfer function design. In: Eurographics / IEEE VGTC Symposium on Volume and Point-Based Graphics - 7th International Symposium on Volume Graphics, 5th Symposium on Point-Based Graphics, VG/PBG@SIGGRAPH 2008, Los Angeles, CA, USA, August 10–11, 2008, pp. 41–48. Citeseer, Eurographics Association (2008)

  35. Guo, H., Li, W., Yuan, X.: Transfer function map. In: 2014 IEEE Pacific Visualization Symposium, pp. 262–266. IEEE (2014)

  36. Guo, H., Mao, N., Yuan, X.: Wysiwyg (what you see is what you get) volume visualization. IEEE Trans. Vis. Comput. Graph. 17(12), 2106–2114 (2011)

    Article  Google Scholar 

  37. Guo, H., Yuan, X.: Local wysiwyg volume visualization. In: 2013 IEEE Pacific Visualization Symposium (PacificVis), pp. 65–72. IEEE (2013)

  38. Binyahib, R., Peterka, T., Larsen, M., Ma, K.L., Childs, H.: A scalable hybrid scheme for ray-casting of unstructured volume data. IEEE transactions on visualization and computer graphics 25(7), 2349–2361 (2018)

    Article  Google Scholar 

  39. Berger, M., Li, J., Levine, J.A.: A generative model for volume rendering. IEEE Trans. Vis. Comput. Graph. 25(4), 1636–1650 (2019)

    Article  Google Scholar 

  40. Shi, N., Tao, Y.: Cnns based viewpoint estimation for volume visualization. ACM Trans. Intell. Syst. Technol. 10(3), 1–22 (2019)

    Article  Google Scholar 

  41. Luo, D., Lin, Y., Zhang, J.: Gpu-based multi-slice per pass algorithm in interactive volume illumination rendering. Front. Inf. Technol. Electron. Eng. 22(8), 1092–1103 (2021)

    Article  Google Scholar 

  42. Zhan, X., Ma, B.: Kernel nonparametric discriminant analysis. In: 2011 International Conference on Electrical and Control Engineering, pp. 4544–4547. IEEE (2011)

  43. Tokumaru, M., Muranaka, N., Imanishi, S.: Color design support system considering color harmony. In: 2002 IEEE world congress on computational intelligence. 2002 IEEE international conference on fuzzy systems. FUZZ-IEEE’02. Proceedings (Cat. No. 02CH37291), vol. 1, pp. 378–383. IEEE (2002)

  44. Zhou, J., Takatsuka, M.: Automatic transfer function generation using contour tree controlled residue flow model and color harmonics. IEEE Trans. Vis. Comput. Graph. 15(6), 1481–1488 (2009)

    Article  Google Scholar 

  45. Itten, J., van Haagen, E.: The Art of color: the subjective experience and objective rationale of color. Van Nostrand Reinhold (1991)

    Google Scholar 

  46. Soula, A., Tbarki, K., Ksantini, R., Said, S.B., Lachiri, Z.: A novel incremental kernel nonparametric svm model (ikn-svm) for data classification: an application to face detection. Eng. Appl. Artif. Intell. 89, 103468 (2020)

    Article  Google Scholar 

  47. Asuncion, A., Newman, D., Bache, K., Lichman, M.: UCI Machine Learning Repository. Meta, vol. 2003. University of California, Irvine, School of Information and Computer (2003)

  48. LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database (2010). AT&T Labs Online. Available: http://yann.lecun.com/exdb/mnist

  49. Hull, J.J.: A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell. 16(5), 550–554 (1994)

    Article  Google Scholar 

  50. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

  51. Bradley, A.P.: The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recogn. 30(7), 1145–1159 (1997)

    Article  Google Scholar 

  52. Yang, C., Bruzzone, L., Guan, R., Lu, L., Liang, Y.: Incremental and decremental affinity propagation for semisupervised clustering in multispectral images. IEEE Trans. Geosci. Remote Sens. 51(3), 1666–1679 (2012)

    Article  Google Scholar 

  53. Roettger, S.: Volume library (online). January 2012. Ohm Hochschule Nurnberg, Nurnberg, Germany. Available: http://www9.informatik.uni-erlangen.de/External/vollib

  54. KitwareTM. 2015. VolView. (2015). Retrieved July 10, 2021, from http://www.kitware.com/volview/.

Download references

Author information

Authors and Affiliations

  1. Mediatron Lab, Higher School of Communications of Tunis, University of Carthage, Tunis, Tunisia

    Marwa Salhi & Belhassen Zouari

  2. Department of Computer Science, College of IT, University of Bahrain, Sakhir, Bahrain

    Riadh Ksantini

Authors
  1. Marwa Salhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Riadh Ksantini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Belhassen Zouari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marwa Salhi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salhi, M., Ksantini, R. & Zouari, B. A real-time image-centric transfer function design based on incremental classification. J Real-Time Image Proc 19, 185–203 (2022). https://doi.org/10.1007/s11554-021-01176-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01176-x

Keywords

Search

Navigation