Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

An improved neural network-based saliency detection scheme for light field images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a class weight-optimized Convolutional Neural Network (CNN) architecture for light field all-in-focus image-based saliency detection. The proposed architecture uses a novel technique based on the Co-occurrence matrix and Grey Wolf Optimization to optimize the class weights of the proposed CNN’s loss function. An improved guided filter-based image fusion is implemented for the fusion of sub-aperture light field images into an all-in-focus image. These all-in-focus images are appended to the existing dataset along with other adversarial samples to make the dataset more varied and generalizable. F-measure, E-measure, S- measure and Mean Absolute Error are the metrics used for model evaluation. The proposed technique efficiently uses all-in-focus and focal stack light field images to extract salient regions without imposing heavy computational requirements. According to simulation results, effective weight initialization increases model performance and reduces training time since it promotes faster convergence. The proposed saliency detection model achieved an average increase of 13.97% in the F-measure and an average decrease of 39.83% in the Mean Absolute Error when compared to the state-of-the-art models discussed here. The class weight optimization logic achieved a reduction of 27% in the training time required. The improved guided filter fusion contributed an increase of 10.40% in PSNR and a decrease of 34.75% in Maximum Difference when compared with the conventional guided filter fusion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The datasets analyzed during the current study are available at   https://www.eecis.udel.edu/~nianyi/LFSD.htm [42]. https://github.com/OIPLabDUT/ICCV2019_Deeplightfield_Saliency [34].

References

  1. Duan F, Wu Y, Guan H, Wu C (2022) Saliency detection of light field images by fusing focus degree and GrabCut. Sensors 22(19):7411

    Article  Google Scholar 

  2. Jiang B, Xu D, Shi J (2022, October) Light field saliency detection based on multi-modal fusion. In: Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering (pp 253–260)

  3. Feng M, Liu K, Zhang L, Yu H, Wang Y, Mian A (2022) Learning from pixel-level noisy label: a new perspective for light field saliency detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit, pp 1756–1766

  4. Gao SH, Tan YQ, Cheng MM, Lu C, Chen Y, Yan S (2020) Highly efficient salient object detection with 100k parameters. European Conference on Computer Vision. Springer International Publishing, Cham, pp 702–721

    Google Scholar 

  5. Zhang J, Yu X, Li A, Song P, Liu B, Dai Y (2020) Weakly-supervised salient object detection via scribble annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 12546–12555)

  6. Yu S, Zhang B, Xiao J, Lim EG (2021) Structure-consistent weakly supervised salient object detection with local saliency coherence. In Proc AAAI Conference Artif Intell 35(4):3234–3242

    Google Scholar 

  7. Liu N, Zhao W, Zhang D, Han J, Shao L (2021) Light field saliency detection with dual local graph learning and reciprocative guidance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (pp 4712–4721)

  8. Wang X, Dong Y, Zhang Q, Wang Q (2021) Region-based depth feature descriptor for saliency detection on light field. Multimed Tools App 80(11):16329–16346

    Article  Google Scholar 

  9. Zhang J, Liu Y, Zhang S, Poppe R, Wang M (2020) Light field saliency detection with deep convolutional networks. IEEE Trans Image Process 29:4421–4434

    Article  Google Scholar 

  10. Piao Y, Rong Z, Zhang M, Li X, Lu H (2019) Deep Light-field-driven Saliency Detection from a Single View. In IJCAI (pp 904–911)

  11. Li N, Sun B, Yu J (2015) A weighted sparse coding framework for saliency detection. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 5216–5223

  12. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615

    Article  Google Scholar 

  13. Zhang M, Ji W, Piao Y, Li J, Zhang Y, Xu S, Lu H (2020) LFNet: Light field fusion network for salient object detection. IEEE Trans Image Process 29:6276–6287. https://doi.org/10.1109/TIP.2020.2990341

    Article  Google Scholar 

  14. Piao Y, Jiang Y, Zhang M, Wang J, Lu H (2021) PANet: Patch-aware network for light field salient object detection. IEEE Trans Cybernetics. https://doi.org/10.1109/TCYB.2021.3095512

    Article  Google Scholar 

  15. Wang A (2020) Three-stream cross-modal feature aggregation network for light field salient object detection. IEEE Signal Process Lett 28:46–50. https://doi.org/10.1109/LSP.2020.3044544

    Article  Google Scholar 

  16. Hou Q, Cheng MM, Hu X, Borji A, Tu Z, Torr PH (2017) Deeply supervised salient object detection with short connections. In Proc IEEE Conference Comput Vision Pattern Recognition 41:3203–3212. https://doi.org/10.1109/TPAMI.2018.2815688

    Article  Google Scholar 

  17. Zhang X, Wang T, Qi J, Lu H, Wang G (2018) Progressive attention guided recurrent network for salient object detection. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 714–722. https://doi.org/10.1109/CVPR.2018.00081

  18. Qu L, He S, Zhang J, Tian J, Tang Y, Yang Q (2017) RGBD salient object detection via deep fusion. IEEE Trans Image Process 26(5):2274–2285. https://doi.org/10.1109/TIP.2017.2682981

    Article  MathSciNet  Google Scholar 

  19. Zhang J, Wang M, Lin L, Yang X, Gao J, Rui Y (2017) Saliency detection on light field: A multi-cue approach. ACM Trans Multimed Comput Commun Appl (TOMM) 13(3):1–22. https://doi.org/10.1145/3107956

    Article  Google Scholar 

  20. Ju R, Liu Y, Ren T, Ge L, Wu G (2015) Depth-aware salient object detection using anisotropic center-surround difference. Signal Process: Image Commun 38:115–126. https://doi.org/10.1016/j.image.2015.07.002

    Article  Google Scholar 

  21. Li G, Yu Y (2016) Deep contrast learning for salient object detection. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 478–487. https://doi.org/10.48550/arXiv.1603.01976

  22. Qin Y, Lu H, Xu Y, Wang H (2015) Saliency detection via cellular automata. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 110–119. https://doi.org/10.1109/CVPR.2015.7298606

  23. Tu WC, He S, Yang Q, Chien SY (2016) Real-time salient object detection with a minimum spanning tree. In Proc IEEE Conference Comput Vision Pattern Recognition 1:2334–2342. https://doi.org/10.1109/CVPR.2016.256

    Article  Google Scholar 

  24. Fan DP, Lin Z, Zhang Z, Zhu M, Cheng MM (2020) Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks. IEEE Trans Neural Netw Learning Syst 32(5):2075–2089. https://doi.org/10.1109/TNNLS.2020.2996406

    Article  Google Scholar 

  25. Zhao JX, Cao Y, Fan DP, Cheng MM, Li XY, Zhang L (2019) Contrast prior and fluid pyramid integration for RGBD salient object detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit, pp 3927–3936. https://doi.org/10.1109/CVPR.2019.00405

  26. Han J, Chen H, Liu N, Yan C, Li X (2017) CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion. IEEE Trans Cybernetics 48(11):3171–3183. https://doi.org/10.1109/TCYB.2017.2761775

    Article  Google Scholar 

  27. Srinivasu PN, Ahmed S, Alhumam A, Kumar AB, Ijaz MF (2021) An aw-haris based automated segmentation of human liver using ct images. Comput Mater Continua 69(3):3303

    Article  Google Scholar 

  28. Srinivasu PN, Rao TS, Balas VE (2020) A systematic approach for identification of tumor regions in the human brain through HARIS algorithm. In: Deep Learning Techniques for Biomedical and Health Informatics (pp 97–118). Academic Press

  29. Ren J, Gong X, Yu L, Zhou W, Ying Yang M (2015) Exploiting global priors for RGB-D saliency detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp 25–32). https://doi.org/10.1109/CVPRW.2015.7301391

  30. Lang C, Nguyen TV, Katti H, Yadati K, Kankanhalli M, Yan S (2012) Depth matters: influence of depth cues on visual saliency. In: Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II 12 (pp 101–115). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-33709-3_8

  31. Desingh K, Krishna KM, Rajan D, Jawahar CV (2013, September) Depth really Matters: Improving Visual Salient Region Detection with Depth. In BMVC (pp 1–11). https://doi.org/10.5244/C.27.98

  32. Shigematsu R, Feng D, You S, Barnes N (2017) Learning RGB-D salient object detection using background enclosure, depth contrast, and top-down features. In: Proceedings of the IEEE international conference on computer vision workshops (pp 2749–2757). https://doi.org/10.1109/ICCVW.2017.323

  33. Zhang J, Wang M, Gao J, Wang Y, Zhang X, Wu X (2015) Saliency detection with a deeper investigation of light field. In IJCAI (pp 2212–2218)

  34. Wang T, Piao Y, Li X, Zhang L, Lu H (2019) Deep learning for light field saliency detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (pp 8838–8848). https://doi.org/10.1109/ICCV.2019.00893

  35. Mirjalili S, Mirjalili SM, Lewis A (2014) Grey wolf optimizer. Adv Eng Softw 69:46–61. https://doi.org/10.1016/j.advengsoft.2013.12.007

    Article  Google Scholar 

  36. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans Image Process 22(7):2864–2875. https://doi.org/10.1109/TIP.2013.2244222

    Article  Google Scholar 

  37. Kahraman HT, Aras S, Gedikli E (2020) Fitness-distance balance (FDB): a new selection method for meta-heuristic search algorithms. Knowl-Based Syst 190:105169. https://doi.org/10.1016/j.knosys.2019.105169

    Article  Google Scholar 

  38. Rao R (2016) Jaya: A simple and new optimization algorithm for solving constrained and unconstrained optimization problems. Int J Ind Eng Comput 7(1):19–34. https://doi.org/10.5267/j.ijiec.2015.8.004

    Article  Google Scholar 

  39. Srinivasu PN, SivaSai JG, Ijaz MF, Bhoi AK, Kim W, Kang JJ (2021) Classification of skin disease using deep learning neural networks with MobileNet V2 and LSTM. Sensors 21(8):2852

    Article  Google Scholar 

  40. Albregtsen F (2008) Statistical texture measures computed from gray level coocurrence matrices. Image processing laboratory, department of informatics, university of oslo 5(5)

  41. Li N, Ye J, Ji Y, Ling H, Yu J (2014) Saliency detection on light field. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 2806–2813. https://doi.org/10.1109/CVPR.2014.359

  42. Achanta R, Hemami S, Estrada F, Susstrunk S (2009, June) Frequency-tuned salient region detection. In 2009 IEEE Conf Comput Vis Pattern Recognit (pp 1597–1604). IEEE. https://doi.org/10.1109/CVPR.2009.5206596

  43. Fan DP, Cheng MM, Liu Y, Li T, Borji A (2017) Structure-measure: a new way to evaluate foreground maps. In: Proceedings of the IEEE international conference on computer vision (pp 4548–4557). https://doi.org/10.48550/arXiv.1708.00786

  44. Fan DP, Gong C, Cao Y, Ren B, Cheng MM, Borji A (2018) Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421

  45. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 (pp 234–241). Springer International Publishing

  46. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc IEEE Conf Comput Vis Pattern Recognit, pp 770–778

  47. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst 25

Download references

Funding

This study was partially funded by the Centre for Engineering Research and Development (CERD), APJ Abdul Kalam Technological University, Kerala, India.

Author information

Authors and Affiliations

  1. Department of Electronics and Communication, Mar Baselios College of Engineering and Technology, Thiruvananthapuram, Kerala, India

    Parvathy Prathap & J. Jayakumari

Authors
  1. Parvathy Prathap
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Jayakumari
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.P.- Conceptualization, Methodology, Formal Analysis, Writing original draft;J.J.—Supervision, Project Administration, Writing – review and editing.

Corresponding author

Correspondence to Parvathy Prathap.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prathap, P., Jayakumari, J. An improved neural network-based saliency detection scheme for light field images. Multimed Tools Appl 83, 56549–56567 (2024). https://doi.org/10.1007/s11042-023-17683-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17683-x

Keywords

Search

Navigation