Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Simultaneously retargeting and super-resolution for stereoscopic video

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel approach that is able to resize stereoscopic video to fit various display environments with different aspect-ratios, while preserving the prominent content, keeping temporally consistent, adapting depth, as well as increasing the resolution. Our proposed approach can deal with retargeting and super-resolution problems simultaneously via replacing the down-sampling matrix appearing in super-resolution algorithm with a novel one, named as content-aware-sampling matrix, derived from retargeting method. The new matrix can sample the image into any resolution while preserving its important information as much as possible. Our approach can be roughly subdivided into three steps. In the first step, we calculate the overall saliency map for a shot, while considering the conspicuous information such as motion, depth, and structures. In the second step, given a certain resolution, we compute the retargeting parameters by a global optimization and formulate them into a matrix. Finally, we substitute the matrix into the objective function of super-resolution to achieve high visual quality images with expected resolution. In addition, we propose a novel single image super-resolution method inspired by a blind image deblurring method. The experimental results based on user studies verify the effectiveness of our approach. And the comparisons with the-state-of-the-art single image super-resolution methods validate the potential of our super-resolution method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://sp.cs.tut.fi/mobile3dtv/stereo-video/.

References

  1. Almeida MSC, Almeida LB (2010) Blind and semi-blind deblurring of natural images. IEEE Trans Image Process 19:36–52

    Article  MathSciNet  Google Scholar 

  2. Almeida MSC, Figueiredo MAT (2011) New stopping criteria for iterative blind image deblurring based on residual whiteness measures. In: IEEE statistical signal processing workshop (SSP)

  3. Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH 2007 26

  4. Bevilacqua M, Roumy A, More CGMLA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British machine vision conference (BMVC)

  5. Brust H, Tech G, Muller K (2009) Report on generation of mixed spatial resolution stereo data base. Tech. rep. MOBILE3DTV project

  6. Chang CH, Liang CK, Chuang YY (2011) Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans Multimed 13:589–601

    Article  Google Scholar 

  7. Dollar P, Zitnick CL (2013) Structured forests for fast edge detection. In: IEEE international conference on computer vision

  8. Farsiu S, Robinson MD, Elad M., Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13:1327–1344

    Article  Google Scholar 

  9. Garcia DC, Dorea C, de Queiroz RL (2012) Super resolution for multiview images using depth information. IEEE Trans Circ Syst Video Technol 22:1249–1256

    Article  Google Scholar 

  10. Guthier B, Kiess J, Kopf S, Effelsberg W (2013) Seam carving for stereoscopic video. In: IEEE IVMSP workshop

  11. He L, Qi H, Zaretzki R (2013) Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR)

  12. Kopf S, Guthier B, Hipp C, Kiess J, Effelsberg W (2014) Warping-based video retargeting for stereoscopic video. In: IEEE international conference on image processing (ICIP)

  13. Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10:1521– 1527

    Article  Google Scholar 

  14. Liu C. (2009) Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis. Massachusetts Institute of Technology

  15. Liu C, Sun D (2011) A bayesian approach to adaptive video super resolution. In: IEEE Conference on computer vision and pattern recognition (CVPR)

  16. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110

    Article  Google Scholar 

  17. Niu Y, Liu F, Feng WC, Jin H (2012) Aesthetics-based stereoscopic photo cropping for heterogeneous displays. IEEE Trans Multimed 14:783–796

    Article  Google Scholar 

  18. Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH 2008 27

  19. Timofte R, Smet VD, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International conference on computer vision (ICCV)

  20. Timofte R, Smet VD, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International conference on computer vision (ICCV)

  21. Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: 12th Asian conference on computer vision

  22. Utsugi K, Shibahara T, Koike T, Takahashi K, Naemura T (2010) Seam carving for stereo images. In: 3DTV-conference: the true vision - capture, transmission and display of 3D video (3DTV-CON)

  23. Villena S, Vega M, Molina R, Katsaggelos AK (2009) Bayesian super-resolution image reconstruction using an l1 prior. In: Proceedings of 6th international symposium on image and signal processing and analysis

  24. Wang YS, Tai CL, Sorkine O, Lee TY (2008) Optimized scale-and-stretch for image resizing. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH Asia 2008 27

  25. Yang CY, Yang MH (2013) Fast direct super-resolution by simple functions. In: IEEE International conference oncomputer vision (ICCV)

  26. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19:2861–2873

    Article  MathSciNet  Google Scholar 

  27. Yu J, Wang Z (2014) 3d facial motion tracking by combining online appearance model and cylinder head model in particle filtering. Sci Chin Inf Sci 57:274–280

    Google Scholar 

  28. Yu J, Wang Z (2015) A video, text and speech driven realistic 3d virtual head for human-machine interface. IEEE Trans Cybern 45:977–988

    Google Scholar 

  29. Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse representations. Curves Surf 2011 6920:711–730

    MathSciNet  MATH  Google Scholar 

  30. Zhang J, Cao Y, Wang Z (2013) A simultaneous method for 3d video super-resolution and high-quality depth estimation. In: IEEE International conference on image processing (ICIP)

  31. Zhang J, Cao Y, Zha ZJ, Zheng Z, Chen CW, Wang Z (2014) A unified scheme for super-resolution and depth estimation from asymmetric stereoscopic video. IEEE Trans Circ Syst Video Technol. doi:10.1109/TCSVT.2014.2367356

  32. Zhang J, Cao Y, Zheng Z, Chen C, Wang Z (2014) A new closed loop method of super-resolution for multi-view images. Mach Vis Appl 25:1685–1695

    Article  Google Scholar 

  33. Zhu Y, Zhang Y, Yuille AL (2014) Single image super-resolution using deformable patches. In: IEEE Conference on computer vision and pattern recognition (CVPR)

Download references

Author information

Authors and Affiliations

  1. Department of Automation, University of Science and Technology of China, Hefei, Anhui, People’s Republic of China

    Kai Kang, Yang Cao & Zengfu Wang

  2. Institute of Intelligent Machines, Chinese Academy of Sciences, HeiFei, Anhui, People’s Republic of China

    Zengfu Wang

Authors
  1. Kai Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zengfu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zengfu Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, K., Cao, Y. & Wang, Z. Simultaneously retargeting and super-resolution for stereoscopic video. Multimed Tools Appl 76, 11081–11095 (2017). https://doi.org/10.1007/s11042-015-3226-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-3226-9

Keywords

Search

Navigation