research-article

High-Quality Depth Estimation Using an Exemplar 3D Model for Stereo Conversion

Authors:

Junyong NohAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 21, Issue 7

Pages 835 - 847

https://doi.org/10.1109/TVCG.2015.2398440

Published: 01 July 2015 Publication History

Abstract

High-quality depth painting for each object in a scene is a challenging task in 2D to 3D stereo conversion. One way to accurately estimate the varying depth within the object in an image is to utilize existing 3D models. Automatic pose estimation approaches based on 2D-3D feature correspondences have been proposed to obtain depth from a given 3D model. However, when the 3D model is not identical to the target object, previous methods often produce erroneous depth in the vicinity of the silhouette of the object. This paper introduces a novel 3D model-based depth estimation method that effectively produces high-quality depth information for rigid objects in a stereo conversion workflow. Given an exemplar 3D model and user correspondences, our method generates detailed depth of an object by optimizing the initial depth obtained by the application of structural fitting and silhouette matching in the image domain. The final depth is accurate up to the given 3D model, while consistent with the image. Our method was applied to various image sequences containing objects with different appearances and varying poses. The experiments show that our method can generate plausible depth information that can be utilized for high-quality 2D to 3D stereo conversion.

References

[1]

O. Wang, M. Lang, M. Frei, A. Hornung, A. Smolic, and M. Gross, “Stereobrush: Interactive 2d to 3d conversion using discontinuous warps,” in Proc. EUROGRAPHICS Symp. Sketch-Based Interfaces Model., 2011, pp. 45 –74.

[2]

M. Guttmann, L. Wolf, and D. Cohen-Or, “ Semi-automatic stereo extraction from video footage,” in Proc. IEEE 12th Int. Conf. Comput. Vis., Oct. 2009, pp. 136–142.

[3]

R. B. Ribera, S. Choi, Y. Kim, J. Lee, and J. Noh, “Video panorama for 2d to 3d conversion,” Comput. Graph. Forum, vol. 31, no. 7, pp. 2067–2076, 2012.

[4]

B. Ward, S. B. Kang, and E. Bennett, “Depth director: A system for adding depth to movies,” IEEE Comput. Graph. Appl., vol. 31, no. 1, pp. 36–48, Jan./Feb. 2011.

Digital Library

[5]

A. P. V. Pernis and M. S. DeJohn, “Dimensionalization: Converting 2d films to 3d,” in Proc. Stereoscopic Displays and Appl. XIX, vol. 6803, pp. 68 030T- 68 030T-5.

[6]

H. Hwang, K. Kim, R. B. i. Ribera, and J. Noh, “Stereoscopic image generation of background terrain scenes,” Comput. Animation Virtual Worlds, vol. 22, nos. 2-3, pp. 317–323, 2011.

Digital Library

[7]

A. McKenzie, E. Vendrovsky, and J. Noh, “ Terrain geometry from monocular image sequences.” J. Comput. Sci. Eng., vol. 2, no. 1, pp. 98–108, 2008.

[8]

B. M. Oh, M. Chen, J. Dorsey, and F. Durand, “Image-based modeling and photo editing,” in Proc. 28th Annu. Conf. Computer Graphics and Interactive Techniques, ser. SIGGRAPH ’01, New York, NY, USA: ACM, 2001, pp. 433–442.

Digital Library

[9]

V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in Proc. 26th Annu. Conf. Computer Graphics and Interactive Techniques, ser. SIGGRAPH ’99, ACM, 1999, pp. 187–194.

Digital Library

[10]

M. Leotta and J. Mundy, “Predicting high resolution image edges with a generic, adaptive, 3-d vehicle model,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1311–1318.

[11]

P. Guan, A. Weiss, A. O. Balan, and M. J. Black, “Estimating human shape and pose from a single image,” in Proc. Int. Conf. Comput. Vis., 2009, pp. 1381–1388.

[12]

I. Matthews and J. Xiao, “2d vs. 3d deformable face models: Representational power, construction, and real-time fitting,” Int. J. Comput. Vis., vol. 75, 2007, pp. 93–113.

Digital Library

[13]

A. Levin, D. Lischinski, and Y. Weiss, “ Colorization using optimization,” ACM Trans. Graph., vol. 23, no. 3, pp. 689–694, Aug. 2004.

Digital Library

[14]

J. Wang and M. Cohen, “Optimized color sampling for robust matting,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1–8.

[15]

Y. Seol, J. Seo, P. H. Kim, J. P. Lewis, and J. Noh, “Artist friendly facial animation retargeting,” in Proc. SIGGRAPH Asia Conf., ser. SA ’11, ACM, 2011, pp. 162:1–162:10.

[16]

V. C. Orvalho, E. Zacur, and A. Susin, “Transferring the rig and animations from a character to different face models,” Comput. Graph. Forum , pp. 1997–2012, 2008.

[17]

R. M. Murray, Z. Li, and S. S. Sastry, “A mathematical introduction to robotic manipulation,” CRC Press; 1 edition, p. 480, 1994.

[18]

O. Sorkine, D. Cohen-Or, Y. Lipman, M. Alexa, C. Rössl, and H.-P. Seidel, “Laplacian surface editing,” in Proc. Eurographics/ACM SIGGRAPH Symp. Geometry Process., 2004, pp. 179–188.

[19]

A. Nealen, O. Sorkine, M. Alexa, and D. Cohen-Or, “A sketch-based interface for detail-preserving mesh editing,” in ACM SIGGRAPH 2005 Papers, ser. SIGGRAPH ’05, New York, NY, USA: ACM, 2005, pp. 1142–1147.

Digital Library

[20]

D. Vlasic, I. Baran, W. Matusik, and J. Popović, “Articulated mesh animation from multi-view silhouettes,” ACM Trans. Graph., vol. 27, pp. 97:1–97:9, 2008.

[21]

V. Kraevoy, A. Sheffer, and M. van de Panne, “ Modeling from contour drawings,” in Proc. 6th Eurographics Symp. Sketch-Based Interfaces Model., ser. SBIM ’09, New York, NY, USA: ACM, 2009, pp. 37–44.

Digital Library

[22]

K. Xu, H. Zheng, H. Zhang, D. Cohen-Or, L. Liu, and Y. Xiong, “ Photo-inspired model-driven 3d object modeling,” in ACM SIGGRAPH 2011 papers, ser. SIGGRAPH ’11, New York, NY, USA: ACM, 2011, pp. 80:1–80:10.

[23]

N. Kholgade, T. Simon, A. Efros, and Y. Sheikh, “3d object manipulation in a single photograph using stock 3d models,” ACM Trans. Graph., vol. 33, no. 4, pp. 127:1–127:12, Jul. 2014.

[24]

R.-F. Tong, Y. Zhang, and K.-L. Cheng, “ Stereopasting: Interactive composition in stereoscopic images,” IEEE Trans. Vis. Comput. Graph., vol. 19, no. 8, pp. 1375–1385, Aug. 2013.

Digital Library

[25]

M. Lang, A. Hornung, O. Wang, S. Poulakos, A. Smolic, and M. Gross, “Nonlinear disparity mapping for stereoscopic 3d,” ACM Trans. Graph., vol. 29, no. 4, pp. 75:1–75:10, Jul. 2010.

[26]

S. Lee, Y. Kim, J. Lee, K. Kim, K. Lee, and J. Noh, “ Depth manipulation using disparity histogram analysis for stereoscopic 3d,” The Visual Computer, vol. 30, no. 4, pp. 455–465, 2014.

Digital Library

[27]

L. Zhang and W. J. Tam, “Stereoscopic image generation based on depth images for 3d tv,” IEEE Trans. Broadcasting, vol. 51, no. 2, pp. 191– 199, May 2005.

[28]

C. Fehn, “Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv,” in Proc. SPIE Stereoscopic Displays Virtual Reality Syst. XI, 2004, vol. 5291, pp. 93–104.

[29]

A. Smolic, P. Kauff, S. Knorr, A. Hornung, M. Kunter, M. Muller, and M. Lang, “ Three-dimensional video postproduction and processing,” Proc. IEEE, vol. 99, no. 4, pp. 607–625, Mar. 2011.

[30]

A. Criminisi, P. Perez, and K. Toyama, “Object removal by exemplar-based inpainting,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2, Jun. 2003, pp. II–721 –II–728.

[31]

L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk, “State of the art in example-based texture synthesis,” in Eurographics ’09 State of the Art Reports (STARs), Eurographics, Mar. 2009.

[32]

B. Rosenhahn and G. Sommer, “Pose estimation of free-form objects,” in Proc. Comput. Vis. ECCV 2004, ser. Lecture Notes Comput. Sci., 2004, vol. 3021, pp. 414 –427.

[33]

B. Rosenhahn, T. Brox, and J. Weickert, “ Three-dimensional shape knowledge for joint image segmentation and pose tracking,” Int. J. Comput. Vis., vol. 73, pp. 243–262, 2007 .

Digital Library

Cited By

Hsia SWang STsai H(2022)Real-time 2D to 3D Image Conversion Algorithm and VLSI Architecture for Natural SceneCircuits, Systems, and Signal Processing10.1007/s00034-022-01983-y41:8(4455-4478)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00034-022-01983-y
Dong TQi XZhang QLi WXiong L(2019)Overview on Vision-Based 3D Object Recognition MethodsImage and Graphics10.1007/978-3-030-34110-7_21(243-254)Online publication date: 23-Aug-2019
https://dl.acm.org/doi/10.1007/978-3-030-34110-7_21

Index Terms

High-Quality Depth Estimation Using an Exemplar 3D Model for Stereo Conversion
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
  2. Computer graphics
    1. Image manipulation
    2. Shape modeling

Index terms have been assigned to the content through auto-classification.

Recommendations

Gradient-based 2D-to-3D Conversion for Soccer Videos
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

A wide spread adoption of 3D videos and technologies is hindered by the lack of high-quality 3D content. One promising solution to address this problem is to use automated 2D-to-3D conversion. However, current conversion methods, while general, produce ...
Depth estimation for semi-automatic 2D to 3D conversion
MM '12: Proceedings of the 20th ACM international conference on Multimedia

The conversion of monoscopic footage into stereoscopic or multiview content is a difficult and time consuming task. A number of semi-automatic methods have been developed to speed up the process and provide some control to the user. However these ...
Development of Semi-Automatic Methodology for Extraction of Depth for 2D-to-3D Conversion
ICMLC '17: Proceedings of the 9th International Conference on Machine Learning and Computing

Recently, stereoscopic image/video conversion has been most demanding as it has a great stereo video experience. This paper presents the development of the methodology for generation of the depth map used for stereoscopic view. Normally, Stereoscopic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 21, Issue 7

July 2015

104 pages

ISSN:1077-2626

Issue’s Table of Contents

Copyright © 2015.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 July 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hsia SWang STsai H(2022)Real-time 2D to 3D Image Conversion Algorithm and VLSI Architecture for Natural SceneCircuits, Systems, and Signal Processing10.1007/s00034-022-01983-y41:8(4455-4478)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00034-022-01983-y
Dong TQi XZhang QLi WXiong L(2019)Overview on Vision-Based 3D Object Recognition MethodsImage and Graphics10.1007/978-3-030-34110-7_21(243-254)Online publication date: 23-Aug-2019
https://dl.acm.org/doi/10.1007/978-3-030-34110-7_21

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents