Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Performance Capture from Multi-View Video

  • Chapter
  • First Online:
Image and Geometry Processing for 3-D Cinematography

Part of the book series: Geometry and Computing ((GC,volume 5))

Abstract

Nowadays, increasing performance of computing hardware makes it feasible to simulate ever more realistic humans even in real-time applications for the end-user. To fully capitalize on these computational resources, all aspects of the human, including textural appearance and lighting, and, most importantly, dynamic shape and motion have to be simulated at high fidelity in order to convey the impression of a realistic human being. In consequence, the increase in computing power is flanked by increasing requirements to the skills of the animators. In this chapter, we describe several recently developed performance capture techniques that enable animators to measure detailed animations from real world subjects recorded on multi-view video. In contrast to classical motion capture, performance capture approaches don’t only measure motion parameters without the use of optical markers, but also measure detailed spatio-temporally coherent dynamic geometry and surface texture of a performing subject. This chapter gives an overview of recent state-of-the-art performance capture approaches from the literature. The core of the chapter describes a new mesh-based performance capture algorithm that uses a combination of deformable surface and volume models for high-quality reconstruction of people in general apparel, i.e. also wide dresses and skirts. The chapter concludes with a discussion of the different approaches, pointers to additional literature and a brief outline of open research questions for the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

eBook
USD 15.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Ahmed, N., Theobalt, C., Rössl, C., Thrun, S., Seidel, H.P.: Dense correspondence finding for parametrization-free animation reconstruction from video. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2008), pp. 1–8 (2008)

    Google Scholar 

  2. Allen, B., Curless, B., Popović, Z.: Articulated body deformation from range scan data. ACM Trans. Graph. 21(3), 612–619 (2002)

    Article  Google Scholar 

  3. Anguelov, D., Koller, D., Srinivasan, P., Thrun, S., Pang, H.C., Davis, J.: The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces. In: Advances in Neural Information Processing Systems (NIPS 2004) (2004)

    Google Scholar 

  4. Balan, A.O., Sigal, L., Black, M.J., Davis, J.E., Haussecker, H.W.: Detailed human shape and pose from images. In: Proc. CVPR (2007)

    Google Scholar 

  5. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., Gross, M.: Multi-scale capture of facial geometry and motion. In: Proc. of SIGGRAPH, p. 33 (2007)

    Google Scholar 

  6. Botsch, M., Pauly, M., Wicke, M., Gross, M.: Adaptive space deformations based on rigid cells. Comput. Graph. Forum 26(3), 339–347 (2007)

    Article  Google Scholar 

  7. Botsch, M., Sorkine, O.: On linear variational surface deformation methods. IEEE Trans. Visual. Comput. Graph. 14(1), 213–230 (2008)

    Article  Google Scholar 

  8. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., Boubekeur, T.: Markerless garment capture. In: SIGGRAPH ’08: ACM SIGGRAPH 2008 Papers, pp. 1–9. ACM (2008)

    Google Scholar 

  9. Byrd, R., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comp. 16(5), 1190–1208 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  10. Carranza, J., Theobalt, C., Magnor, M., Seidel, H.P.: Free-viewpoint video of human actors. In: Proc. SIGGRAPH, pp. 569–577 (2003)

    Google Scholar 

  11. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. In: SIGGRAPH ’08: ACM SIGGRAPH 2008 papers, pp. 1–10. ACM (2008)

    Google Scholar 

  12. de Aguiar, E., Theobalt, C., Stoll, C., Seidel, H.: Marker-less 3d feature tracking for mesh-based human motion capture. In: Proc. ICCV Workshop on Human Motion, pp. 1–15 (2007)

    Google Scholar 

  13. de Aguiar, E., Theobalt, C., Stoll, C., Seidel, H.P.: Marker-less deformable mesh tracking for human shape and motion capture. In: Proc. CVPR, pp. 1–8. IEEE (2007)

    Google Scholar 

  14. de Aguiar, E., Theobalt, C., Thrun, S., Seidel, H.P.: Automatic conversion of mesh animations into skeleton-based animations. Comput. Graph. Forum (Proc. Eurographics EG’08) 27(2), 389–397 (2008)

    Google Scholar 

  15. Einarsson, P., Chabert, C.F., Jones, A., Ma, W.C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., Debevec, P.: Relighting human locomotion with flowed reflectance fields. In: Proc. Eurographics Symposium on Rendering, pp. 183–194 (2006)

    Google Scholar 

  16. Exluna, Inc.: Entropy 3.1 Technical Reference (2002)

    Google Scholar 

  17. Fedkiw, R., Stam, J., Jensen, H.W.: Visual simulation of smoke. In: Fiume, E. (ed.) Proceedings of SIGGRAPH, pp. 15–22. ACM (2001)

    Google Scholar 

  18. Goesele, M., Curless, B., Seitz, S.M.: Multi-view stereo revisited. In: Proc. CVPR, pp. 2402–2409 (2006)

    Google Scholar 

  19. Gross, M., Würmlin, S., Näf, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L.V., Lang, S., Strehlke, K., Moere, A.V., Staadt, O.: Blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans. Graph. 22(3), 819–827 (2003)

    Article  Google Scholar 

  20. Jobson, D.J., Rahman, Z., Woodell, G.A.: Retinex image processing: improved fidelity to direct visual observation. In: Proceedings of the IS&T Fourth Color Imaging Conference: Color Science, Systems, and Applications, vol. 4, pp. 124–125 (1995)

    Google Scholar 

  21. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: constructing virtual worlds from real scenes. Proc. IEEE MultiMedia 4(1), 34–47 (1997)

    Article  Google Scholar 

  22. Kartch, D.: Efficient rendering and compression for full-parallax computer-generated holographic stereograms. Ph.D. thesis, Cornell University (2000)

    Google Scholar 

  23. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proc. SGP, pp. 61–70 (2006)

    Google Scholar 

  24. Landis, H.: Global illumination in production. In: ACM SIGGRAPH 2002 Course #16 Notes (2002)

    Google Scholar 

  25. Leordeanu, M., Hebert, M.: A spectral technique for correspondence problems using pairwise constraints. In: Proc. ICCV (2005)

    Google Scholar 

  26. Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., Fulk, D.: The digital michelangelo project. In: Akeley, K. (ed.) Proceedings of SIGGRAPH, pp. 131–144 (2000)

    Google Scholar 

  27. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proc. ICCV, vol. 2, p. 1150ff (1999)

    Google Scholar 

  28. Matusik, W., Buehler, C., Raskar, R., Gortler, S., McMillan, L.: Image-based visual hulls. In: Proc. SIGGRAPH, pp. 369–374 (2000)

    Google Scholar 

  29. Menache, A.: Understanding Motion Capture for Computer Animation and Video Games. Morgan Kaufmann, San Francisco (1999)

    Google Scholar 

  30. Mitra, N.J., Flory, S., Ovsjanikov, M., Gelfand, N., AS, L.G., Pottmann, H.: Dynamic geometry registration. In: Proc. Symposium on Geometry Processing, pp. 173–182 (2007)

    Google Scholar 

  31. Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Understand. 104(2), 90–126 (2006)

    Article  Google Scholar 

  32. Müller, M., Dorsey, J., McMillan, L., Jagnow, R., Cutler, B.: Stable real-time deformations. In: Proc. of SCA, pp. 49–54. ACM (2002)

    Google Scholar 

  33. Nobuhara, S., Matsuyama, T.: Deformable mesh model for complex multi-object 3D motion estimation from multi-viewpoint video. In: 3DPVT06, pp. 264–271 (2006)

    Google Scholar 

  34. Paramount: Beowulf movie page. http://www.beowulfmovie.com/ (2007)

  35. Park, S.I., Hodgins, J.K.: Capturing and animating skin deformation in human motion. ACM Trans. Graph. (SIGGRAPH 2006) 25(3) (2006)

    Google Scholar 

  36. Park, S.W., Linsen, L., Kreylos, O., Owens, J.D., Hamann, B.: Discrete sibson interpolation. IEEE Trans. Visual. Comput. Graph. 12(2), 243–253 (2006)

    Article  Google Scholar 

  37. Parke, F.I., Waters, K.: Computer Facial Animation. A. K. Peters, Natick (1996)

    Google Scholar 

  38. Pellacini, F., Vidimče, K., Lefohn, A., Mohr, A., Leone, M., Warren, J.: Lpics: a hybrid hardware-accelerated relighting engine for computer cinematography. ACM Trans. Graph. 24(3), 464–470 (2005)

    Article  Google Scholar 

  39. Poppe, R.: Vision-based human motion analysis: an overview. Comput. Vis. Image Understand. 108(1–2), 4–18 (2007)

    Article  Google Scholar 

  40. Rosenhahn, B., Kersting, U., Powel, K., Seidel, H.P.: Cloth x-ray: Mocap of people wearing textiles. In: LNCS 4174: Proc. DAGM, pp. 495–504 (2006)

    Google Scholar 

  41. Sako, Y., Fujimura, K.: Shape similarity by homotropic deformation. Vis. Comput. 16(1), 47–61 (2000)

    Article  Google Scholar 

  42. Sand, P., McMillan, L., Popović, J.: Continuous capture of skin deformation. ACM Trans. Graph. 22(3) (2003)

    Google Scholar 

  43. Scholz, V., Stich, T., Keckeisen, M., Wacker, M., Magnor, M.: Garment motion capture using color-coded patterns. Comput. Graph. Forum (Proc. Eurographics EG’05) 24(3), 439–448 (2005)

    Google Scholar 

  44. Shinya, M.: Unifying measured point sequences of deforming objects. In: Proc. of 3DPVT, pp. 904–911 (2004)

    Google Scholar 

  45. Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Proc. SGP, pp. 109–116 (2007)

    Google Scholar 

  46. Starck, J., Hilton, A.: Spherical matching for temporal correspondence of non-rigid surfaces. In: IEEE Int. Conf. Computer Vision, pp. 1387–1394 (2005)

    Google Scholar 

  47. Starck, J., Hilton, A.: Correspondence labelling for wide-timeframe free-form surface matching. In: Proc. ICCV, pp. 1–8 (2007)

    Google Scholar 

  48. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)

    Article  Google Scholar 

  49. Stoll, C., de Aguiar, E., Theobalt, C., Seidel, H.P.: A volumetric approach to interactive shape editing. Research Report MPI-I-2007-4-004, Max-Planck-Institut für Informatik (2007)

    Google Scholar 

  50. Stoll, C., Karni, Z., Rössl, C., Yamauchi, H., Seidel, H.P.: Template deformation for point cloud fitting. In: Proc. SGP, pp. 27–35 (2006)

    Google Scholar 

  51. Sumner, R.W., Popović, J.: Deformation transfer for triangle meshes. In: SIGGRAPH ’04, pp. 399–405 (2004)

    Google Scholar 

  52. Vedula, S., Baker, S., Kanade, T.: Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Trans. Graph. 24(2), 240–261 (2005)

    Article  Google Scholar 

  53. Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. ACM Trans. Graph. 27(3), 1–9 (2008)

    Article  Google Scholar 

  54. Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., Schilling, A.: Reconstruction of deforming geometry from time-varying point clouds. In: Proc. SGP, pp. 49–58 (2007)

    Google Scholar 

  55. Waschbüsch, M., Würmlin, S., Cotting, D., Sadlo, F., Gross, M.: Scalable 3D video of dynamic scenes. In: Proc. Pacific Graphics, pp. 629–638 (2005)

    Google Scholar 

  56. White, R., Crane, K., Forsyth, D.: Capturing and animating occluded cloth. In: ACM TOG (Proc. SIGGRAPH) (2007)

    Google Scholar 

  57. Wilburn, B., Joshi, N., Vaish, V., Talvala, E., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graph. 24(3), 765–776 (2005)

    Article  Google Scholar 

  58. Xu, W., Zhou, K., Yu, Y., Tan, Q., Peng, Q., Guo, B.: Gradient domain editing of deforming mesh sequences. In: Proc. SIGGRAPH, p. 84ff. ACM (2007)

    Google Scholar 

  59. Yamauchi, H., Gumhold, S., Zayer, R., Seidel, H.P.: Mesh segmentation driven by gaussian curvature. Vis. Comput. 21(8–10), 649–658 (2005)

    Google Scholar 

  60. Yee, Y.L.H.: Spatiotemporal sensistivity and visual attention for efficient rendering of dynamic environments. Master’s thesis, Cornell University (2000)

    Google Scholar 

  61. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MPI Informatik, Saarbruecken, Germany

    Christian Theobalt, Edilson de Aguiar, Carsten Stoll & Hans-Peter Seidel

  2. Stanford University, Stanford, CA, USA

    Christian Theobalt & Sebastian Thrun

Authors
  1. Christian Theobalt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edilson de Aguiar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carsten Stoll
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hans-Peter Seidel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sebastian Thrun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Theobalt .

Editor information

Editors and Affiliations

  1. LEAR Team, INRIA, avenue de l'Europe, Montbonnot, 38330, France

    Rémi Ronfard

  2. Div. Engineering, Brown University, Providence, 02912, USA

    Gabriel Taubin

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Theobalt, C., de Aguiar, E., Stoll, C., Seidel, HP., Thrun, S. (2010). Performance Capture from Multi-View Video. In: Ronfard, R., Taubin, G. (eds) Image and Geometry Processing for 3-D Cinematography. Geometry and Computing, vol 5. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12392-4_6

Download citation

Publish with us

Policies and ethics

Search

Navigation