Article

Free-viewpoint video of human actors

Authors:

Christian Theobalt,

Marcus A. Magnor,

Hans-Peter SeidelAuthors Info & Claims

SIGGRAPH '03: ACM SIGGRAPH 2003 Papers

Pages 569 - 577

https://doi.org/10.1145/1201775.882309

Published: 01 July 2003 Publication History

Abstract

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its structure can be exploited for robust motion estimation and efficient visualization. This paper describes a system that uses multi-view synchronized video footage of an actor's performance to estimate motion parameters and to interactively re-render the actor's appearance from any viewpoint.The actor's silhouettes are extracted from synchronized video frames via background segmentation and then used to determine a sequence of poses for a 3D human body model. By employing multi-view texturing during rendering, time-dependent changes in the body surface are reproduced in high detail. The motion capture subsystem runs offline, is non-intrusive, yields robust motion parameter estimates, and can cope with a broad range of motion. The rendering subsystem runs at real-time frame rates using ubiquous graphics hardware, yielding a highly naturalistic impression of the actor. The actor can be placed in virtual environments to create composite dynamic scenes. Free-viewpoint video allows the creation of camera fly-throughs or viewing the action interactively from arbitrary perspectives.

Supplementary Material

MP4 File (theobalt_freeviewpoint.mp4)

Download
38.94 MB

References

[1]

ALLEN, B., CURLESS, B., AND POPOVIC, Z. 2002. Articulated body deformations from range scan data. In Proceedings of ACM SIGGRAPH 02, 612--619.]]

Digital Library

[2]

BOROVIKOV, E., AND DAVIS, L. 2000. A dristibuted system for real-time volume reconstruction. In Proceedings of Intl. Workshop on Computer Architectures for Machine Perception, 183ff.]]

Digital Library

[3]

BOTTINO, A., AND LAURENTINI, A. 2001. A silhouette based technique for the reconstruction of human movement. CVIU 83, 79--95.]]

Digital Library

[4]

BREGLER, C., AND MALIK, J. 1998. Tracking people with twists and exponential maps. In Proc. of CVPR 98, 8--15.]]

Digital Library

[5]

BUEHLER, C., BOSSE, M., MCMILLAN, L., GORTLER, S. J., AND COHEN, M. F. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 01, ACM Press, S. Spencer, Ed., 425--432.]]

Digital Library

[6]

CHEUNG, K., KANADE, T., BOUGUET, J.-Y., AND HOLLER, M. 2000. A real time system for robust 3D voxel reconstruction of human motions In Proc. of CVPR, vol. 2, 714--720.]]

[7]

CURLESS, B., AND SEITZ, S. 2000. 3D photography Course Notes. ACM SIGGRAPH 00.]]

[8]

DANA, K., VAN GINNEKEN, B., NAYAR, S., AND KOENDERINK, J. 1999 Reflectance and texture of real-world surfaces. ACM Transactions on Graphics 18, 1, 1--34.]]

Digital Library

[9]

DEBEVEC, P., TAYLOR, C., MALIK, J., LEVIN, G., G. BORSHUKOV, AND YU, Y. 1998. Image-based modeling and rendering of architecture with interactive photogrammetry and view-dependent texture mapping. Proc. IEEE International Symposium on Circuits and Systems (ISCAS'98), Monterey, USA 5 (June), 514--517.]]

[10]

DELAMARRE, Q., AND FAUGERAS, O. 1999. 3D articulated models and multi-view tracking with silhouettes. In Proc. of ICCV 99, 716--721.]]

Digital Library

[11]

GAVRILA, D., AND DAVIS, L. 1996. 3D model-based tracking of humans in action: A multi-view approach. In Proc. of CVPR 96, 73--80.]]

Digital Library

[12]

GAVRILA, D. 1999. The visual analysis of human movement. CVIU 73, 1 (January), 82--98.]]

Digital Library

[13]

GRAMMALIDIS, N., GOUSSIS, G., TROUFAKOS, G., AND STRINTZIS, M. 2001. Estimating body animation parameters from depth images using analysis by synthesis. In Proc. of Second International Workshop on Digital and Computational Video (DCV'01), 93ff.]]

Digital Library

[14]

JAIN, R., KASTURI, R., AND SCHUNCK, B. 1995. Machine Vision. McGraw-Hill.]]

Digital Library

[15]

KILGARD, M. J., 2002. Nvidia opengl extension specifications. http://developer.nvidia.com/docs/IO/3260/ATT/nv30specs.pdf.]]

[16]

KOENEN, R., 2002. Mpeg-4 overview. http://mpeg.telecomitalialab.com/standards/mpeg-4/mpeg-4.htm.]]

[17]

LAURENTINI, A. 1994. The visual hull concept for silhouette-based image understanding. Pattern Analysis and Machine Intelligence 16, 2 (February), 150--162.]]

Digital Library

[18]

LENSCH, H., HEIDRICH, W., AND SEIDEL, H. P. 2001. A silhouette-based algorithm for texture registration and stitching. Graphical Models 64(3), 245--262.]]

Digital Library

[19]

LEUNG, M., AND YANG, Y. 1995. First sight: A human body outline labeling system. PAMI 17(4), 359--379.]]

Digital Library

[20]

LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In Proceedings of ACM SIGGRAPH 96, vol. 30, 31--42.]]

Digital Library

[21]

LUCK, J., AND SMALL, D. 2002. Real-time markerless motion tracking using linked kinematic chains. In Proc. of CVPRIP02.]]

[22]

MARTINEZ, G. 1995. 3D motion estimation of articulated objects for object-based analysis-synthesis coding (OBASC). In VLBV 95.]]

[23]

MATSUYAMA, T., AND TAKAI, T. 2002. Generation, visualization, and editing of 3D video. In Proc. of 1st International Symposium on 3D Data Processing Visualization and Transmission (3DPVT'02), 234ff.]]

[24]

MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S., AND MCMILLAN, L. 2000. Image-based visual hulls. In Proceedings of ACM SIGGRAPH 00, 369--374.]]

Digital Library

[25]

MATUSIK, W., BUEHLER, C., AND MCMILLAN, L. 2001. Polyhedral visual hulls for real-time rendering. In Proceedings of 12th Eurographics Workshop on Rendering, 116--126.]]

Digital Library

[26]

MENACHE, A. 1995. Understanding Motion Capture for Computer Animation and Video Games. Morgan Kaufmann.]]

Digital Library

[27]

MIKIĆ, I., TRIVERDI, M., HUNTER, E., AND COSMAN, P. 2001. Articulated body posture estimation from multicamera voxel data. In Proc. of CVPR.]]

[28]

MOEZZI, S., TAI, L.-C., AND GERARD, P. 1997. Virtual view generation for 3D digital video. IEEE MultiMedia 4, 1 (Jan.--Mar.), 18--26.]]

Digital Library

[29]

MULLIGAN, J., AND DANIILIDIS, K. 2000. View-independent scene acquisition for telepresence. In Proceedings of the International Symposium on Augmented Reality, 105--108.]]

[30]

NARAYANAN, P., RANDER, P., AND KANADE, T. 1998. Constructing virtual worlds using dense stereo. In Proc. of ICCV 98, 3--10.]]

Digital Library

[31]

PLAENKERS, R., AND FUA, P. 2001. Tracking and modeling people in video sequences. CVIU 81, 3 (March), 285--302.]]

Digital Library

[32]

PRESS, W., TEUKOLSKY, S., VETTERLING, W., AND FLANNERY, B. 1992. Numerical Recipes. Cambridge University Press.]]

[33]

RASKAR, R., AND LOW, K.-L. 2002. Blending multiple views. In Proceedings of Pacific Graphics 2002, 145--153.]]

Digital Library

[34]

ROHR, K. 1993. Incremental recognition of pedestrians from image sequences. In Proc. of CVPR 93, 8--13.]]

[35]

SILAGHI, M.-C., PLAENKERS, R., BOULIC, R., FUA, P., AND THALMANN, D. 1998. Local and global skeleton fitting techniques for optical motion capture. In Modeling and Motion Capture Techniques for Virtual Environments, Springer, no. 1537 in LNAI, No1537, 26--40.]]

Digital Library

[36]

TERZOPOULOS, D., CARLBOM, I., FREEMAN, W., KLINKER, G., LORENSEN, W., SZELISKI, R., AND WATERS, K. 1995. Computer vision for computer graphics. In ACM SIGGRAPH 95 Course Notes, vol. 25.]]

[37]

THEOBALT, C., MAGNOR, M., SCHUELER, P., AND SEIDEL, H.-P. 2002. Combining 2D feature tracking and volume reconstruction for online video-based human motion capture. In Proceedings of Pacific Graphics 2002, 96--103.]]

Digital Library

[38]

TSAI, R. 1986. An efficient and accurate camera calibration technique for 3D machine vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'86), 364--374.]]

[39]

VEDULA, S., BAKER, S., AND KANADE, T. 2002. Spatio-temporal view interpolation. In Proceedings of the 13th ACM Eurographics Workshop on Rendering, 65--75.]]

Digital Library

[40]

WREN, C., AZARBAYEJANI, A., DARRELL, T., AND PENTLAND, A. 1997. Pfinder: Real-time tracking of the human body. PAMI 19, 7, 780--785.]]

Digital Library

[41]

WUERMLIN, S., LAMBORAY, E., STAADT, O., AND GROSS, M. 2002. 3d video recorder. In Proceedings of Pacific Graphics 2002, IEEE Computer Society Press, 325--334.]]

Digital Library

[42]

YONEMOTO, S., ARITA, D., AND TANIGUCHI, R. 2000. Real-time human motion analysis and IK-based human figure control. In Proceedings of IEEE Workshop on Human Motion, 149--154.]]

Digital Library

Cited By

Hu TXu HLuo LYu TZheng ZZhang HLiu YZwicker M(2024)HVTR++: Image and Pose Driven Human Avatars Using Hybrid Volumetric-Textural RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329772130:8(5478-5492)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3297721
Işık MRünz MGeorgopoulos MKhakhulin TStarck JAgapito LNießner M(2023)HumanRF: High-Fidelity Neural Radiance Fields for Humans in MotionACM Transactions on Graphics10.1145/359241542:4(1-12)Online publication date: 26-Jul-2023
https://doi.org/10.1145/3592415
Carballeira PCarmona CDiaz CBerjon DCorregidor DCabrera JMoran FDoblado CArnaldo SMartin MGarcia N(2022)FVV Live: A Real-Time Free-Viewpoint Video System With Consumer Electronics HardwareIEEE Transactions on Multimedia10.1109/TMM.2021.307971124(2378-2391)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3079711
Show More Cited By

Index Terms

Free-viewpoint video of human actors
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Animation

Recommendations

Free-viewpoint video of human actors

In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its ...
Free-viewpoint video rendering for mobile devices
MIRAGE '13: Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and Applications

Free-viewpoint video renderers (FVVR) allow a user to view captured video footage from any position and direction. Despite the obvious appeal of such systems, they have yet to make a major impact on digital entertainment. Current FVVR implementations ...
Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '03: ACM SIGGRAPH 2003 Papers

July 2003

683 pages

ISBN:1581137095

DOI:10.1145/1201775

Conference Chair:
Alyn P. Rockwood
Colorado School of Mines

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGGRAPH03

Sponsor:

SIGGRAPH

SIGGRAPH03: Special Interest Group on Computer Graphics and Interactive Techniques

July 27 - 31, 2003

California, San Diego

Acceptance Rates

SIGGRAPH '03 Paper Acceptance Rate 81 of 424 submissions, 19%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

125
Total Citations
View Citations
3,105
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hu TXu HLuo LYu TZheng ZZhang HLiu YZwicker M(2024)HVTR++: Image and Pose Driven Human Avatars Using Hybrid Volumetric-Textural RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329772130:8(5478-5492)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3297721
Işık MRünz MGeorgopoulos MKhakhulin TStarck JAgapito LNießner M(2023)HumanRF: High-Fidelity Neural Radiance Fields for Humans in MotionACM Transactions on Graphics10.1145/359241542:4(1-12)Online publication date: 26-Jul-2023
https://doi.org/10.1145/3592415
Carballeira PCarmona CDiaz CBerjon DCorregidor DCabrera JMoran FDoblado CArnaldo SMartin MGarcia N(2022)FVV Live: A Real-Time Free-Viewpoint Video System With Consumer Electronics HardwareIEEE Transactions on Multimedia10.1109/TMM.2021.307971124(2378-2391)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3079711
Toussaint BGenisson MFranco J(2022)Fast Gradient Descent for Surface Capture Via Differentiable Rendering2022 International Conference on 3D Vision (3DV)10.1109/3DV57658.2022.00049(373-382)Online publication date: Sep-2022
https://doi.org/10.1109/3DV57658.2022.00049
Hu TYu TZheng ZZhang HLiu YZwicker M(2022)HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars2022 International Conference on 3D Vision (3DV)10.1109/3DV57658.2022.00032(197-208)Online publication date: Sep-2022
https://doi.org/10.1109/3DV57658.2022.00032
Shere MKim HHilton A(2021)Temporally Consistent 3D Human Pose Estimation Using Dual 360° Cameras2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00013(81-90)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00013
Habermann MXu WZollhoefer MPons-Moll GTheobalt C(2021)A Deeper Look into DeepCapIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3093553(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2021.3093553
Shishido HOkada YKameda YKoido MKitahara I(2020)[Papers] Method of Multiview Video Switching for Soccer Game Analysis in Large Scale SpaceITE Transactions on Media Technology and Applications10.3169/mta.8.708:2(70-80)Online publication date: 2020
https://doi.org/10.3169/mta.8.70
MacQuarrie ASteed A(2020)Exploring the Use of Skeletal Tracking for Cheaper Motion Graphs and On-Set Decision Making in Free-Viewpoint Video ProductionProceedings of the 17th ACM SIGGRAPH European Conference on Visual Media Production10.1145/3429341.3429353(1-10)Online publication date: 7-Dec-2020
https://dl.acm.org/doi/10.1145/3429341.3429353
Salazar GLuo XNavarro Newball AZuniga CLozano-Garzon C(2019)Multiple Character Motion Adaptation in Virtual Cities Using Procedural Animation2019 International Conference on Virtual Reality and Visualization (ICVRV)10.1109/ICVRV47840.2019.00053(223-226)Online publication date: Nov-2019
https://doi.org/10.1109/ICVRV47840.2019.00053
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents