article

Pop-up light field: An interactive image-based modeling and rendering system

Authors:

Heung-Yeung Shum,

Shuntaro Yamazaki,

Chi-Keung TangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 23, Issue 2

Pages 143 - 162

https://doi.org/10.1145/990002.990005

Published: 01 April 2004 Publication History

Abstract

In this article, we present an image-based modeling and rendering system, which we call pop-up light field, that models a sparse light field using a set of coherent layers. In our system, the user specifies how many coherent layers should be modeled or popped up according to the scene complexity. A coherent layer is defined as a collection of corresponding planar regions in the light field images. A coherent layer can be rendered free of aliasing all by itself, or against other background layers. To construct coherent layers, we introduce a Bayesian approach, coherence matting, to estimate alpha matting around segmented layer boundaries by incorporating a coherence prior in order to maintain coherence across images.We have developed an intuitive and easy-to-use user interface (UI) to facilitate pop-up light field construction. The key to our UI is the concept of human-in-the-loop where the user specifies where aliasing occurs in the rendered image. The user input is reflected in the input light field images where pop-up layers can be modified. The user feedback is instant through a hardware-accelerated real-time pop-up light field renderer. Experimental results demonstrate that our system is capable of rendering anti-aliased novel views from a sparse light field.

References

[1]

Baker, H. and Bolles, R. 1989. Generalizing epipolar-plane image analysis on the spatiotemporal surface. Int. J. Comput. Vis. 3, 1, 33--39.

[2]

Baker, S., Szeliski, R., and Anandan, P. 1998. A layered approach to stereo reconstruction. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'98). IEEE Computer Society Press, Los Alamitos, Calif., 434--441.

[3]

Berman, A., Dadourian, A., and Vlahos, P. 2000. Method for removing from an image the background surrounding a selected object. U. S. Patent 6, 134, 346.

[4]

Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH'01. ACM, New York, 425--432.

[5]

Chai, J.-X., Tong, X., Chan, S.-C., and Shum, H.-Y. 2000. Plenoptic sampling. In Proceedings of ACM SIGGRAPH'00. ACM, New York, 307--318.

[6]

Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. In Proceedings of ACM SIGGRAPH'02. ACM, New York, 243--248.

[7]

Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A Bayesian approach to digital matting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'01). II:264--271.

[8]

Criminisi, A., Reid, I., and Zisserman, A. 1999. Single view metrology. In Proceedings of 7th International Conference on Computer Vision (ICCV'99). ACM, New York, 434--441.

[9]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. Proceedings of ACM SIGGRAPH'96. ACM, New York, 11--20.

[10]

Décoret, X., Durand, F., Sillion, F., and Dorsey, J. 2003. Billboard clouds for extreme model simplification. In Proceedings of the ACM Siggraph 2003. ACM, New York.

[11]

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In Proceedings of ACM SIGGRAPH'96. ACM, New York, 43--54.

[12]

Heigl, B., Koch, R., Pollefeys, M., Denzler, J., and Gool, L. V. 1999. Plenoptic modeling and rendering from image sequences taken by a hand-held camera. In Symposium fur Mustererkennung. 94--101.

[13]

Horn, B. K. P. and Brooks, M. J. 1989. Shape From Shading. MIT Press, Cambridge, Mass.

[14]

Horry, Y., Anjyo, K.-I., and Arai, K. 1997. Tour into the picture: Using a spidery mesh interface to make animation from a single image. In Proceedings of ACM SIGGRAPH'97. ACM, New York, 225--232.

[15]

Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In Proceedings of ACM SIGGRAPH'00. ACM, New York, 297--306.

[16]

Kang, S. B., Szeliski, R., and Chai, J. 2001. Handling occlusions in dense multi-view stereo. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'01). IEEE Computer Society Press, Los Alamitos, Calif., 103--110.

[17]

Ke, Q. and Kanade, T. 2002. A robust subspace approach to layer extraction. In Proceedings of the IEEE Workshop on Motion and Video Computing. IEEE Computer Society Press, Los Alamitos, Calif., 233--242.

[18]

Kolmogorov, V. and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. In Proceedings of the European Conference on Computer Vision. 82--97.

[19]

Lengyel, J. and Snyder, J. 1997. Rendering with coherent layers. In Proceedings of ACM SIGGRAPH'97. ACM, New York, 233--242.

[20]

Levoy, M. and Hanrahan, P. 1996. Light field rendering. In Proceedings of ACM SIGGRAPH'96. ACM, New York, 31--42.

[21]

Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. In Proceedings of ACM SIGGRAPH'00. ACM, New York, 369--374.

[22]

Matusik, W., Pfister, H., Ngan, A., Beardsley, P., Ziegler, R., and McMillan., L. 2002. Image-based 3d photography using opacity hulls. In Proceedings of ACM SIGGRAPH'02. ACM, New York, 427--437.

[23]

Mishima, Y. 1993. Soft edge chroma-key generation based upon hexoctahedral color space. U.S. Patent 5,355,174.

[24]

Mok, B., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of ACM SIGGRAPH'01. ACM, New York, 433--442.

[25]

Porter, T. and Duff, T. 1984. Compositing digital images. Comput. Graph. 18, 3 (July), 253--259.

[26]

Rademacher, P. 1999. View-dependent geometry. In Proceedings of ACM SIGGRAPH'99. ACM, New York, 439--446.

[27]

Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3d model acquisition. In Proceedings of ACM SIGGRAPH'02. ACM, New York, 438--446.

[28]

Ruzon, M. A. and Tomasi, C. 2000. Alpha estimation in natural images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'00). IEEE Computer Society Press, Los Alamitos, Calif., 18--25.

[29]

Schaufler, G. 1995. Dynamically generated impostors. In Proceedings of the GI Workshop on Modeling, Virtual Worlds, Distributed Graphics. 129--135.

[30]

Schirmacher, H., Heidrich, W., and Seidel, H.-P. 2000. High-quality interactive lumigraph rendering through warping. In Graphics Interface. 87--94.

[31]

Seitz, S. and Kutulakos, K. 1998. Plenoptic image editing. In Proceedings of 7th International Conference on Computer Vision (ICCV'98). 17--24.

[32]

Shade, J., Gortler, S., wei He, L., and Szeliski, R. 1998. Layered depth images. In Proceedings of ACM SIGGRAPH'98. ACM, New York, 231--242.

[33]

Smith, A. R. and Blinn, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH'96. ACM, New York, 259--268.

[34]

Stewart, J., Yu, J., Gortler, S., and McMillan, L. 2003. A new reconstruction filter for undersampled light fields. In Eurographics Symposium on Rendering, 2003.

[35]

Szeliski, R. and Shum, H.-Y. 1997. Creating full view panoramic image mosaics and environment maps. In Proceedings of ACM SIGGRAPH 1997. ACM, New York, 251--258.

[36]

Wang, J. and Adelson, E. 1994. Representing moving images with layers. IEEE Trans. Image Proc. 3, 233--242.

[37]

Wexler, Y., Fitzgibbon, A., and Zisserman., A. 2002. Image-based environment matting. In Proceedings of the 13th Eurographics Workshop on Rendering, Italy.

[38]

Yamazaki, S., Sagawa, R., Kawasaki, H., Ikeuchi, K., and Sakauchi, M. 2002. Microfacet billboarding. In Proceedings of the 13th Eurographics Workshop on Rendering. 175--186.

[39]

Zhang, Z., Wang, L., Guo, B., and Shum, H.-Y. 2002. Feature-based light field morphing. In Proceedings of ACM SIGGRAPH'02. ACM, New York, 457--464.

Cited By

Zhang YWang CCui MRen PXie XHua XBao HHuang QXu WShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Attention-guided Temporally Coherent Video Object MattingProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475623(5128-5137)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475623
Zhu CYu L(2021)Image-assisted geometry simplification for the plenoptic sampling2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2016.7471945(1591-1595)Online publication date: 11-Mar-2021
https://dl.acm.org/doi/10.1109/ICASSP.2016.7471945
Chan S(2021)Light FieldComputer Vision10.1007/978-3-030-63416-2_5(748-755)Online publication date: 13-Oct-2021
https://doi.org/10.1007/978-3-030-63416-2_5
Show More Cited By

Index Terms

Pop-up light field: An interactive image-based modeling and rendering system
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Image segmentation
        Video segmentation
      2. Image and video acquisition
        3D imaging
  2. Computer graphics
    1. Animation
    2. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction

Image-based relighting (IBL) is a technique to change the illumination of an image-based object/scene. In this paper, we define a representation called the reflected irradiance field which records the light reflected from a scene as viewed at a fixed ...
Dynamically reparameterized light fields
SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques

This research further develops the light field and lumigraph image-based rendering methods and extends their utility. We present alternate parameterizations that permit 1) interactive rendering of moderately sampled light fields of scenes with ...
Surface light fields for 3D photography
SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques

A surface light field is a function that assigns a color to each ray originating on a surface. Surface light fields are well suited to constructing virtual images of shiny objects under complex lighting conditions. This paper presents a framework for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 23, Issue 2

April 2004

112 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/990002

Issue’s Table of Contents

Copyright © 2004 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2004

Published in TOG Volume 23, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
1,772
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YWang CCui MRen PXie XHua XBao HHuang QXu WShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Attention-guided Temporally Coherent Video Object MattingProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475623(5128-5137)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475623
Zhu CYu L(2021)Image-assisted geometry simplification for the plenoptic sampling2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2016.7471945(1591-1595)Online publication date: 11-Mar-2021
https://dl.acm.org/doi/10.1109/ICASSP.2016.7471945
Chan S(2021)Light FieldComputer Vision10.1007/978-3-030-63416-2_5(748-755)Online publication date: 13-Oct-2021
https://doi.org/10.1007/978-3-030-63416-2_5
Chan S(2021)Image-Based RenderingComputer Vision10.1007/978-3-030-63416-2_4(656-664)Online publication date: 13-Oct-2021
https://doi.org/10.1007/978-3-030-63416-2_4
Zhu CZhang HYu L(2018)Structure Models for Image-Assisted Geometry Measurement in Plenoptic SamplingIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2017.275595867:1(150-166)Online publication date: Jan-2018
https://doi.org/10.1109/TIM.2017.2755958
Wu Y(2018)Research on Depth Estimation Method of Light Field Imaging Based on Big Data in Internet of Things From Camera ArrayIEEE Access10.1109/ACCESS.2018.28703946(52308-52320)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2870394
Karacan LErdem AErdem E(2017)Alpha Matting With KL-Divergence-Based Sparse SamplingIEEE Transactions on Image Processing10.1109/TIP.2017.271866426:9(4523-4536)Online publication date: 11-Jul-2017
https://dl.acm.org/doi/10.1109/TIP.2017.2718664
Ortin MJarabo AMasia BGutierrez D(2017)Analyzing Interfaces and Workflows for Light Field EditingIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2017.274626311:7(1162-1172)Online publication date: Oct-2017
https://doi.org/10.1109/JSTSP.2017.2746263
Schedl DBirklbauer CGschnaller JBimber O(2016)Generalized Depth-of-Field Light-Field RenderingComputer Vision and Graphics10.1007/978-3-319-46418-3_9(95-105)Online publication date: 10-Sep-2016
https://doi.org/10.1007/978-3-319-46418-3_9
Shidanshidi HSafaei FLi W(2016)Quality Assessment, Evaluation, and Optimization of Free Viewpoint Video Systems by Using Effective Sampling DensityConnected Media in the Future Internet Era10.1007/978-1-4939-4026-4_2(9-55)Online publication date: 9-Oct-2016
https://doi.org/10.1007/978-1-4939-4026-4_2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents