research-article

Unwrap mosaics: a new representation for video editing

Authors:

Pushmeet Kohli,

Carsten Rother,

Andrew FitzgibbonAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 27, Issue 3

Pages 1 - 11

https://doi.org/10.1145/1360612.1360616

Published: 01 August 2008 Publication History

Abstract

We introduce a new representation for video which facilitates a number of common editing tasks. The representation has some of the power of a full reconstruction of 3D surface models from video, but is designed to be easy to recover from a priori unseen and uncalibrated footage. By modelling the image-formation process as a 2D-to-2D transformation from an object's texture map to the image, modulated by an object-space occlusion mask, we can recover a representation which we term the "unwrap mosaic". Many editing operations can be performed on the unwrap mosaic, and then re-composited into the original sequence, for example resizing objects, repainting textures, copying/cutting/pasting objects, and attaching effects layers to deforming objects.

Supplementary Material

MOV File (a17-rav_acha.mov)

Download
43.64 MB

References

[1]

2d3 Ltd., 2008. Boujou 4: The virtual interchangeable with the real. http://www.2d3.com.

[2]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S. M., Colburn, A., Curless, B., Salesin, D., and Cohen, M. F. 2004. Interactive digital photomontage. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3, 294--302.

Digital Library

[3]

Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M., and Szeliski, R. 2007. A database and evaluation methodology for optical flow. In Proc. ICCV.

[4]

Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering.

Digital Library

[5]

Black, M. J., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Proc. ICCV, 231--236.

[6]

Blake, A., and Zisserman, A. 1987. Visual Reconstruction. MIT Press.

Digital Library

[7]

Boykov, Y., and Jolly, M.-P. 2001. Interactive graph cuts for optimal boundary and region segmentation of objects in n-D images. In Proc. ICCV, 105--112.

[8]

Brand, M. 2001. Morphable 3D models from video. In Proc. CVPR, vol. 2, 456--463.

[9]

Bregler, C., Hertzmann, A., and Biermann, H. 2000. Recovering non-rigid 3D shape from image streams. In Proc. CVPR, 690--696.

[10]

Brown, M., and Lowe, D. G. 2007. Automatic panoramic image stitching using invariant features. Intl. J. Comput. Vision 74, 1, 59--73.

Digital Library

[11]

Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV, 25--36.

[12]

Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Intl. J. of Computer Vision 61, 3, 211--231.

Digital Library

[13]

Costeira, J. P., and Kanade, T. 1998. A multibody factorization method for independently moving objects. Intl. J. of Computer Vision 29, 3, 159--179.

Digital Library

[14]

Cox, M., and Cox, M. A. A. 2001. Multidimensional Scaling. Chapman and Hall.

[15]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs. In Proc. ACM Siggraph.

Digital Library

[16]

Fleet, D., Jepson, A., and Black, M. 2002. A layered motion representation with occlusion and compact spatial support. In Proc. ECCV, 692--706.

Digital Library

[17]

Frey, B. J., Jojic, N., and Kannan, A. 2003. Learning appearance and transparency manifolds of occluded objects in layers. In Proc. CVPR.

Digital Library

[18]

Gay-Bellile, V., Bartoli, A., and Sayd, P. 2007. Direct estimation of non-rigid registrations with image-based self-occlusion reasoning. In Proc. ICCV.

[19]

Gu, X., Gortler, S. J., and Hoppe, H. 2002. Geometry images. ACM Trans. Graph. (Proc. of SIGGRAPH), 355--361.

Digital Library

[20]

Irani, M., Anandan, P., and Hsu, S. 1995. Mosaic based representations of video sequences and their applications. In Proc. ICCV.

Digital Library

[21]

Lempitsky, V., and Ivanov, D. 2007. Seamless mosaicing of image-based texture maps. In Proc. CVPR, 1--6.

[22]

Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Trans. Graph. (Proc. of SIGGRAPH) 24, 3, 595--600.

Digital Library

[23]

Rav-Acha, A., Kohli, P., Rother, C., and Fitzgibbon, A. 2008. Unwrap mosaics. Tech. rep., Microsoft Research. http://research.microsoft.com/unwrap.

[24]

Sand, P., and Teller, S. J. 2006. Particle video: Longrange motion estimation using point trajectories. In Proc. CVPR, 2195--2202.

Digital Library

[25]

Seetzen, H., Heidrich, W., Stuerzlinger, W., Ward, G., Whitehead, L., Trentacoste, M., Ghosh, A., and Vorozcovs, A. 2004. High dynamic range display systems. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3, 760--768.

Digital Library

[26]

Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. CVPR, vol. 1, 519--526.

Digital Library

[27]

Seymour, M. 2006. Art of optical flow. fxguide.com: Feature Stories (Dec.).

[28]

Shade, J. W., Gortler, S. J., He, L.-W., and Szeliski, R. 1998. Layered depth images. In Proc. ACM Siggraph, 231--242.

Digital Library

[29]

Shi, J., and Malik, J. 1997. Normalized cuts and image segmentation. In Proc. CVPR, 731--743.

Digital Library

[30]

Thormählen, T., and Broszio, H., 2008. Voodoo Camera Tracker: A tool for the integration of virtual and real scenes. http://www.digilab.uni-hannover.de/docs/manual.html.

[31]

Toklu, C., Erdem, A. T., and Tekalp, A. M. 2000. Two-dimensional mesh-based mosaic representation for manipulation of video objects with occlusion. IEEE Trans. Image Proc. 9, 9, 1617--1630.

Digital Library

[32]

Torresani, L., Hertzmann, A., and Bregler, C. 2008. Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. IEEE Trans. PAMI, (to appear).

Digital Library

[33]

Turk, G., and Levoy, M. 1994. Zippered polygon meshes from range images. In Proc. ACM Siggraph, 311--318.

Digital Library

[34]

van den Hengel, A., Dick, A., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. VideoTrace: Rapid interactive scene modelling from video. ACM Trans. Graph. (Proc. of SIGGRAPH).

Digital Library

[35]

Wang, J. Y. A., and Adelson, E. H. 1994. Representing moving images with layers. IEEE Trans. Image Proc. 3, 5, 625--638.

Digital Library

[36]

Woodford, O. J., Reid, I. D., and Fitzgibbon, A. W. 2007. Efficient new-view synthesis using pairwise dictionary priors. In Proc. CVPR.

[37]

Zhou, K., Wang, X., Tong, Y., Desbrun, M., Guo, B., and Shum, H.-Y. 2005. Texture-Montage: Seamless texturing of surfaces from multiple images. ACM Trans. Graph. (Proc. of SIGGRAPH), 1148--1155.

Digital Library

[38]

Zigelman, G., Kimmel, R., and Kiryati, N. 2002. Texture mapping using surface flattening via multi-dimensional scaling. IEEE Trans. on Visualization and Computer Graphics 8, 2, 198--207.

Digital Library

Cited By

Ouyang HWang QXiao YBai QZhang JZheng KZhou XChen QChen Q(2024)CoDeF: Content Deformation Fields for Temporally Consistent Video Processing2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00773(8089-8099)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00773
Chan CYuan CSun CChen H(2023)Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00712(7709-7719)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00712
Jafarian YWang TCeylan DYang JCarr NZhou YPark H(2023)Normal-guided Garment UV Prediction for Human Re-texturing2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00449(4627-4636)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00449
Show More Cited By

Recommendations

Generalized Parallel-Perspective Stereo Mosaics from Airborne Video

Abstract--In this paper, we present a new method for automatically and efficiently generating stereoscopic mosaics by seamless registration of images collected by a video camera mounted on an airborne platform. Using a parallel-perspective ...
Panoramic Depth Imaging: Single Standard Camera Approach

In this paper we present a panoramic depth imaging system. The system is mosaic-based which means that we use a single rotating camera and assemble the captured images in a mosaic. Due to a setoff of the camera's optical center from the rotational ...
Model-based 2D&3D dominant motion estimation for mosaicing and video representation
ICCV '95: Proceedings of the Fifth International Conference on Computer Vision

It is fairly common in video sequences that a mostly fixed background (scene) is imaged with or without objects. The dominant background changes in the image plane mostly due to camera operations and motion (zoom, pan, tilt, track etc.). We address the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 27, Issue 3

August 2008

844 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1360612

Issue’s Table of Contents

Copyright © 2008 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2008

Published in TOG Volume 27, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
1,718
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ouyang HWang QXiao YBai QZhang JZheng KZhou XChen QChen Q(2024)CoDeF: Content Deformation Fields for Temporally Consistent Video Processing2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00773(8089-8099)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00773
Chan CYuan CSun CChen H(2023)Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00712(7709-7719)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00712
Jafarian YWang TCeylan DYang JCarr NZhou YPark H(2023)Normal-guided Garment UV Prediction for Human Re-texturing2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00449(4627-4636)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00449
Jung KHa HJeon IHong J(2022)Object panorama construction using large-parallax imagesMultimedia Tools and Applications10.1007/s11042-022-13134-181:27(39059-39075)Online publication date: 27-Apr-2022
https://doi.org/10.1007/s11042-022-13134-1
Perez-Rua JMiksik OCrivelli TBouthemy PTorr PPerez P(2020)ROAM: A Rich Object Appearance Model with Application to RotoscopingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2019.290496342:8(1996-2010)Online publication date: 1-Aug-2020
https://doi.org/10.1109/TPAMI.2019.2904963
Shysheya AZakharov EAliev KBashirov RBurkov EIskakov KIvakhnenko AMalkov YPasechnik IUlyanov DVakhitov ALempitsky V(2019)Textured Neural Avatars2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2019.00249(2382-2392)Online publication date: Jun-2019
https://doi.org/10.1109/CVPR.2019.00249
Wiles OKoepke AZisserman A(2018)X2Face: A Network for Controlling Face Generation Using Images, Audio, and Pose CodesComputer Vision – ECCV 201810.1007/978-3-030-01261-8_41(690-706)Online publication date: 8-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-030-01261-8_41
Bonneel NTompkin JSun DWang OSunkavalli KParis SPfister H(2017)Consistent Video Filtering for Camera ArraysComputer Graphics Forum10.1111/cgf.1313536:2(397-407)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1111/cgf.13135
Glennerster A(2016)A moving observer in a three-dimensional worldPhilosophical Transactions of the Royal Society B: Biological Sciences10.1098/rstb.2015.0265371:1697(20150265)Online publication date: 6-Jun-2016
https://doi.org/10.1098/rstb.2015.0265
Meyer SSorkine-Hornung AGross M(2016)Phase-Based Modification Transfer for VideoComputer Vision – ECCV 201610.1007/978-3-319-46487-9_39(633-648)Online publication date: 17-Sep-2016
https://doi.org/10.1007/978-3-319-46487-9_39
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents