research-article

Video tapestries with continuous temporal zoom

Authors:

Connelly Barnes,

Dan B. Goldman,

Adam FinkelsteinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 29, Issue 4

Article No.: 89, Pages 1 - 9

https://doi.org/10.1145/1778765.1778826

Published: 26 July 2010 Publication History

Abstract

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.

Supplementary Material

Supplemental material. (089.zip)

The attached .zip file contains a video summarizing our paper: we demonstrate how tapestries can be used for navigation, how tapestries are computed, show the continuous zoom feature, and review the user study.

Download
71.53 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graphics 23, 3, 294--302.

Digital Library

[2]

Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. In ACM Intl. Conference on Computer Graphics and Interactive Techniques, 667--676.

Digital Library

[3]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graphics 28, 3.

Digital Library

[4]

Berkhin, P. 2002. Grouping Multidimensional Data: A survey of clustering data mining techniques. Springer.

[5]

Bernstein, S. 1994. Film Production, Second Edition. Focal Press.

[6]

Boreczky, J., Girgensohn, A., Golovchinsky, G., and Uchihashi, S. 2000. An interactive comic book presentation for exploring video. In Proceedings of SIGCHI, ACM, 185--192.

Digital Library

[7]

Bourdev, L., and Brandt, J. 2005. Robust object detection via soft cascade. In IEEE CVPR 2005, vol. 2.

Digital Library

[8]

Chiu, P., Girgensohn, A., and Liu, Q. 2004. Stained-glass visualization for highly condensed video summaries. In IEEE ICME 2004.

[9]

Christel, M., Hauptmann, A., Wactlar, H., and Ng, T. 2002. Collages as dynamic summaries for news video. In ACM Multimedia, 561--569.

Digital Library

[10]

Cockburn, A., Karlson, A., and Bederson, B. B. 2008. A review of overview+detail, zooming, and focus+context interfaces. ACM Comput. Surv. 41, 1, 1--31.

Digital Library

[11]

Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Trans. Graphics 29, 3.

Digital Library

[12]

Davis, M. 1995. Media streams: representing video for retrieval and repurposing. PhD thesis, Wesleyan University.

Digital Library

[13]

Dementhon, D., Kobla, V., and Doermann, D. 1998. Video summarization by curve simplifiation. In ACM Multimedia, 211--218.

Digital Library

[14]

Hauser, T. 2008. The Art of Wall-E. Chronicle Books LLC.

[15]

Kang, H., Matsushita, Y., Tang, X., Chen, X., Hefei, P., and Beijing, P. 2006. Space-time video montage. In CVPR06, 1331--1338.

Digital Library

[16]

Kim, K., Essa, I., and Abowd, G. D. 2006. Interactive mosaic generation for video navigation. In ACM Multimedia, 655--658.

Digital Library

[17]

Kraaij, W., Smeaton, A., Over, P., and Arlandis, J. 2004. Trecvid 2004-an overview. In TRECVID video retrieval online proceedings.

[18]

Kwatra, V., Schdl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graphics 22, 3, 277--286.

Digital Library

[19]

Ma, Y., and Zhang, H. 2002. A model of motion attention for video skimming. In Proc. Image Processing, Int'l Conf., vol. 1, I-129--I-132.

[20]

Mei, T., Yang, B., Yang, S., and Hua, X. 2009. Video collage: presenting a video sequence using a single image. The Visual Computer 25, 1, 39--51.

Digital Library

[21]

Murch, W. 1995. In the Blink of an Eye: A Perspective on Film Editing. Silman-James Press, Los Angeles.

[22]

Rother, C., Kumar, S., Kolmogorov, V., and Blake, A. 2005. Digital tapestry. In IEEE CVPR, I: 589--596.

Digital Library

[23]

Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. ACM Trans. Graphics 25, 3, 847--852.

Digital Library

[24]

Shipman, F., Girgensohn, A., and Wilcox, L. 2003. Generation of interactive multi-level video summaries. In ACM Multimedia, 392--401.

Digital Library

[25]

Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR 2008.

[26]

Sivic, J., Kaneva, B., Torralba, A., Avidan, S., and Freeman, W. 2008. Creating and exploring a large photorealistic virtual space. In IEEE CVPR Workshops, 2008., 1--8.

[27]

Smith, M., and Kanade, T. 1995. Video skimming for quick browsing based on audio and image characterization. Technical Report CMU-CS-95-186, School of Computer Science, Carnegie Mellon University.

[28]

Smith, M., and Kanade, T. 1997. Video skimming and characterization through the combination of image and language understanding techniques. In 1997 IEEE CVPR, 775--781.

Digital Library

[29]

Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: Extracting and packing panoramas for video browsing. In ACM Multimedia, 427--436.

Digital Library

[30]

Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1, 3.

Digital Library

[31]

Uchihashi, S., Foote, J., Girgensohn, A., and Boreczky, J. 1999. Video manga: generating semantically meaningful video summaries. In ACM Multimedia, ACM, 383--392.

Digital Library

[32]

Wang, T., Mei, T., Hua, X.-S., Liu, X., and Zhou, H.-Q. 2007. Video collage: A novel presentation of video sequence. In ICME, IEEE, 1479--1482.

[33]

Yang, B., Mei, T., Sun, L.-F., Yang, S.-Q., and Hua, X.-S. 2008. Free-shaped video collage. Multi-Media Modeling (MMM), 175--185.

Digital Library

Cited By

Swift MAyers WPallanck SWehrwein S(2023)Visualizing the Passage of Time with Video Temporal PyramidsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320945429:1(171-181)Online publication date: Jan-2023
https://doi.org/10.1109/TVCG.2022.3209454
Chan CYuan CSun CChen H(2023)Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00712(7709-7719)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00712
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-15082-w
Show More Cited By

Index Terms

Video tapestries with continuous temporal zoom
1. Computing methodologies
  1. Computer graphics

Recommendations

Video tapestries with continuous temporal zoom
SIGGRAPH '10: ACM SIGGRAPH 2010 papers

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly ...
Interactive Exploration of Surveillance Video through Action Shot Summarization and Trajectory Visualization

We propose a novel video visual analytics system for interactive exploration of surveillance video data. Our approach consists of providing analysts with various views of information related to moving objects in a video. To do this we first extract each ...
The continuous zoom: a constrained fisheye technique for viewing and navigating large information spaces
UIST '95: Proceedings of the 8th annual ACM symposium on User interface and software technology

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 29, Issue 4

July 2010

942 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1778765

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2010

Published in TOG Volume 29, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Information and Intelligent Systems

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
1,161
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Swift MAyers WPallanck SWehrwein S(2023)Visualizing the Passage of Time with Video Temporal PyramidsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320945429:1(171-181)Online publication date: Jan-2023
https://doi.org/10.1109/TVCG.2022.3209454
Chan CYuan CSun CChen H(2023)Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00712(7709-7719)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00712
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-15082-w
Nie YLi ZZhang ZZhang QMa TSun H(2020)Collision-Free Video Synopsis Incorporating Object Speed and Size ChangesIEEE Transactions on Image Processing10.1109/TIP.2019.294254329(1465-1478)Online publication date: 2020
https://doi.org/10.1109/TIP.2019.2942543
Zhang FBarnes CZhang HZhao JSalas G(2020)Coherent video generation for multiple hand-held cameras with dynamic foregroundComputational Visual Media10.1007/s41095-020-0187-36:3(291-306)Online publication date: 3-Sep-2020
https://doi.org/10.1007/s41095-020-0187-3
Gutwin Cvan der Kamp MUddin MStanley KStavness IVail SBrewster SFitzpatrick GCox AKostakos V(2019)Improving Early Navigation in Time-Lapse Video with Spread-Frame LoadingProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300785(1-12)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300785
Lu KWang CWu KGong MWang Y(2019)A unified framework for exploring time-varying volumetric data based on block correspondenceVisual Informatics10.1016/j.visinf.2019.10.0013:4(157-165)Online publication date: Dec-2019
https://doi.org/10.1016/j.visinf.2019.10.001
Zhang ZZuo RGuo RLi YZhou TXue HMa CWang H(2019)Multi-scale visualization based on sketch interaction for massive surveillance video dataPersonal and Ubiquitous Computing10.1007/s00779-019-01281-6Online publication date: 12-Dec-2019
https://doi.org/10.1007/s00779-019-01281-6
Fang FYi MFeng HHu SXiao C(2018)Narrative Collage of Image Collections by Scene Graph RecombinationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2017.275926524:9(2559-2572)Online publication date: 1-Sep-2018
https://doi.org/10.1109/TVCG.2017.2759265
Kolesar IBruckner SViola IHauser H(2017)A Fractional Cartesian Composition Model for Semi-Spatial Comparative Visualization DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2016.259887023:1(851-860)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1109/TVCG.2016.2598870
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents