research-article

Open access

VRMixer: mixing video and real world with video segmentation

Authors:

Tatsunori Hirai,

Satoshi Nakamura,

Tsubasa Yumura,

Shigeo MorishimaAuthors Info & Claims

ACE '14: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology

Article No.: 30, Pages 1 - 7

https://doi.org/10.1145/2663806.2663834

Published: 11 November 2014 Publication History

Abstract

This paper presents VRMixer, a system that mixes real world and a video clip letting a user enter the video clip and realize a virtual co-starring role with people appearing in the clip. Our system constructs a simple virtual space by allocating video frames and the people appearing in the clip within the user's 3D space. By measuring the user's 3D depth in real time, the time space of the video clip and the user's 3D space become mixed. VRMixer automatically extracts human images from a video clip by using a video segmentation technique based on 3D graph cut segmentation that employs face detection to detach the human area from the background. A virtual 3D space (i.e., 2.5D space) is constructed by positioning the background in the back and the people in the front. In the video clip, the user can stand in front of or behind the people by using a depth camera. Real objects that are closer than the distance of the clip's background will become part of the constructed virtual 3D space. This synthesis creates a new image in which the user appears to be a part of the video clip, or in which people in the clip appear to enter the real world. We aim to realize "video reality," i.e., a mixture of reality and video clips using VRMixer.

Supplementary Material

WMV File (a30-hirai.wmv)

Download
23.84 MB

References

[1]

Boykov, Y., and Kolmogorov, V. An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision. IEEE Trans. on PAMI Vol. 26, 9 (2004), 1124--1137.

Digital Library

[2]

Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., and Durand, F. The video mesh: A data structure for image-based three-dimensional video editing. In Proc. of ICCP2011 (2011), 1--8.

[3]

Corduneanu, A., and Bishop, C. Variational bayesian model selection for mixture distributions. In Proc. of AI Statics2001 (2001), 27--34.

[4]

Dalal, N., and Triggs, B. Histograms of oriented gradients for human detection. In Proc. of CVPR2005 (2005), 886--8893.

Digital Library

[5]

Farneback, G. Two-frame motion estimation based on polynomial expansion. Image Analysis (2003), 363--370.

Digital Library

[6]

Hunter, S., Maes, P., Tang, A., Inkpen, K., and Hessey, S. Waazam!: supporting creative play at a distance in customized video environments. In Proc. of CHI2014 (2014), 1197--1206.

Digital Library

[7]

Iizuka, S., Kanamori, Y., Mitani, J., and Furui, Y. Efficiently modeling 3d scenes from a single image. IEEE Computer Graphics and Aplications Vol. 32, 6 (2012), 18--25.

Digital Library

[8]

Irie, A., Takagiwa, M., Moriyama, K., and Yamashita, T. Improvements to facial contour detection by hierarchical fitting and regression. In Proc. of ACPR2011 (2011), 273--277.

[9]

Kurihara, T., Okabe, M., and Onai, R. Ddmixer2.5d: drag and drop to mix 2.5d video objects. In Proc. of UIST2013 Adjunct (2013), 69--70.

Digital Library

[10]

Li, Y., Sun, J., and Shum, H. Video object cut and paste. In ACM Trans. on Graphics, vol. 24 (2005), 595--600.

Digital Library

[11]

Maejima, A., Wemler, S., Machida, T., Takebayashi, M., and Morishima, S. Instant casting movie theater: The future cast system. IEICE Trans. on Information and Systems (2008), 1135--1148.

Digital Library

[12]

Morishima, S., Yagi, Y., and Nakamura, S. Instant movie casting with personality dive into the movie system. Virtual and Mixed Reality 2011 (2011), 187--196.

Digital Library

[13]

Rother, C., Kolmogorov, V., and Blake, A. "grabcut": interactive foreground extraction using iterated graph cuts. ACM Trans. on Graphics Vol. 23, 3 (2004), 309--314.

Digital Library

[14]

Sodhi, R., Poupyrev, I., Glisson, M., and Israr, A. Aireal: Interactive tactile experiences in free air. ACM Trans. on Graphics Vol. 32, 4 (2013).

Digital Library

[15]

Tsuchida, S., Terada, T., and Tsukamoto, M. A system for practicing formations in dance performance supported by self-propelled screen. In Proc. of AH2013 (2013), 178--185.

Digital Library

[16]

Wang, J., Bhat, P., Colburn, R., Agrawala, M., and Cohen, M. Interactive video cutout. In ACM Trans. on Graphics, vol. 24 (2005), 585--594.

Digital Library

[17]

Wang, X., Han, T., and Yan, S. An hog-lbp human detector with partial occlusion handling. In Proc. of ICCV2009 (2009), 32--39.

Cited By

TSUCHIDA SFUKAYAMA SKATO JYAKURA HGOTO M(2024)DanceUnisoner: A Parametric, Visual, and Interactive Simulation Interface for Choreographic Composition of Group DanceIEICE Transactions on Information and Systems10.1587/transinf.2023EDP7063E107.D:3(386-399)Online publication date: 1-Mar-2024
https://doi.org/10.1587/transinf.2023EDP7063
Yamaura HTamura MNakamura S(2018)Image Blurring Method for Enhancing Digital Content Viewing ExperienceHuman-Computer Interaction. Theories, Methods, and Human Issues10.1007/978-3-319-91238-7_29(355-370)Online publication date: 1-Jun-2018
https://doi.org/10.1007/978-3-319-91238-7_29
Tamura MNakamura S(2018)A Method for Enriching Video-Watching Experience with Applied Effects Based on Eye MovementsMultiMedia Modeling10.1007/978-3-030-05716-9_44(547-553)Online publication date: 11-Dec-2018
https://doi.org/10.1007/978-3-030-05716-9_44
Show More Cited By

Index Terms

VRMixer: mixing video and real world with video segmentation
1. Human-centered computing

Recommendations

Crossmodal Rhythm Perception
HAID '08: Proceedings of the 3rd international workshop on Haptic and Audio Interaction Design

Research on rhythm perception has mostly been focused on the auditory and visual modalities. Previous studies have shown that the auditory modality dominates rhythm perception. Rhythms can also be perceived through the tactile senses, for example, as ...
Streaming of Kinect Data for Interactive In-Browser Audio Performances
AM '16: Proceedings of the Audio Mostly 2016

Distributed audio applications typically follow a server-client model, in which the server produces an audio stream that is sent to the client for consumption. Then, if any modification of the content is needed, it is a product of audio post-processing ...
Project Hindsight: Seamless Interactive Film For Virtual Reality
CHI PLAY '17 Extended Abstracts: Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play

Project Hindsight is an experience for virtual reality combining both computer-generated (CG) art assets as well as live film elements to create an interactive narrative concerned with bad driving habits. Using the Oculus Rift and Oculus Touch motion ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACE '14: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology

November 2014

422 pages

ISBN:9781450329453

DOI:10.1145/2663806

General Chair:
Yoram Chisik
University of Madeira, Portugal
,
Program Chairs:
Christian Geiger
FH Düsseldorf, Germany
,
Shoichi Hasegawa
Tokyo Institute of Technology, Japan

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACE '14

ACE '14: 11th ADVANCES IN COMPUTER ENTERTAINMENT TECHNOLOGY CONFERENCE

November 11 - 14, 2014

Funchal, Portugal

Acceptance Rates

ACE '14 Paper Acceptance Rate 36 of 90 submissions, 40%;

Overall Acceptance Rate 36 of 90 submissions, 40%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
420
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)11

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

TSUCHIDA SFUKAYAMA SKATO JYAKURA HGOTO M(2024)DanceUnisoner: A Parametric, Visual, and Interactive Simulation Interface for Choreographic Composition of Group DanceIEICE Transactions on Information and Systems10.1587/transinf.2023EDP7063E107.D:3(386-399)Online publication date: 1-Mar-2024
https://doi.org/10.1587/transinf.2023EDP7063
Yamaura HTamura MNakamura S(2018)Image Blurring Method for Enhancing Digital Content Viewing ExperienceHuman-Computer Interaction. Theories, Methods, and Human Issues10.1007/978-3-319-91238-7_29(355-370)Online publication date: 1-Jun-2018
https://doi.org/10.1007/978-3-319-91238-7_29
Tamura MNakamura S(2018)A Method for Enriching Video-Watching Experience with Applied Effects Based on Eye MovementsMultiMedia Modeling10.1007/978-3-030-05716-9_44(547-553)Online publication date: 11-Dec-2018
https://doi.org/10.1007/978-3-030-05716-9_44
Goto M(2017)OngaCREST Project: Building a Similarity-Aware Information Environment for a Content-Symbiotic SocietyHuman-Harmonized Information Technology, Volume 210.1007/978-4-431-56535-2_1(1-39)Online publication date: 21-Apr-2017
https://doi.org/10.1007/978-4-431-56535-2_1
Alizadeh HWitcraft ATang ASharlin EPopa TMoffatt K(2016)HappyFeetProceedings of the 42nd Graphics Interface Conference10.5555/3076132.3076156(117-124)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.5555/3076132.3076156

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents