Consistent Video Filtering for Camera Arrays
Authors: 
Nicolas Bonneel, 
James Tompkin, Deqing Sun, Oliver Wang, Kalyan Sunkavalli, Sylvain Paris, Hanspeter PfisterAuthors Info & Claims
Abstract
Visual formats have advanced beyond single-view images and videos: 3D movies are commonplace, researchers have developed multi-view navigation systems, and VR is helping to push light field cameras to mass market. However, editing tools for these media are still nascent, and even simple filtering operations like color correction or stylization are problematic: naively applying image filters per frame or per view rarely produces satisfying results due to time and space inconsistencies. Our method preserves and stabilizes filter effects while being agnostic to the inner working of the filter. It captures filter effects in the gradient domain, then uses input frame gradients as a reference to impose temporal and spatial consistency. Our least-squares formulation adds minimal overhead compared to naive data processing. Further, when filter cost is high, we introduce a filter transfer strategy that reduces the number of per-frame filtering computations by an order of magnitude, with only a small reduction in visual quality. We demonstrate our algorithm on several camera array formats including stereo videos, light fields, and wide baselines.
References
[1]
<label>{AG16}¿¿</label> Alperovich A., Goldluecke B.: A variational model for intrinsic light field decomposition. In Asian Conference on Computer Vision ACCV 2016. 2
[2]
<label>{AGB*16}¿¿</label> Anderson R., Gallup D., Barron J.T., Kontkanen J., Snavely N., Hernández C., Agarwal S., Seitz S.M.: Jump: Virtual reality video. ACM Trans. Graph. Volume 35, Issue 6 Nov. 2016, pp.198:1-198:13. 10
[3]
<label>{AZJ*15}¿¿</label> Ao H., Zhang Y., Jarabo A., Masia B., Liu Yebin L., Gutierrez, Qionghai D.: Light field editing based on reparameterization. In Adv. in Multimedia Information Proc. 2015. 2
[4]
<label>{BBPP10}¿¿</label> Ballan L., Brostow G.J., Puwein J., Pollefeys M.: Unstructured video-based rendering: Interactive exploration of casually captured videos. pp.1-11. 10
[5]
<label>{BBS14}¿¿</label> Bell S., Bala K., Snavely N.: Intrinsic images in the wild. ACM Trans. on Graphics SIGGRAPH Volume 33, Issue 4 2014. 7
[6]
<label>{BPK*13}¿¿</label> Baek J., Pajak D., Kim K., Pulli K., Levoy M.: WYSIWYG computational photography via viewfinder editing. ACM Trans. Graph. Volume 32, Issue 6 2013. 2
[7]
<label>{BSFG09}¿¿</label> Barnes C., Shechtman E., Finkelstein A., Goldman D.B.: PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. SIGGRAPH Volume 28, Issue 3 2009. 2, 3
[8]
<label>{BTS*15}¿¿</label> Bonneel N., Tompkin J., Sunkavalli K., Sun D., Paris S., Pfister H.: Blind video temporal consistency. ACM Trans. Graph. SIGGRAPH Asia Volume 34, Issue 6 2015. "https://github.com/nbonneel/blindconsistency". 1, 2, 4, 5, 8, 9, 11
[9]
<label>{BZCC10}¿¿</label> Bhat P., Zitnick C.L., Cohen M., Curless B.: GradientShop: A gradient-domain optimization framework for image and video filtering. ACM Trans Graph SIGGRAPH Volume 29, Issue 2 2010. 2
[10]
<label>{DD10}¿¿</label> Delon J., Desolneux A.: Stabilization of flicker-like effects in image sequences through local contrast correction. SIAM Journal on Imaging Sciences Volume 3, Issue 4 2010, pp.703-734. 2
[11]
<label>{DYY15}¿¿</label> Dang K., Yang J., Yuan J.: Adaptive exponential smoothing for online filtering of pixel prediction maps. In 2015 IEEE International Conference on Computer Vision ICCV Dec 2015, pp. pp.3209-3217. 2
[12]
<label>{FK00}¿¿</label> Foote J., Kimber D.: Flycam: Practical panoramic video and automatic camera control. In IEEE Int. Conf. on Multimedia and Expo ICME 2000, vol. Volume 3, pp. pp.1419-1422. 1
[13]
<label>{FL11}¿¿</label> Farbman Z., Lischinski D.: Tonal stabilization of video. ACM Trans. on Graphics SIGGRAPH Volume 30, Issue 4 2011, pp.89:1-89:9. 2
[14]
<label>{GEB15}¿¿</label> Gatys L.A., Ecker A.S., Bethge M.: A neural algorithm of artistic style. CoRR abs/1508.06576 2015. 8
[15]
<label>{GEZ*16}¿¿</label> Garces E., Echevarria J.I., Zhang W., Wu H., Zhou K., Gutierrez D.: Intrinsic light fields. CoRR abs/1608.04342 2016. 2, 7, 8
[16]
<label>{HRF13}¿¿</label> Herbst E., Ren X., Fox D.: Rgb-d flow: Dense 3-d motion estimation using color and depth. In ICRA 2013, IEEE. 10
[17]
<label>{HSGL13}¿¿</label> HaCohen Y., Shechtman E., Goldman D.B., Lischinski D.: Optimizing color consistency in photo collections. ACM Trans. Graph. SIGGRAPH Volume 32, Issue 4 2013, pp.85:1-85:9. 2
[18]
<label>{JMB*14}¿¿</label> Jarabo A., Masia B., Bousseau A., Pellacini F., Gutierrez D.: How do people edit light fields? ACM Trans. Graph. SIGGRAPH Volume 33, Issue 4 2014. 2
[19]
<label>{JMG11}¿¿</label> Jarabo A., Masia B., Gutierrez D.: Efficient propagation of light field edits. In Proc. of the V Ibero-American Symposium in Computer Graphics 2011, SIACG 2011, pp. pp.75-80. 2
[20]
<label>{KFLA15}¿¿</label> Kovács P.T., Fekete A., Lackner K., Adhikarla V.K.: Big buck bunny light-field test sequences. MPEG contribution ISO/IEC JTC1/SC29/WG11 M35721, Feb. 2015, 2015. 10
[21]
<label>{LBP*12}¿¿</label> Laffont P.-Y., Bousseau A., Paris S., Durand F., Drettakis G.: Coherent intrinsic images from photo collections. ACM Trans. Graph. Volume 31, Issue 6 2012. 2
[22]
<label>{LHW*10}¿¿</label> Lang M., Hornung A., Wang O., Poulakos S., Smolic A., Gross M.: Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. Graph. Volume 29, Issue 3 2010, pp.10. 2
[23]
<label>{LNJ13}¿¿</label> Liu F., Niu Y., Jin H.: Joint subspace stabilization for stereoscopic video. In Computer Vision ICCV, 2013 IEEE International Conference on Dec 2013, pp. pp.73-80. 2
[24]
<label>{Low04}¿¿</label> Lowe D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision Volume 60, Issue 2 Nov. 2004, pp.91-110. 4
[25]
<label>{LSS*15}¿¿</label> Luo S.-J., Sun Y.-T., Shen I.-C., Chen B.-Y., Chuang Y.-Y.: Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Trans. Vis. and Comp. Graph. Volume 21, Issue 1 2015, pp.56-67. 2
[26]
<label>{LvBK*10}¿¿</label> Lo W.-Y., van Baar J., Knaus C., Zwicker M., Gross M.: Stereoscopic 3d copy & paste. ACM Trans. Graph. Volume 29, Issue 6 2010, pp.147:1-147:10. 2
[27]
<label>{LWA*12}¿¿</label> Lang M., Wang O., Aydin T., Smolic A., Gross M.: Practical temporal consistency for image-based graphics applications. ACM Trans. Graph. SIGGRAPH Volume 31, Issue 4 2012, pp.34:1-34:8. 2, 8
[28]
<label>{MHCP12}¿¿</label> Morse B., Howard J., Cohen S., Price B.: Patchmatchbased content completion of stereo image pairs. In Int. Conf 3D Imaging, Mod., Proc., Vis. and Transmission 3DIMPVT 2012, pp. pp.555-562. 2
[29]
<label>{MJG14}¿¿</label> Masia B., Jarabo A., Gutierrez D.: Favored workflows in light field editing. In Int. Conf. on Comp. Graph., Vis., Computer Vision and Image Proc. 2014, CGVCVIP. 2
[30]
<label>{MOT15}¿¿</label> Mordvintsev A., Olah C., Tyka M.: Inceptionism: Going deeper into neural networks, June 2015. 8
[31]
<label>{NLB*05}¿¿</label> Ng R., Levoy M., Brédif M., Duval G., Horowitz M., Hanrahan P.: Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR Volume 2, Issue 11 2005. 1
[32]
<label>{PKCK06}¿¿</label> Pitié F., Kent B., Collis B., Kokaram A.: Localised deflicker of moving images. In IEEE European Conference on Visual Media Production 2006. 2
[33]
<label>{PSZ*15}¿¿</label> Perazzi F., Sorkine-Hornung A., Zimmer H., Kaufmann P., Wang O., Watson S., Gross M.H.: Panoramic video from unstructured camera arrays. Comp. Graph. Forum Volume 34, Issue 2 2015, pp.57-68. 1
[34]
<label>{RAKRF08}¿¿</label> Rav-Acha A., Kohli P., Rother C., Fitzgibbon A.: Unwrap mosaics: A new representation for video editing. ACM Trans. Graph. Volume 27, Issue 3 2008. 2
[35]
<label>{RDB16}¿¿</label> Ruder M., Dosovitskiy A., Brox T.: Artistic style transfer for videos. 8
[36]
<label>{RPZSH13}¿¿</label> Richardt C., Pritch Y., Zimmer H., Sorkine-Hornung A.: Megastereo: Constructing high-resolution stereo panoramas. In 2013 IEEE Conference on Computer Vision and Pattern Recognition June 2013, pp. pp.1256-1263. 10
[37]
<label>{RWHS15}¿¿</label> Revaud J., Weinzaepfel P., Harchaoui Z., Schmid C.: EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow. In CVPR 2015. 4
[38]
<label>{SK02}¿¿</label> Seitz S.M., Kutulakos K.N.: Plenoptic image editing. International Journal of Computer Vision Volume 48, Issue 2 2002. 2
[39]
<label>{SSML14}¿¿</label> Sheng B., Sun H., Magnor M., Li P.: Video colorization using parallel optimization in feature space. IEEE Trans. Circuits and Systems for Video Tech. Volume 24, Issue 3 2014, pp.407-417. 2
[40]
<label>{sup15}¿¿</label> Supplemental materials for consistent video filtering for camera array. "http://liris.cnrs.fr/~nbonneel/cameraarrays/data/", 2015. 4, 6, 9
[41]
<label>{VLD*13}¿¿</label> Venkataraman K., Lelescu D., Duparré J., McMahon A., Molina G., Chatterjee P., Mullis R., Nayar S.: Picam: An ultra-thin high performance monolithic camera array. ACM Trans. Graph. Volume 32, Issue 6 Nov. 2013, pp.166:1-166:13. 1
[42]
<label>{WB15}¿¿</label> Wulff J., Black M.J.: Efficient sparse-to-dense optical flow estimation using a learned basis and layers. In IEEE Conference on Computer Vision and Pattern Recognition June 2015. 4
[43]
<label>{WJYG08}¿¿</label> Wang L., Jin H., Yang R., Gong M.: Stereoscopic inpainting: Joint color and depth completion from stereo images. In IEEE Conference on Computer Vision and Pattern Recognition 2008. 2
[44]
<label>{YJHS12}¿¿</label> Yücer K., Jacobson A., Hornung A., Sorkine O.: Transfusive image manipulation. ACM Trans. Graph. Volume 31, Issue 6 2012. 2
[45]
<label>{ZTD*12}¿¿</label> Zhao Q., Tan P., Dai Q., Shen L., Wu E., Lin S.: A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. Volume 34, Issue 7 2012, pp.1437-1444. 8
[46]
<label>{ZWS*16}¿¿</label> Zhang F.-L., Wang J., Shechtman E., Zhou Z.-Y., Shi J.-X., Hu S.-M.: Plenopatch: Patch-based plenoptic image manipulation. IEEE Trans. on Vis. and Comp. Graph. 2016. 2
Recommendations
High-Order Recursive Filtering of Non-Uniformly Sampled Signals for Image and Video Processing
We present a discrete-time mathematical formulation for applying recursive digital filters to non-uniformly sampled signals. Our solution presents several desirable features: it preserves the stability of the original filters; is well-conditioned for ...
Panoramic Video from Unstructured Camera Arrays
We describe an algorithm for generating panoramic video from unstructured camera arrays. Artifact-free panorama stitching is impeded by parallax between input views. Common strategies such as multi-level blending or minimum energy seams produce seamless ...
Scene collages and flexible camera arrays
EGSR'07: Proceedings of the 18th Eurographics conference on Rendering TechniquesThis paper presents an automatic method for creating a collage from a collection of photos of a scene taken from different viewpoints. The collage is constructed by aligning the images (in terms of their positions, rotations and scales) using a least-...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Publisher
The Eurographs Association & John Wiley & Sons, Ltd.
Chichester, United Kingdom
Publication History
Published: 01 May 2017
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Nov 2024
Other Metrics
Citations
View Options
View options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in