Psycho-visual modulation based information display: introduction and survey
Abstract
Industry and academia have been making great efforts in improving refresh rates and resolutions of display devices to meet the ever increasing needs of consumers for better visual quality. As a result, many modern displays have spatial and temporal resolutions far beyond the discern capability of human visual systems. Thus, leading to the possibility of using those display-eye redundancies for innovative usages. Temporal/spatial psycho-visual modulation (TPVM/SPVM) was proposed to exploit those redundancies to generate multiple visual percepts for different viewers or to transmit non-visual data to computing devices without affecting normal viewing. This paper reviews the STPVM technology from both conceptual and algorithmic perspectives, with exemplary applications in multi-view display, display with visible light communication, etc. Some possible future research directions are also identified.
References
[1]
Wu X and Zhai G Temporal psychovisual modulation: a new paradigm of information display [exploratory DSP] IEEE Signal Processing Magazine 2013 30 1 136-141
[2]
Kelly D H Spatio-temporal frequency characteristics of color-vision mechanisms Journal of the Optical Society America 1974 64 7 983-990
[3]
Varner D, Jameson D, and Hurvich L M Temporal sensitivities related to color theory Journal of the Optical Society of America A 1984 1 5 474-481
[4]
Instruments T. DLP discovery 4100 development kit. See Ti Website, 2015
[5]
Corporation N. NVIDIA 3D Vision. See Nvidia Website, 2014
[6]
Ko H, Paik J, Zalewski G. Stereoscopic screen sharing method and apparatus. 2010. US Patent App. 12/503,029
[7]
Corporation N. Pixel density display listing. See Pixensity Website, 2018
[8]
Karnik A, Martinez Plasencia D, Mayol-Cuevas W, Subramanian S. PiVOT: personalized view-overlays for tabletops. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. 2012, 271–280
[9]
Wetzstein G, Lanman D, Hirsch M, and Raskar R Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting ACM Transactions on Graphics 2012 31 4 80
[10]
Ye G, State A, Fuchs H. A practical multi-viewer tabletop autostereoscopic display. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality. 2010, 147–156
[11]
Karnik A, Mayol-Cuevas W, Subramanian S. MUSTARD: a multi user see through AR display. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012, 2541–2550
[12]
Nashel A, Fuchs H. Random hole display: a non-uniform barrier autostereoscopic display. In: Proceedings of 3DTV Conference: The True Vision — Capture, Transmission and Display of 3D Video. 2009, 1–4
[13]
Lanman D, Wetzstein G, Hirsch M, Heidrich W, Raskar R. Polarization fields: dynamic light field display using multi-layer LCDs. In: Proceedings of the SIGGRAPH Asia Conference. 2011, 1–10
[14]
Zhai G and Wu X Multiuser collaborative viewport via temporal psychovisual modulation [applications corner] IEEE Signal Processing Magazine 2014 31 5 144-149
[15]
Wu X, Zhai G. Backward compatible stereoscopic displays via temporal psychovisual modulation. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012
[16]
Jiao L, Shu X, and Wu X LED backlight adjustment for backward-compatible stereoscopic display IEEE Signal Processing Letters 2013 20 12 1203-1206
[17]
Ma R, Au O C, Wan P, Xu L, Sun W, Hu W. Improved temporal psychovisual modulation for backward-compatible stereoscopic display. In: Proceedings of IEEE Global Conference on Signal and Information Processing. 2014, 1034–1038
[18]
Chen Y, Zhai G, Zhou J, Wan Z, Tang L. Global quality of assessment and optimization for the backward-compatible stereoscopic display system. In: Proceedings of IEEE International Conference on Image Processing. 2017, 191–195
[19]
Fujimura W, Koide Y, Songer R, Hayakawa T, Shirai A, Yanaka K. 2x3D: real time shader for simultaneous 2D/3D hybrid theater. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012, 1–2
[20]
Scher S, Liu J, Vaish R, Gunawardane P, and Davis J 3D+2DTV: 3D displays with no ghosting for viewers without glasses ACM Transactions on Graphics 2013 32 3 21
[21]
Gao Z, Zhai G, Min X. Information security display system based on temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2014, 449–452
[22]
Hu C, Zhai G, Gao Z, Min X. Information security display system based on spatial psychovisual modulation. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2014, 1–4
[23]
Chen Y, Liu N, Zhai G, Gao Z, Gu K. Information security display system on android device. In: Proceedings of IEEE Region 10 Conference. 2016, 1634–1637
[24]
Li X, Zhai G, Wang J, Gu K. Portable information security display system via spatial psychovisual modulation. In: Proceedings of IEEE Visual Communications and Image Processing. 2017, 1–4
[25]
Hu C, Zhai G, Gao Z, Min X. Simultaneous dual-subtitles exhibition via spatial psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2014, 1–4
[26]
Hu C, Zhai G, Gao Z, Min X. Simultaneous triple subtitles exhibition via temporal psychovisual modulation. In: Proceedings of the 9th IEEE Conference on Industrial Electronics and Applications. 2014, 944–947
[27]
Sun W, Zhai G, Gao Z, Chen T, Zhu Y, Wang Z. Dual-view oracle bone script recognition system via temporal-spatial psychovisual modulation. In: Proceedings of IEEE Conference on Multimedia Information Processing and Retrieval. 2020, 193–198
[28]
Gao Z, Zhai G, Hu C, Min X. Dual-view medical image visualization based on spatial-temporal psychovisual modulation. In: Proceedings of IEEE International Conference on Image Processing. 2014
[29]
Fang W, Zhai G, Yang X, Liu J, Chen Y. An eye-friendly dual-view projection system using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–5
[30]
Zhai G and Wu X Defeating camcorder piracy by temporal psychovisual modulation Journal of Display Technology 2014 10 9 754-757
[31]
Gao Z, Zhai G, Wu X, Min X, Zhi C. DLP based anti-piracy display system. In: Proceedings of IEEE Conference on Visual Communications and Image Processing. 2014, 145–148
[32]
Chen Y, Zhai G, Gao Z, Gu K, Zhang W, Hu M, Liu J. Movie piracy tracking using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–4
[33]
Gao Z, Zhai G, Hu C. The invisible QR code. In: Proceedings of the ACM International Conference on Multimedia. 2015, 675–678
[34]
Lu X, You B, Lin P Y. Augmented reality via temporal psycho-visual modulation. In: Proceedings of IEEE International Conference on Multimedia Expo Workshops. 2016, 1–4
[35]
Chen Q, Chen Y. Polarization based invisible barcode display. In: Proceedings of International Forum on Digital TV and Wireless Multimedia Communication. 2018, 67–77
[36]
Shi S, Chen L, Hu W, Gruteser M. Reading between lines: high-rate, non-intrusive visual codes within regular videos via implicitcode. In: Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2015, 157–168
[37]
Wu X, Shu X. Combining information display and visible light wireless communication. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. 2015, 1657–1661
[38]
Shu X, Wu X. Frame untangling for unobtrusive display-camera visible light communication. In: Proceedings of the 24th ACM International Conference on Multimedia. 2016, 650–654
[39]
Liu K, Wu X, Shu X. On display-camera synchronization for visible light communication. In: Proceedings of Visual Communications and Image Processing. 2015, 1–4
[40]
Hu C, Zhai G, Gao Z. Visible light communication via temporal psychovisual modulation. In: Proceedings of the 23rd ACM International Conference on Multimedia. 2015, 785–788
[41]
Fang W, Zhai G, Yang X. A flash light system for individuals with visual impairment based on TPVM. In: Proceedings of International Conference on Cloud Computing and Big Data. 2016, 362–366
[42]
Zhang Y, Zhai G, Liu J, Weng X, Chen Y. ‘window of visibility’ inspired security lighting system. In: Proceedings of International Conference on Systems, Signals and Image Processing. 2017, 1–5
[43]
Gao Z, Zhai G, and Zhou J Factorization algorithms for temporal psychovisual modulation display IEEE Transactions on Multimedia 2016 18 4 614-626
[44]
Feng J, Huo X, Song L, Yang X, and Zhang W Evaluation of different algorithms of nonnegative matrix factorization in temporal psychovisual modulation IEEE Transactions on Circuits and Systems for Video Technology 2014 24 4 553-565
[45]
Wang L, Zhai G. Constrained nmf for multiple exhibition on a single display. In: Proceedings of Picture Coding Symposium. 2015, 292–296
[46]
Gao Z, Zhai G, Gu X, Zhou J. Adapting hierarchical ALS algorithms for temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2015, 2756–2759
[47]
Gao Z, Zhai G, and Wang J Spatially-weighted nonnegative matrix factorization with application to temporal psychovisual modulation Digital Signal Processing 2017 67 123-130
[48]
Kim J and Park H Fast nonnegative matrix factorization: an active-set-like method and comparisons SIAM Journal on Scientific Computing 2011 33 6 3261-3281
[49]
Lee D D and Seung H S Learning the parts of objects by non-negative matrix factorization Nature 1999 401 6755 788-791
[50]
Gonzalez E F, Zhang Y. Accelerating the Lee-Seung algorithm for nonnegative matrix factorization. Department of Computational and Applied Mathematics, Rice University, Houston, TX, Technical Report, TR-05-02, 2005
[51]
Berry M W, Browne M, Langville A N, Pauca V P, and Plemmons R J Algorithms and applications for approximate nonnegative matrix factorization Computational Statistics & Data Analysis 2007 52 1 155-173
[52]
Kim J, Park H. Toward faster nonnegative matrix factorization: a new algorithm and comparisons. In: Proceedings of IEEE International Conference on Data Mining. 2008, 353–362
[53]
Kim H and Park H Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method SIAM Journal on Matrix Analysis and Applications 2008 30 2 713-730
[54]
Cichocki A and Anh-Huy P Fast local algorithms for large scale nonnegative matrix and tensor factorizations IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 2009 92 3 708-721
[55]
Gillis N and Glineur F Accelerated multiplicative updates and hierarchical als algorithms for nonnegative matrix factorization Neural Computation 2012 24 4 1085-1105
[56]
Itti L, Koch C, and Niebur E A model of saliency-based visual attention for rapid scene analysis IEEE Transactions on Pattern Analysis and Machine Intelligence 1998 20 11 1254-1259
[57]
Reinhard E, Ward G, Pattanaik S, and Debevec P High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting 2005 San Francisco, CA, USA Morgan Kaufmann Publishers Inc.
[58]
Li D, Gao Z, Zhang X P, Zhai G, and Yang X Generative adversarial networks for non-negative matrix factorization in temporal psycho-visual modulation Digital Signal Processing 2020 100 102681
[59]
Gao Z and Zhai G Dual-view display based on spatial psychovisual modulation IEEE Access 2018 6 41356-41366
[60]
Nñsñnen R E, Kukkonen H T, and Rovamo J M A window model for spatial integration in human pattern discrimination Investigative Ophthalmology & Visual Science 1995 36 9 1855-1862
[61]
Watamaniuk S N J and Sekuler R Temporal and spatial integration in dynamic random-dot stimuli Vision Research 1992 32 12 2341-2347
[62]
Sun W, Gao Z, Zhai G, Zhang J, Wang Z, Zhu Y. An improved algorithm for real-time dual-view display. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2020, 1–5
[63]
Zhai G and Min X Perceptual image quality assessment: a survey Science China Information Sciences 2020 63 11 211301
[64]
Chen Y, Liu N, Zhai G, Gu K, Wang J, Gao Z, Zhu Y. Quality assessment for dual-view display system. In: Proceedings of Visual Communications and Image Processing. 2016, 1–4
[65]
Chen Y, Zhai G, Gu K, Zhang X, Lin W, Zhou J. Benchmarking screen content image quality evaluation in spatial psychovisual modulation display system. In: Proceedings of Pacific Rim Conference on Multimedia. 2018, 629–640
[66]
Scherzinger A L and Hendee W R Basic principles of magnetic resonance imaging-an update The Western Journal of Medicine 1985 143 6 782-792
[67]
Bushong S C, Clarke G. Magnetic Resonance Imaging: Physical and Biological Principles. Elsevier Health Sciences, 2014
[68]
Lambooij M, IJsselsteijn W, and Heynderickx I Visual discomfort of 3d tv: assessment methods and modeling Displays 2011 32 4 209-218
[69]
Taylor S A. CCD and CMOS imaging array technologies: technology review. Xerox Research Centre Europe,Technical Report EPC-1998-106, 1998, 1–14
[70]
ISO/IEC. Information technology-automatic identification and data capture techniques-QR Code 2005 bar code symbology specification. see Iso.org Website, 2006
[71]
Song K, Liu N, Gao Z, Zhang J, Zhai G, Zhang X P. Deep restoration of invisible QR code from TPVM display. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops. 2020, 1–6
[72]
Siwek S E. The true cost of copyright industry piracy to the US economy. IPI Center for Technology Freedom, 2007
[73]
Dorning J. Intellectual Property Theft: A Threat to U.S. Workers, Industries, and Our Economy. DPE Research Department, 2014
[74]
NEWS B. The fact and fiction of camcorder piracy. see Bbc.co.uk Website, 2015
[75]
Byers S, Cranor L F, Cronin E, Korman D, and McDaniel P An analysis of security vulnerabilities in the movie production and distribution process Telecommunications Policy 2004 28 78 619-644
[76]
Haitsma J, Kalker T. A watermarking scheme for digital cinema. In: Proceedings of International Conference on Image Processing. 2001, 487–489
[77]
Nguyen P, Balter R, Montfort N, Baudry S. Registration methods for nonblind watermark detection in digital cinema applications. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 553–562
[78]
Lubin J, Bloom J A, Cheng H. Robust content-dependent high-fidelity watermark for tracking in digital cinema. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 536–545
[79]
Nakashima Y, Tachibana R, and Babaguchi N Watermarked movie soundtrack finds the position of the camcorder in a theater IEEE Transactions on Multimedia 2009 11 3 443-454
[80]
Davis J, Hsieh Y H, and Lee H C Humans perceive flicker artifacts at 500 hz Scientific Reports 2015 5 7861
Index Terms
- Psycho-visual modulation based information display: introduction and survey
Index terms have been assigned to the content through auto-classification.
Recommendations
Saliency detection based on scale selectivity of human visual system
ICONIP'11: Proceedings of the 18th international conference on Neural Information Processing - Volume Part IIt is well known that visual attention and saliency mechanisms play an important role in human visual perception. This paper proposes a novel bottom-up saliency mechanism through scale space analysis. The research on human perception had shown that our ...
Survey of visual just noticeable difference estimation
The concept of just noticeable difference (JND), which accounts for the visibility threshold (visual redundancy) of the human visual system, is useful in perception-oriented signal processing systems. In this work, we present a comprehensive review of ...
Visual acuity inspired saliency detection by using sparse features
In this paper, we propose a new computational model of visual attention based on the relevant characteristics of the Human Visual System (HVS) and sparse features. The input image is first divided into small image patches. Then the sparse features of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© Higher Education Press 2020.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 01 June 2021
Accepted: 05 March 2019
Received: 01 August 2018
Author Tags
Qualifiers
- Review-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in