PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR Displays
Authors: 
Kenneth Chen, Thomas Wan, Nathan Matsuda, Ajit Ninan, Alexandre Chapiro, Qi SunAuthors Info & Claims
Article No.: 67, Pages 1 - 17
Abstract
Display power consumption is an emerging concern for untethered devices. This goes double for augmented and virtual extended reality (XR) displays, which target high refresh rates and high resolutions while conforming to an ergonomically light form factor. A number of image mapping techniques have been proposed to extend battery usage. However, there is currently no comprehensive quantitative understanding of how the power savings provided by these methods compare to their impact on visual quality. We set out to answer this question.
To this end, we present a perceptual evaluation of algorithms (PEA) for power optimization in XR displays (PODs). Consolidating a portfolio of six power-saving display mapping approaches, we begin by performing a large-scale perceptual study to understand the impact of each method on perceived quality in the wild. This results in a unified quality score for each technique, scaled in just-objectionable-difference (JOD) units. In parallel, each technique is analyzed using hardware-accurate power models.
The resulting JOD-to-Milliwatt transfer function provides a first-of-its-kind look into tradeoffs offered by display mapping techniques, and can be directly employed to make architectural decisions for power budgets on XR displays. Finally, we leverage our study data and power models to address important display power applications like the choice of display primary, power implications of eye tracking, and more1.
Supplementary Material
supplemental
- Download
- 49.21 MB
References
[1]
Takeyuki Ajito, Takashi Obi, Masahiro Yamaguchi, and Nagaaki Ohyama. 2000. Expanded color gamut reproduced by six-primary projection display. In Projection Displays 2000: Sixth in a Series, Vol. 3954. SPIE, 130--137.
[2]
Bhojan Anand, Karthik Thirugnanam, Jeena Sebastian, Pravein G Kannan, Akhihebbal L Ananda, Mun Choon Chan, and Rajesh Krishna Balan. 2011. Adaptive display power management for mobile games. In Proceedings of the 9th international conference on Mobile systems, applications, and services. 57--70.
[3]
Yuta Asano and Minqi Wang. 2024. An investigation of color difference for binocular rivalry and a preliminary rivalry metric, Δ E* bino. Color Research & Application 49, 1 (2024), 51--64.
[4]
ITU Recommendation BT. 2002. Parameter values for the hdtv standards for production and international programme exchange. International Telecommunication Union, Recommendation, May (2002).
[5]
Shaoyu Chen, Budmonde Duinkharjav, Xin Sun, Li-Yi Wei, Stefano Petrangeli, Jose Echevarria, Claudio Silva, and Qi Sun. 2022. Instant reality: Gaze-contingent perceptual optimization for 3d virtual reality streaming. IEEE Transactions on Visualization and Computer Graphics 28, 5 (2022), 2157--2167.
[6]
Wei-Chung Cheng and Massoud Pedram. 2004. Power minimization in a backlit TFT-LCD display by concurrent brightness and contrast scaling. IEEE Transactions on Consumer Electronics 50, 1 (2004), 25--32.
[7]
Inseok Choi, Hojun Shim, and Naehyuck Chang. 2002. Low-power color TFT LCD display for hand-held embedded systems. In Proceedings of the 2002 international symposium on Low power electronics and design. 112--117.
[8]
Michael A. Cohen, Thomas L. Botch, and Caroline E. Robertson. 2020. The limits of color awareness during active, real-world vision. Proceedings of the National Academy of Sciences 117, 24 (2020), 13821--13827. arXiv:https://www.pnas.org/content/117/24/13821.full.pdf
[9]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248--255.
[10]
Andrew M Derrington, John Krauskopf, and Peter Lennie. 1984. Chromatic mechanisms in lateral geniculate nucleus of macaque. The Journal of physiology 357, 1 (1984), 241--265.
[11]
Takahiro Doi, Laurie Wilcox, and T Scott Murdison. 2023. Stereopsis from interocular temporal delay: disentangling the effects of target versus background luminance. Journal of Vision 23, 9 (2023), 5159--5159.
[12]
Mian Dong and Lin Zhong. 2011a. Chameleon: A color-adaptive web browser for mobile OLED displays. In Proceedings of the 9th international conference on Mobile systems, applications, and services. 85--98.
[13]
Mian Dong and Lin Zhong. 2011b. Power modeling and optimization for OLED displays. IEEE Transactions on Mobile Computing 11, 9 (2011), 1587--1599.
[14]
Budmonde Duinkharjav, Praneeth Chakravarthula, Rachel Brown, Anjul Patney, and Qi Sun. 2022a. Image features influence reaction time: A learned probabilistic perceptual model for saccade latency. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--15.
[15]
Budmonde Duinkharjav, Kenneth Chen, Abhishek Tyagi, Jiayi He, Yuhao Zhu, and Qi Sun. 2022b. Color-Perception-Guided Display Power Reduction for Virtual Reality. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 41, 6 (2022), 144:1--144:16.
[16]
Walter H Ehrenstein, Birgit E Arnold-Schulz-Gahmen, and Wolfgang Jaschinski. 2005. Eye preference within the context of binocular functions. Graefe's Archive for Clinical and Experimental Ophthalmology 243 (2005), 926--932.
[17]
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023. DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data. arXiv preprint arXiv:2306.09344 (2023).
[18]
Franco Gatti, Andrea Acquaviva, Luca Benini, and Bruno Ricco'. 2002. Low power control techniques for TFT LCD displays. In Proceedings of the 2002 international conference on Compilers, architecture, and synthesis for embedded systems. 218--224.
[19]
B Hahn. 2016. Closing the green efficiency gap, status and recent approaches. In Proc. DOE Workshop Raleigh.
[20]
B Hahn, A Weimar, M Peter, and J Baur. 2008. High-power InGaN LEDs: Present status and future prospects. Light-Emitting Diodes: Research, Manufacturing, and Applications XII 6910 (2008), 9--16.
[21]
Thorsten Hansen, Lars Pracejus, and Karl R Gegenfurtner. 2009. Color perception in the intermediate periphery of the visual field. Journal of vision 9, 4 (2009), 26--26.
[22]
Injoon Hong, Kyeongryeol Bong, and Hoi-Jun Yoo. 2018. Challenges of eye tracking systems for mobile XR glasses. In Applications of Digital Image Processing XLI, Vol. 10752. SPIE, 391--397.
[23]
HTC. 2020. VIVE Pro Eye Specs & User Guide. https://developer.vive.com/resources/hardware-guides/vive-pro-eye-specs-user-guide/.
[24]
Yu Hu, Minchen Wei, and Ming Ronnier Luo. 2020. Observer metamerism to display white point using different primary sets. Optics Express 28, 14 (2020), 20305--20323.
[25]
Fu-Chung Huang, Dawid Pająk, Jonghyun Kim, Jan Kautz, and David Luebke. 2017. Mixed-primary factorization for dual-frame computational displays. ACM Trans. Graph. 36, 4, Article 149 (jul 2017), 13 pages.
[26]
Yize Jin, Meixu Chen, Todd Goodall Bell, Zhaolin Wan, and Alan Bovik. 2020. Study of 2D foveated video quality in virtual reality. In Applications of digital image processing XLIII, Vol. 11510. SPIE, 18--26.
[27]
Yize Jin, Meixu Chen, Todd Goodall, Anjul Patney, and Alan Bovik. 2019. LIVE-Facebook Technologies-Compressed Virtual Reality (LIVE-FBT-FCVR) Databases. Online: http://live.ece.utexas.edu/research/LIVEFBTFCVR/index.html (2019).
[28]
Yize Jin, Meixu Chen, Todd Goodall, Anjul Patney, and Alan C Bovik. 2021. Subjective and objective quality assessment of 2D and 3D foveated video compression in virtual reality. IEEE Transactions on Image Processing 30 (2021), 5905--5919.
[29]
Akshay Jindal, Krzysztof Wolski, Karol Myszkowski, and Rafał K Mantiuk. 2021. Perceptual model for adaptive local shading and refresh rate. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1--18.
[30]
Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.
[31]
Isaac Kauvar, Samuel J. Yang, Liang Shi, Ian McDowall, and Gordon Wetzstein. 2015. Adaptive color display via perceptually-driven factored spectral projection. ACM Trans. Graph. 34, 6, Article 165 (nov 2015), 10 pages.
[32]
Koray Kavaklı, Liang Shi, Hakan Urey, Wojciech Matusik, and Kaan Akşit. 2023. Multicolor Holograms Improve Brightness in Holographic Displays. In SIGGRAPH Asia 2023 Conference Papers (Sydney, NSW, Australia) (SA '23). Association for Computing Machinery, New York, NY, USA, Article 20, 11 pages.
[33]
Louis Kerofsky and Scott Daly. 2006. Brightness preservation for LCD backlight dimming. Journal of the Society for Information Display 14, 12 (2006), 1111--1118.
[34]
Jeong-Sik Kim and Seung-Woo Lee. 2020. Peripheral dimming: A new low-power technology for OLED display based on gaze tracking. IEEE Access 8 (2020), 209064--209073.
[35]
F Konig, Kenro Ohsawa, Masahiro Yamaguchia, Nagaaki Ohyama, and Bernhard Hill. 2002. A multiprimary display: Optimized control values for displaying tristimulus values. In IS AND TS PICS CONFERENCE. Society for Imaging Science & Technology, 215--220.
[36]
Brooke Krajancich, Petr Kellnhofer, and Gordon Wetzstein. 2023. Towards Attention-aware Foveated Rendering. ACM Trans. Graph. 42, 4, Article 77 (jul 2023), 10 pages.
[37]
Timo Kunkel and Ajit Ninan. 2023. HDR and Wide Color Gamut Display Technologies and Considerations. Fundamentals and Applications of Colour Engineering (2023), 311--334.
[38]
Hongru Li, Han Gao, Gökhan Kirca, and Zhichun Lei. 2020. Energy-saving display by color pixel re-representation. IEEE Transactions on Circuits and Systems for Video Technology 30, 12 (2020), 4728--4738.
[39]
Bruce Lindbloom. 2017. Chromatic Adaptation. http://www.brucelindbloom.com/index.html?Eqn_ChromAdapt.html.
[40]
MR Luo and RWG Hunt. 1998. The structure of the CIE 1997 colour appearance model (CIECAM97s). Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 23, 3 (1998), 138--146.
[41]
M Ronnier Luo et al. 2000. A review of chromatic adaptation transforms. Review of Progress in Coloration and Related Topics 30 (2000), 77--92.
[42]
Rafał K. Mantiuk, Maliha Ashraf, and Alexandre Chapiro. 2022. stelaCSF: a unified model of contrast sensitivity as the function of spatio-temporal frequency, eccentricity, luminance and area. ACM Trans. Graph. 41, 4, Article 145 (jul 2022), 16 pages.
[43]
Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: a visible difference predictor for wide field-of-view video. ACM Trans. Graph. 40, 4, Article 49 (jul 2021), 19 pages.
[44]
Nathan Matsuda, Alex Chapiro, Yang Zhao, Clinton Smith, Romain Bachy, and Douglas Lanman. 2022. Realistic Luminance in VR. In SIGGRAPH Asia 2022 Conference Papers (Daegu, Republic of Korea) (SA '22). Association for Computing Machinery, New York, NY, USA, Article 21, 8 pages.
[45]
Johannes Meyer. 2022. Towards energy efficient mobile eye tracking for AR glasses through optical sensor technology. arXiv preprint arXiv:2212.03189 (2022).
[46]
Microsoft. 2022. Adaptive brightness. https://support.microsoft.com/en-us/windows/content-adaptive-brightness-control-in-windows-292d1f7f-9e02-4b37-a9c8-dab3e1727e78.
[47]
Aliaksei Mikhailiuk, Clifford Wilmot, Maria Perez-Ortiz, Dingcheng Yue, and Rafal Mantiuk. 2021. Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization. In 2020 IEEE International Conference on Pattern Recognition (ICPR).
[48]
Bernard Moulden et al. 1988. Border effects on brightness: a review of findings, models and issues. Spatial vision 3, 4 (1988), 225--262.
[49]
Yukio Narukawa, Masatsugu Ichikawa, Daisuke Sanga, Masahiko Sano, and Takashi Mukai. 2010. White light emitting diodes with super-high luminous efficacy. Journal of physics D: Applied physics 43, 35 (2010), 354002.
[50]
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--12.
[51]
Jonathan Peirce, Jeremy R Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. 2019. PsychoPy2: Experiments in behavior made easy. Behavior research methods 51 (2019), 195--203.
[52]
Maria Perez-Ortiz, Aliaksei Mikhailiuk, Emin Zerman, Vedad Hulusic, Giuseppe Valenzise, and Rafał K Mantiuk. 2019. From pairwise comparisons and rating to a unified quality scale. IEEE Transactions on Image Processing 29 (2019), 1139--1151.
[53]
M Peter, A Laubsch, P Stauss, A Walter, J Baur, and B Hahn. 2008. Green ThinGaN power-LED demonstrates 100 lm. physica status solidi c 5, 6 (2008), 2050--2052.
[54]
Linghui Rao, Yongmin Park, Alex Klement, Chris Kang, Eric Park, Jim Zhuang, Cheonhong Kim, Hsin-Ying Chiu, Ross Ning, Daozhi Wang, et al. 2023. 5-1: Invited Paper: Infinite Display for Meta Quest Pro. In SID Symposium Digest of Technical Papers, Vol. 54. Wiley Online Library, 32--35.
[55]
Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. 2010. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann. 152--159 pages.
[56]
Michael Reiss and Gilfe Reiss. 1997. Ocular dominance: some family data. Laterality: Asymmetries of Body, Brain and Cognition 2, 1 (1997), 7--16.
[57]
Dario Schiavon, Michael Binder, Andreas Loeffler, and Matthias Peter. 2013. Optically pumped GaInN/GaN multiple quantum wells for the realization of efficient green light-emitting devices. Applied Physics Letters 102, 11 (2013).
[58]
Kalpana Seshadrinathan and Alan C Bovik. 2007. A structural similarity metric for video based on motion models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07, Vol. 1. IEEE, I--869.
[59]
Gaurav Sharma, Wencheng Wu, and Edul N Dalal. 2005. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 30, 1 (2005), 21--30.
[60]
Rahul Singh, Muhammad Huzaifa, Jeffrey Liu, Anjul Patney, Hashim Sharif, Yifan Zhao, and Sarita Adve. 2023. Power, performance, and image quality tradeoffs in foveated rendering. In 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 205--214.
[61]
Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2017. How do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics (2017).
[62]
Daniel Steigerwald, Serge Rudaz, Heng Liu, R Scott Kern, Werner Götz, and Robert Fletcher. 1997. III--V nitride semiconductors for high-performance blue and green light-emitting devices. Jom 49 (1997), 18--23.
[63]
Masanori Takaya, Ken Ito, Gosuke Ohashi, and Yoshifumi Shimodaira. 2005. Color-conversion method for a multi-primary display to reduce power consumption and conversion time. Journal of the Society for Information Display 13, 8 (2005), 685--690.
[64]
Kiat Wee Tan, Tadashi Okoshi, Archan Misra, and Rajesh Krishna Balan. 2013. FOCUS: a usable & effective approach to OLED display power management. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 573--582.
[65]
Matthew Trentacoste, Wolfgang Heidrich, Lorne Whitehead, Helge Seetzen, and Greg Ward. 2007a. Photometric image processing for high dynamic range displays. Journal of Visual Communication and Image Representation 18, 5 (2007), 439--451.
[66]
Matthew Trentacoste, Wolfgang Heidrich, Lorne Whitehead, Helge Seetzen, and Greg Ward. 2007b. Photometric image processing for high dynamic range displays. Journal of Visual Communication and Image Representation 18 (10 2007), 439--451.
[67]
Minqi Wang, Jian Ding, Dennis M Levi, and Emily A Cooper. 2023. The effect of interocular contrast differences on the appearance of augmented reality imagery. ACM Transactions on Applied Perception (2023).
[68]
Andrew B Watson and Denis G Pelli. 1983. QUEST: A Bayesian adaptive psychometric method. Perception & psychophysics 33, 2 (1983), 113--120.
[69]
Tan Kiat Wee, Eduardo Cuervo, and Rajesh Balan. 2018. FocusVR: Effective 8 usable VR display power management. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--25.
[70]
Shu Wei, Desmond Bloemers, and Aitor Rovira. 2023. A preliminary study of the eye tracker in the meta quest pro. In Proceedings of the 2023 ACM International Conference on Interactive Media Experiences. 216--221.
[71]
Waloddi Weibull. 1951. A statistical distribution function of wide applicability. Journal of applied mechanics (1951).
[72]
Jeremy M Wolfe. 1983. Influence of spatial frequency, luminance, and duration on binocular rivalry and abnormal fusion of briefly presented dichoptic stimuli. Perception 12, 4 (1983), 447--456.
[73]
Hao Xie, Carlos Eduardo Rodríguez-Pardo, and Gaurav Sharma. 2017. Multiobjective optimization for color display primary designs. Journal of Electronic Imaging 26, 6 (2017), 063013--063013.
[74]
Aron Yu and Kristen Grauman. 2015. Just noticeable differences in visual attributes. In Proceedings of the IEEE International Conference on Computer Vision. 2416--2424.
[75]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586--595.
[76]
Fangcheng Zhong, Akshay Jindal, Ali Özgür Yöntem, Param Hanji, Simon J. Watt, and Rafał K. Mantiuk. 2021. Reproducing reality with a high-dynamic-range multi-focal stereo display. ACM Trans. Graph. 40, 6, Article 241 (dec 2021), 14 pages.
[77]
Fangcheng Zhong, George Alex Koulieris, George Drettakis, Martin S Banks, Mathieu Chambe, Frédo Durand, and Rafał K Mantiuk. 2019. DiCE: Dichoptic contrast enhancement for VR and stereo displays. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.
Index Terms
- PEA-PODs: Perceptual Evaluation of Algorithms for Power Optimization in XR Displays
Recommendations
Color-Perception-Guided Display Power Reduction for Virtual Reality
Battery life is an increasingly urgent challenge for today's untethered VR and AR devices. However, the power efficiency of head-mounted displays is naturally at odds with growing computational requirements driven by better resolution, refresh rate, and ...
Exploring Perceptual and Motor Gestalt in Touchless Interactions with Distant Displays
TEI '15: Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied InteractionMarkerless motion-sensing promises to position touchless interactions successfully in various domains (e.g., entertainment or surgery) because they are deemed natural. This naturalness, however, depends upon the mechanics of touchless interaction that ...
Navigating on handheld displays: Dynamic versus static peephole navigation
Handheld displays leave little space for the visualization and navigation of spatial layouts representing rich information spaces. The most common navigation method for handheld displays is static peephole navigation: The peephole is static and we move ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 July 2024
Published in TOG Volume 43, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 394Total Downloads
- Downloads (Last 12 months)394
- Downloads (Last 6 weeks)128
Reflects downloads up to 20 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in