research-article

ChromaCode: A Fully Imperceptible Screen-Camera Communication System

Authors:

Zheng YangAuthors Info & Claims

MobiCom '18: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking

Pages 575 - 590

https://doi.org/10.1145/3241539.3241543

Published: 15 October 2018 Publication History

Abstract

Hidden screen-camera communication techniques emerge as a new paradigm that embeds data imperceptibly into regular videos while remaining unobtrusive to human viewers. Three key goals on imperceptible, high rate, and reliable communication are desirable but conflicting, and existing solutions usually made a trade-off among them. In this paper, we present the design and implementation of ChromaCode, a screen-camera communication system that achieves all three goals simultaneously. In our design, we consider for the first time color space for perceptually uniform lightness modifications. On this basis, we design an outcome-based adaptive embedding scheme, which adapts to both pixel lightness and regional texture. Last, we propose a concatenated code scheme for robust coding and devise multiple techniques to overcome various screen-camera channel errors. Our prototype and experiments demonstrate that ChromaCode achieves remarkable raw throughputs of >700 kbps, data goodputs of 120 kbps with BER of 0.05, and with fully imperceptible flicker for viewing proved by user study, which significantly outperforms previous works.

References

[1]

{n. d.}. FFmpeg. https://ffmpeg.org/

[2]

{n. d.}. The Interactive Effect: A Key to Surviving in the Attention Economy of a Mobile-First World. https://medium.com/ipg-medialab/ the-interactive-effect-a-key-to-surviving-in-the-attentioneconomy- of-a-mobile-first-world-dcd8ace76ab1

[3]

{n. d.}. MediaCodec | Android Developers. https://developer.android. com/reference/android/media/MediaCodec

[4]

{n. d.}. Moiré_pattern. https://en.wikipedia.org/wiki/MoirÃ pattern

[5]

{n. d.}. OpenCV library. https://opencv.org/

[6]

{n. d.}. Qt | Cross-platform software development for embedded & desktop. https://www.qt.io/

[7]

{n. d.}. Xiph.org :: Derf's Test Media Collection. https://media.xiph. org/video/derf/

[8]

{n. d.}. Zootopia | Disney Movies. http://movies.disney.com/zootopia

[9]

2002. Publications Briefly Mentioned: CIE 142--2001, improvement to industrial colour-difference evaluation. Color Research & Application 27, 1 (2002), 61--61.

[10]

2008. Colorimetry -- Part 4: CIE 1976 L*a*b* Colour space.

[11]

2009. Colorimetry -- Part 5: CIE 1976 L*u*v* Colour space and u', v' uniform chromaticity scale diagram.

[12]

2013. Colorimetry -- Part 6: CIEDE2000 Colour-Difference Formula.

[13]

2016. Information technology - Automatic identification and data capture techniques - QR Code 2005 bar code symbology specification.

[14]

G. S. Brindley, J. J. Du Croz, and W. A. H. Rushton. 1966. The flicker fusion frequency of the blue-sensitive mechanism of colour vision. The Journal of Physiology 183, 2 (1966), 497--500.

[15]

J. Canny. 1986. A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8, 6 (1986), 679--698.

Digital Library

[16]

Charles Chubb, George Sperling, and Joshua A. Solomon. 1989. Texture interactions determine perceived contrast. In Proceedings of the National Academy of Sciences of the United States of America (PNAS '89). 9631--9635.

[17]

H.S.M. Coxeter. 1969. Introduction to geometry. Wiley. https://books. google.com/books?id=c0ld-crynsIC

[18]

Hugh S. Fairman, Michael H. Brill, and Henry Hemmendinger. 1997. How the CIE 1931 color-matching functions were derived from Wright-Guild data. Color Research & Application 22, 1 (1997), 11-- 23.

[19]

D. J. Fleet and D. J. Heeger. 1997. Embedding invisible information in color images. In Proceedings of International Conference on Image Processing (ICIP '97), Vol. 1. 532--535 vol.1.

Digital Library

[20]

Arunabha Ghosh, David R. Wolter, Jeffrey G. Andrews, and Runhua Chen. 2005. Broadband wireless access with WiMax/802.16: current performance benchmarks and future potential. IEEE Communications Magazine 43 (2005), 129--136.

Digital Library

[21]

Tian Hao, Ruogu Zhou, and Guoliang Xing. 2012. COBRA: Color Barcode Streaming for Smartphone Systems. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services (MobiSys '12). ACM, New York, NY, USA, 85--98.

Digital Library

[22]

Robert M. Haralick, K. Shanmugam, and Its'Hak Dinstein. 1973. Textural Features for Image Classification. IEEE Transactions on Systems, Man, and Cybernetics SMC-3, 6 (1973), 610--621.

[23]

P. V. C. Hough. 1959. Machine Analysis of Bubble Chamber Pictures. Conf. Proc. C590914 (1959), 554--558.

[24]

Wenjun Hu, Hao Gu, and Qifan Pu. 2013. LightSync: Unsynchronized Visual Communication over Screen-camera Links. In Proceedings of the 19th Annual International Conference on Mobile Computing and Networking (MobiCom '13). ACM, New York, NY, USA, 15--26.

Digital Library

[25]

M. Izz, Z. Li, H. Liu, Y. Chen, and F. Li. 2016. Uber-in-light: Unobtrusive visible light communication leveraging complementary color channel. In The 35th Annual IEEE International Conference on Computer Communications (INFOCOM '16). 1--9.

[26]

Rolf G. Kuehni. 2003. Historical Development of Color Space and Color Difference Formulas. John Wiley & Sons, Inc., 204--270. https://doi. org/10.1002/0471432261.ch6

[27]

IPG Media Lab. 2017. Media Trial Report: MAGNA and IPG Media Lab Turbocharge Skippable Pre-Roll Campaign.

[28]

Tianxing Li, Chuankai An, Xinran Xiao, Andrew T. Campbell, and Xia Zhou. 2015. Real-Time Screen-Camera Communication Behind Any Scene. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '15). ACM, New York, NY, USA, 197--211.

Digital Library

[29]

Hao Xue Liu, Bing Wu, Yu Liu, Min Huang, and Yan Fang Xu. 2013. A Discussion on Printing Color Difference Tolerance by CIEDE2000 Color Difference Formula. In Advances in Printing and Packaging Technologies (Applied Mechanics and Materials), Vol. 262. Trans Tech Publications, 96--99.

[30]

M R Luo and B Rigg. 1986. Uniform Colour Space Based on the CMC(l:c) Colour-difference Formula. Journal of the Society of Dyers and Colourists 102, 5--6 (1986), 164--171.

[31]

David L. MacAdam. 1942. Visual Sensitivities to Color Differences in Daylight?. J. Opt. Soc. Am. 32, 5 (1942), 247--274.

[32]

R. McDonald and K J Smith. 1995. CIE94-a new colour-difference formula*. Journal of the Society of Dyers and Colourists 111, 12 (1995), 376--379.

[33]

Marci Meingast, Christopher Geyer, and Shankar Sastry. 2005. Geometric Models of Rolling-Shutter Cameras. CoRR abs/cs/0503076 (2005). arXiv:cs/0503076 http://arxiv.org/abs/cs/0503076

[34]

S. Naribole, S. Chen, E. Heng, and E. Knightly. 2017. LiRa: A WLAN Architecture for Visible Light Communication with a Wi-Fi Uplink. In 2017 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON '17). 1--9.

[35]

Viet Nguyen, Yaqin Tang, Ashwin Ashok, Marco Gruteser, Kristin Dana, Wenjun Hu, Eric Wengrowski, and Narayan Mandayam. 2016. High-rate flicker-free screen-camera communication with spatially adaptive embedding. In The 35th Annual IEEE International Conference on Computer Communications (INFOCOM '16). 1--9.

Digital Library

[36]

Samuel David Perli, Nabeel Ahmed, and Dina Katabi. 2010. PixNet: LCD-camera Pairs As Communication Links. In Proceedings of the Annual Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM '10). ACM, New York, NY, USA, 451--452.

Digital Library

[37]

Austin Roorda and David R. Williams. 1999. The arrangement of the three cone classes in the living human eye. Nature (1999), 520--522.

[38]

Shuyu Shi, Lin Chen, Wenjun Hu, and Marco Gruteser. 2015. Reading Between Lines: High-rate, Non-intrusive Visual Codes Within Regular Videos via ImplicitCode. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 157--168.

Digital Library

[39]

George Stockman and Linda G. Shapiro. 2001. Computer Vision (1st ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA.

Digital Library

[40]

Zhao Tian, Kevin Wright, and Xia Zhou. 2016. The darkLight Rises: Visible Light Communication in the Dark. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom '16). ACM, New York, NY, USA, 2--15.

Digital Library

[41]

ANDREW J. VITERBI. 2011. Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm. Co-Published with Indian Institute of Science (IISc), Bangalore, India, 41--50.

Digital Library

[42]

Anran Wang, Zhuoran Li, Chunyi Peng, Guobin Shen, Gan Fang, and Bing Zeng. 2015. InFrame++: Achieve Simultaneous Screen-Human Viewing and Hidden Screen-Camera Communication. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '15). ACM, New York, NY, USA, 181--195.

Digital Library

[43]

Anran Wang, Shuai Ma, Chunming Hu, Jinpeng Huai, Chunyi Peng, and Guobin Shen. 2014. Enhancing Reliability to Boost the Throughput over Screen-camera Links. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom '14). ACM, New York, NY, USA, 41--52.

Digital Library

[44]

Anran Wang, Chunyi Peng, Ouyang Zhang, Guobin Shen, and Bing Zeng. 2014. InFrame: Multiflexing Full-Frame Visible Communication Channel for Humans and Devices. In Proceedings of the 13th ACM Workshop on Hot Topics in Networks (HotNets '14). ACM, New York, NY, USA, Article 23, 7 pages.

Digital Library

[45]

Qian Wang, Kui Ren, Man Zhou, Tao Lei, Dimitrios Koutsonikolas, and Lu Su. 2016. Messages Behind the Sound: Real-time Hidden Acoustic Signal Capture with Smartphones. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom '16). ACM, New York, NY, USA, 29--41.

Digital Library

[46]

Zhouping Wei, Jian Wang, Helen Nichol, Sheldon Wiebe, and Dean Chapman. 2012. A median-Gaussian filtering framework for Moiré pattern noise removal from X-ray microscopy image. Micron 43, 2 (2012), 170 -- 176.

[47]

Zhe Yang, Yuting Bao, Chuhao Luo, Xingya Zhao, Siyu Zhu, Chunyi Peng, Yunxin Liu, and Xinbing Wang. 2016. ARTcode: Preserve Art and Code in Any Image. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 904--915.

Digital Library

[48]

Jialiang Zhang, Chi Zhang, Xinyu Zhang, and Suman Banerjee. 2016. Towards a Visible Light Network Architecture for Continuous Communication and Localization. In Proceedings of the 3rd Workshop on Visible Light Communication Systems (VLCS '16). ACM, New York, NY, USA, 49--54.

Digital Library

Cited By

Xiao RZhao LQian FYang LHan JOkoshi TKo JLiKamWa R(2024)Practical Optical Camera Communication Behind Unseen and Complex BackgroundsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661866(113-126)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661866
Liao ZLuo ZHuang QZhang LWu FZhang QChen G(2024)Gesture Recognition Using Visible Light on Mobile DevicesIEEE/ACM Transactions on Networking10.1109/TNET.2024.336999632:4(2920-2935)Online publication date: Aug-2024
https://doi.org/10.1109/TNET.2024.3369996
Jin MHe YLiu YWang X(2024)Covert Communication With Acoustic NoiseIEEE/ACM Transactions on Networking10.1109/TNET.2023.328669232:1(207-221)Online publication date: Feb-2024
https://doi.org/10.1109/TNET.2023.3286692
Show More Cited By

Index Terms

ChromaCode: A Fully Imperceptible Screen-Camera Communication System
1. Human-centered computing
  1. Ubiquitous and mobile computing
2. Networks
  1. Network types
    1. Mobile networks

Recommendations

InFrame++: Achieve Simultaneous Screen-Human Viewing and Hidden Screen-Camera Communication
MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

Recent efforts in visible light communication over screen-camera links have exploited the display for data communication. Such practices, albeit convenient, have led to contention between space allocated for users and content reserved for devices, in ...
Demo: Achieving Simultaneous Screen-Human Viewing and Hidden Screen-Camera Communication
MobiSys '15: Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services

We present and demonstrate INFRAME++, a novel system that enables concurrent, dual-mode, full-frame communication for both users and devices. It achieves unobtrusive screen-camera data communication without affecting the primary video-viewing experience ...
Screen-Camera Communication via Matrix Barcode Utilizing Imperceptible Color Vibration
UIST '18 Adjunct: Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Communication between screens and cameras has attracted attention as a ubiquitous information source, motivated by the widespread use of smartphones and the increase of public advertising and information screens. We propose embedding matrix barcodes ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiCom '18: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking

October 2018

884 pages

ISBN:9781450359030

DOI:10.1145/3241539

General Chairs:
Rajeev Shorey
TCS Innovation Labs, India
,
Rohan Murty
Soroco, USA/India
,
Program Chairs:
Yingying (Jennifer) Chen
WINLAB, Rutgers University, USA
,
Kyle Jamieson
Princeton University, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key Research Plan of China

Conference

MobiCom '18

Sponsor:

SIGMOBILE

MobiCom '18: The 24th Annual International Conference on Mobile Computing and Networking

October 29 - November 2, 2018

New Delhi, India

Acceptance Rates

MobiCom '18 Paper Acceptance Rate 42 of 187 submissions, 22%;

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
816
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xiao RZhao LQian FYang LHan JOkoshi TKo JLiKamWa R(2024)Practical Optical Camera Communication Behind Unseen and Complex BackgroundsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661866(113-126)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661866
Liao ZLuo ZHuang QZhang LWu FZhang QChen G(2024)Gesture Recognition Using Visible Light on Mobile DevicesIEEE/ACM Transactions on Networking10.1109/TNET.2024.336999632:4(2920-2935)Online publication date: Aug-2024
https://doi.org/10.1109/TNET.2024.3369996
Jin MHe YLiu YWang X(2024)Covert Communication With Acoustic NoiseIEEE/ACM Transactions on Networking10.1109/TNET.2023.328669232:1(207-221)Online publication date: Feb-2024
https://doi.org/10.1109/TNET.2023.3286692
Ji XZhang JZou SChen YQu GXu W(2024)MagView++: Data Exfiltration via CPU Magnetic Signals Under Video DecodingIEEE Transactions on Mobile Computing10.1109/TMC.2023.326240023:3(2486-2503)Online publication date: Mar-2024
https://doi.org/10.1109/TMC.2023.3262400
Zhang HYu XZhang ZZhu B(2024)Robust and Imperceptible Commercial Camera-Screen Communication with 60Hz Refresh RateICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446206(4910-4914)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10446206
Yan YWang ZHuang QYang Z(2023)MousePath: Lightweight phone-to-web information sharing via mouse interfacePervasive and Mobile Computing10.1016/j.pmcj.2023.10175690(101756)Online publication date: Mar-2023
https://doi.org/10.1016/j.pmcj.2023.101756
Li JPan CPan JLin JLu MJiang ZFang J(2022)Long-Distance, Real-Time LED Display-Camera Communication System Based on LED Point Clustering and Lightweight Image ProcessingPhotonics10.3390/photonics91007219:10(721)Online publication date: 3-Oct-2022
https://doi.org/10.3390/photonics9100721
Yan YHuang ZXudu FYang Z(2022)Enabling Tangible Interaction on Non-touch Displays with Optical Mouse Sensor and Visible Light CommunicationProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517666(1-14)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517666
Jin MHe YLiu YWang X(2022)Furtively Connecting IoT Devices with Acoustic Noise2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)10.1109/IPSN54338.2022.00023(195-207)Online publication date: May-2022
https://doi.org/10.1109/IPSN54338.2022.00023
Wang THan HWang Z(2020)FareQR: Fast and Reliable Screen-Camera Transfer System for Mobile Devices using QR Code2020 IEEE/ACM Symposium on Edge Computing (SEC)10.1109/SEC50012.2020.00053(364-369)Online publication date: Nov-2020
https://doi.org/10.1109/SEC50012.2020.00053
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

ePub

View this article in ePub.

Media

Figures

Other

Tables

View Table of Contents