research-article

Relative CNN-RNN: Learning Relative Atmospheric Visibility From Images

Authors:

Chi-Keung TangAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 28, Issue 1

Pages 45 - 55

https://doi.org/10.1109/TIP.2018.2857219

Published: 01 January 2019 Publication History

Abstract

We propose a deep learning approach for directly estimating relative atmospheric visibility from outdoor photos without relying on weather images or data that require expensive sensing or custom capture. Our data-driven approach capitalizes on a large collection of Internet images to learn rich scene and visibility varieties. The relative CNN–RNN coarse-to-fine model, where CNN stands for convolutional neural network and RNN stands for recurrent neural network, exploits the joint power of relative support vector machine, which has a good ranking representation, and the data-driven deep learning features derived from our novel CNN–RNN model. The CNN–RNN model makes use of shortcut connections to bridge a CNN module and an RNN coarse-to-fine module. The CNN captures the global view while the RNN simulates human’s attention shift, namely, from the whole image (global) to the farthest discerned region (local). The learned relative model can be adapted to predict absolute visibility in limited scenarios. Extensive experiments and comparisons are performed to verify our method. We have built an annotated dataset consisting of about 40000 images with 0.2 million human annotations. The large-scale, annotated visibility data set will be made available to accompany this paper.

References

[1]

J. J. Koenderink, “ Pictorial relief,” Philos. Trans. Roy. Soc. London A, Math., Phys. Eng. Sci., vol. Volume 356, no. Issue 1740, pp. 1071–1086, 1998.

[2]

V. Mnih, N. Heess, and A. Graves, “ Recurrent models of visual attention,” in Proc. NIPS, 2014, pp. 2204–2212.

Digital Library

[3]

F. Liu, C. Shen, G. Lin, and I. Reid, “ Learning depth from single monocular images using deep convolutional neural fields,” IEEE Trans. Pattern Anal. Mach. Intell., vol. Volume 38, no. Issue 10, pp. 2024–2039, 2016.

Digital Library

[4]

N. Graves and S. Newsam, “ Using visibility cameras to estimate atmospheric light extinction,” in Proc. WACV, 2011, pp. 577–584.

Digital Library

[5]

D. Bäumer, S. Versick, and B. Vogel, “ Determination of the visibility using a digital panorama camera,” Atmos. Environ., vol. Volume 42, no. Issue 11, pp. 2593–2602, 2008.

[6]

N. Hautiére, R. Babari, É. Dumont, R. Brémond, and N. Paparoditis, “ Estimating meteorological visibility using cameras: A probabilistic model-driven approach,” in Proc. ACCV, 2011, pp. 243–254.

Digital Library

[7]

J. P. Oakley and B. L. Satherley, “ Improving image quality in poor visibility conditions using a physical model for contrast degradation,” IEEE Trans. Image Process., vol. Volume 7, no. Issue 2, pp. 167–179, 1998.

Digital Library

[8]

P. Carr and R. Hartley, “ Improved single image dehazing using geometry,” in Proc. Digit. Image Comput., Techn. Appl. (DICTA), 2009, pp. 103–110.

Digital Library

[9]

J. Kopf, “ Deep photo: Model-based photograph enhancement and viewing,” ACM Trans. Graph., vol. Volume 27, no. Issue 5, pp. 116-1–116-10, 2008.

Digital Library

[10]

Y. Y. Schechner, S. G. Narasimhan, and S. K. Nayar, “ Instant dehazing of images using polarization,” in Proc. CVPR, 2001, p. pp.325.

[11]

S. Shwartz, E. Namer, and Y. Y. Schechner, “ Blind haze separation,” in Proc. CVPR, 2006, pp. 1984–1991.

Digital Library

[12]

S. G. Narasimhan and S. K. Nayar, “ Vision and the atmosphere,” Int. J. Comput. Vis., vol. Volume 48, no. Issue 3, pp. 233–254, 2002.

Digital Library

[13]

K. He, J. Sun, and X. Tang, “ Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. Volume 33, no. Issue 12, pp. 2341–2353, 2011.

Digital Library

[14]

Q. Yan, L. Xu, and J. Jia, “ Dense scattering layer removal,” in Proc. SIGGRAPH Asia Tech. Briefs, 2013, Art. no. .

Digital Library

[15]

G. Meng, Y. Wang, J. Duan, S. Xiang, and C. Pan, “ Efficient image dehazing with boundary constraint and contextual regularization,” in Proc. ICCV, 2013, pp. 617–624.

Digital Library

[16]

R. T. Tan, “ Visibility in bad weather from a single image,” in Proc. CVPR, 2008, pp. 1–8.

[17]

R. Fattal, “ Single image dehazing,” ACM Trans. Graph., vol. Volume 27, no. Issue 3, p. pp.72, 2008.

Digital Library

[18]

S. G. Narasimhan and S. K. Nayar, “ Shedding light on the weather,” in Proc. CVPR, 2003, p. pp.665.

Digital Library

[19]

S. Metari and F. Deschenes, “ A new convolution kernel for atmospheric point spread function applied to computer vision,” in Proc. ICCV, 2007, pp. 1–8.

[20]

K. Tang, J. Yang, and J. Wang, “ Investigating haze-relevant features in a learning framework for image dehazing,” in Proc. CVPR, 2014, pp. 2995–3002.

Digital Library

[21]

T. Joachims, “ Optimizing search engines using clickthrough data,” in Proc. SIGKDD, 2002, pp. 133–142.

Digital Library

[22]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ ImageNet classification with deep convolutional neural networks,” in Proc. NIPS, 2012, pp. 1097–1105.

Digital Library

[23]

K. He, X. Zhang, S. Ren, and J. Sun, “ Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770–778.

[24]

D. Parikh and K. Grauman, “ Relative attributes,” in Proc. ICCV, 2011, pp. 503–510.

Digital Library

[25]

L. Page, S. Brin, R. Motwani, and T. Winograd, “ The PageRank citation ranking: Bringing order to the Web,” <institution content-type=institution>Stanford InfoLab</institution>, Stanford, CA, USA, Tech. Rep. 1999-66, 1999.

[26]

C. Burges, “ Learning to rank using gradient descent,” in Proc. ICML, 2005, pp. 89–96.

Digital Library

[27]

Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer, “ An efficient boosting algorithm for combining preferences,” J. Mach. Learn. Res., vol. Volume 4, pp. 933–969, 2003.

Digital Library

[28]

C. Lu, D. Lin, J. Jia, and C.-K. Tang, “ Two-class weather classification,” in Proc. CVPR, 2014, pp. 3718–3725.

Digital Library

[29]

A. Oliva and A. Torralba, “ Modeling the shape of the scene: A holistic representation of the spatial envelope,” in Proc. IJCV, 2001, pp. 145–175.

Digital Library

[30]

V. Dang, “ Ranklib,” <institution content-type=institution>Univ. Massachusetts Amherst</institution>, Amherst, MA, USA, 2011. {Online}. Available: https://sourceforge.net/p/lemur/wiki/RankLib/

Cited By

Guo QZhang ZZhou MYue HPu HLuo J(2023)Image Defogging Based on Regional Gradient Constrained PriorACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361783420:3(1-17)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3617834
Liu HHu H(2023)From Global to Local: An Adaptive Environmental Illumination Estimation for Non-uniform ScatteringProceedings of the 5th ACM International Conference on Multimedia in Asia10.1145/3595916.3626416(1-7)Online publication date: 6-Dec-2023
https://dl.acm.org/doi/10.1145/3595916.3626416
Yang WZhao YLi QZhu FSu Y(2023)Multi visual feature fusion based fog visibility estimation for expressway surveillance using deep learning networkExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121151234:COnline publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.121151
Show More Cited By

Index Terms

Relative CNN-RNN: Learning Relative Atmospheric Visibility From Images
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Combining Position-aware CNN and RNN for Relation Extraction
ACAI '19: Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence

Recurrent neural network (RNN) is often used for relation extraction that is based on long range dependency analyses and connections between nodes within the given sentences. However, RNN has a weaker ability in extracting certain important n-grams from ...
CNN-RNN: a large-scale hierarchical image classification framework

Objects are often organized in a semantic hierarchy of categories, where fine-level categories are grouped into coarse-level categories according to their semantic relations. While previous works usually only classify objects into the leaf categories, ...
Video-based emotion recognition using CNN-RNN and C3D hybrid networks
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

In this paper, we present a video-based emotion recognition system submitted to the EmotiW 2016 Challenge. The core module of this system is a hybrid network that combines recurrent neural network (RNN) and 3D convolutional networks (C3D) in a late-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 28, Issue 1

Jan. 2019

523 pages

ISSN:1057-7149

Issue’s Table of Contents

Copyright © 2019.

Publisher

IEEE Press

Publication History

Published: 01 January 2019

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Guo QZhang ZZhou MYue HPu HLuo J(2023)Image Defogging Based on Regional Gradient Constrained PriorACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361783420:3(1-17)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3617834
Liu HHu H(2023)From Global to Local: An Adaptive Environmental Illumination Estimation for Non-uniform ScatteringProceedings of the 5th ACM International Conference on Multimedia in Asia10.1145/3595916.3626416(1-7)Online publication date: 6-Dec-2023
https://dl.acm.org/doi/10.1145/3595916.3626416
Yang WZhao YLi QZhu FSu Y(2023)Multi visual feature fusion based fog visibility estimation for expressway surveillance using deep learning networkExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121151234:COnline publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.121151
Bhasuran BNatarajan J(2023)DisGeReExT: a knowledge discovery system for exploration of disease–gene associations through large-scale literature-wide analysis studyKnowledge and Information Systems10.1007/s10115-023-01862-165:8(3463-3487)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1007/s10115-023-01862-1
Huang YHuang JWu XJia Y(2022)Dynamic Sign Language Recognition Based on CBAM with Autoencoder Time Series Neural NetworkMobile Information Systems10.1155/2022/32477812022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/3247781
Kadota THayashi HBise RTanaka KUchida S(2022)Deep Bayesian Active-Learning-to-Rank for Endoscopic Image DataMedical Image Understanding and Analysis10.1007/978-3-031-12053-4_45(609-622)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-12053-4_45
Song MHan XLiu XLi Q(2021)Visibility estimation via deep label distribution learning in cloud environmentJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-021-00261-710:1Online publication date: 28-Aug-2021
https://dl.acm.org/doi/10.1186/s13677-021-00261-7
Lepetit PBarthes LMallet CViltard N(2020)Learning to compare visibility on webcam imagesProceedings of the 10th International Conference on Climate Informatics10.1145/3429309.3429323(91-97)Online publication date: 22-Sep-2020
https://dl.acm.org/doi/10.1145/3429309.3429323
Han XSong XYao YXu XNie L(2019)Neural Compatibility Modeling With Probabilistic Knowledge DistillationIEEE Transactions on Image Processing10.1109/TIP.2019.293674229(871-882)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1109/TIP.2019.2936742

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents