research-article

A deep learning based image enhancement approach for autonomous driving at night

Authors:

Keqiang LiAuthors Info & Claims

Volume 213, Issue C

https://doi.org/10.1016/j.knosys.2020.106617

Published: 15 February 2021 Publication History

Abstract

Images of road scenes in low-light situations are lack of details which could increase crash risk of connected autonomous vehicles (CAVs). Therefore, an effective and efficient image enhancement model for low-light images is necessary for safe CAV driving. Though some efforts have been made, image enhancement still cannot be well addressed especially in extremely low light situations (e.g., in rural areas at night without street light). To address this problem, we developed a light enhancement net (LE-net) based on the convolutional neural network. Firstly, we proposed a generation pipeline to transform daytime images to low-light images, and then used them to construct image pairs for model development. Our proposed LE-net was then trained and validated on the generated low-light images. Finally, we examined the effectiveness of our LE-net in real night situations at various low-light levels. Results showed that our LE-net was superior to the compared models, both qualitatively and quantitatively.

References

[1]

Sengupta S., Basak S., Saikia P., Paul S., Tsalavoutis V., Atiah F., Ravi V., Peters A., A review of deep learning with special emphasis on architectures, applications and recent trends, Knowl.-Based Syst. (2020).

[2]

Liu T., Zhao Y., Wei Y., Zhao Y., Wei S., Concealed object detection for activate millimeter wave image, IEEE Trans. Ind. Electron. 66 (2019) 9909–9917.

[3]

Li G., Li S.E., Zou R., Liao Y., Cheng B., Detection of road traffic participants using cost-effective arrayed ultrasonic sensors in low-speed traffic situations, Mech. Syst. Signal Process. 132 (2019) 535–545.

[4]

Gao H., Cheng B., Wang J., Li K., Zhao J., Li D., Object classification using CNN-based fusion of vision and LIDAR in autonomous vehicle environment, IEEE Trans. Ind. Inf. 14 (2018) 4224–4231.

[5]

Ruiz-Sarmiento J.-R., Galindo C., Monroy J., Moreno F.-A., Gonzalez-Jimenez J., Ontology-based conditional random fields for object recognition, Knowl.-Based Syst. 168 (2019) 100–108.

[6]

X. Sun, L. Zheng, Dissecting person re-identification from the viewpoint of viewpoint, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 608-617.

[7]

Ding Y., Fan H., Xu M., Yang Y., Adaptive exploration for unsupervised person re-identification, ACM Trans. Multimed. Comput. Commun. Appl. 16 (1) (2020) 1–19.

Digital Library

[8]

Zhang Y., Ye M., Gan Y., Zhang W., Knowledge based domain adaptation for semantic segmentation, Knowl.-Based Syst. 193 (2020).

[9]

Xu H., Huang C., Wang D., Enhancing semantic image retrieval with limited labeled examples via deep learning, Knowl.-Based Syst. 163 (2019) 252–266.

[10]

Zhang Y., Ye M., Gan Y., Zhang W., Knowledge based domain adaptation for semantic segmentation, Knowl.-Based Syst. 193 (2020).

[11]

He W., Li Z., Chen C.L.P., A survey of human-centered intelligent robots: issues and challenges, IEEE/CAA J. Autom. Sinica 4 (2017) 602–609.

[12]

Li J., Luo G., Cheng N., Yuan Q., Wu Z., Gao S., Liu Z., An end-to-end load balancer based on deep learning for vehicular network traffic control, IEEE Internet Things J. 6 (2019) 953–966.

[13]

Chen B.-H., Huang S.-C., Kuo S.-Y., Error-optimized sparse representation for single image rain removal, IEEE Trans. Ind. Electron. 64 (2017) 6573–6581.

[14]

NHTSA, Traffic Safety Facts 2017 (DOT HS 812 806), National Highway Traffic Safety Administration, U.S. Department of Transportation, Washington, DC, U.S., 2019.

[15]

Li G., Wang Y., Zhu F., Sui X., Wang N., Qu X., Green P., Drivers’ visual scanning behavior at signalized and unsignalized intersections: A naturalistic driving study in China, J. Saf. Res. 71 (2019) 219–229.

[16]

Liu S., Feng Y., Zhang S., Song H., Chen S., L 0 Sparse regularization-based image blind deblurring approach for solid waste image restoration, IEEE Trans. Ind. Electron. 66 (2019) 9837–9845.

[17]

Chen S.-D., Ramli A.R., Contrast enhancement using recursive mean-separate histogram equalization for scalable brightness preservation, IEEE Trans. Consum. Electron. 49 (2003) 1301–1309.

Digital Library

[18]

Zheng L., Shi H., Gu M., Infrared traffic image enhancement algorithm based on dark channel prior and gamma correction, Modern Phys. Lett. B 31 (2017).

[19]

Chang Y., Jung C., Ke P., Song H., Hwang J., Automatic contrast-limited adaptive histogram equalization with dual gamma correction, IEEE Access 6 (2018) 11782–11792.

[20]

Pan J., Sun D., Pfister H., Yang M.-H., Deblurring images via dark channel prior, IEEE Trans. Pattern Anal. Mach. Intell. 40 (2018) 2315–2328.

[21]

Li M., Liu J., Yang W., Sun X., Guo Z., Structure-revealing low-light image enhancement via robust retinex model, IEEE Trans. Image Process. 27 (2018) 2828–2841.

[22]

Gao Y., Hu H.-M., Li B., Guo Q., Naturalness preserved nonuniform illumination estimation for image enhancement based on retinex, IEEE Trans. Multimed. 20 (2018) 335–344.

[23]

Shen J., Li G., Yan W., Tao W., Xu G., Diao D., Green P., Nighttime driving safety improvement via image enhancement for driver face detection, IEEE Access 6 (2018) 45625–45634.

[24]

Guo X., Li Y., Ling H., LIME: Low-light image enhancement via illumination map estimation, IEEE Trans. Image Process. 26 (2017) 982–993.

Digital Library

[25]

Lore K.G., Akintayo A., Sarkar S., Llnet: A deep autoencoder approach to natural low-light image enhancement, Pattern Recognit. 61 (2017) 650–662.

Digital Library

[26]

Shen L., Yue Z., Feng F., Chen Q., Liu S., Ma J., MSR-Net: low-light image enhancement using deep convolutional network, 2017, arXiv:1711.02488.

[27]

Ren W., Liu S., Ma L., Xu Q., Xu X., Cao X., Du J., Yang M.-H., Low-light image enhancement via a deep hybrid network, IEEE Trans. Image Process. 28 (2019) 4364–4375.

[28]

Guo Y., Ke X., Ma J., Zhang J., A pipeline neural network for low-light image enhancement, IEEE Access 7 (2019) 13737–13744.

[29]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, MobileNetV2: inverted residuals and linear bottlenecks, in: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510–4520.

[30]

Y. Hou, Z. Ma, C. Liu, C.C. Loy, Learning lightweight lane detection CNNs by self attention distillation, in: Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1013–1021.

[31]

T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936–944.

[32]

Jobson D.J., Rahman Z., Woodell G.A., A multiscale retinex for bridging the gap between color images and the human observation of scenes, IEEE Trans. Image Process. 6 (1997) 965–976.

Digital Library

[33]

Jobson D.J., Rahman Z., Woodell G.A., Properties and performance of a center/surround retinex, IEEE Trans. Image Process. 6 (1997) 451–462.

Digital Library

[34]

Yang X., Jian L., Wu W., Liu K., Yan B., Zhou Z., Peng J., Implementing real-time RCF-retinex image enhancement method using CUDA, J. Real-Time Image Process. 16 (1) (2019) 115–125.

[35]

Ma L., Jin D., Liu R., Fan X., Luo Z., Joint over and under exposures correction by aggregated retinex propagation for image enhancement, IEEE Signal Process. Lett. 27 (2020) 1210–1214.

[36]

Li G., Yang Y., Qu X., Deep learning approaches on pedestrian detection in hazy weather, IEEE Trans. Ind. Electron. 67 (10) (2020) 8889–8899.

[37]

Q. Chen, J. Xu, V. Koltun, Fast image processing with fully-convolutional networks, in: Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2497–2506.

[38]

Sakurai R., Yamane S., Lee J.-H., Restoring aspect ratio distortion of natural images with convolutional neural network, IEEE Trans. Ind. Inf. 15 (2019) 563–571.

[39]

R. Wang, Q. Zhang, C.-W. Fu, X. Shen, W.-S. Zheng, J. Jia, Underexposed photo enhancement using deep illumination estimation, in: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6849–6857.

[40]

C. Chen, Q. Chen, J. Xu, V. Koltun, Learning to see in the dark, in: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3291–3300.

[41]

Atoum Y., Ye M., Ren L., Tai Y., Liu X., Color-wise attention network for low-light image enhancement, 2019, arXiv:1911.08681.

[42]

C. Chen, Q. Chen, M.N. Do, V. Koltun, Seeing motion in the dark, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 3185-3194.

[43]

H. Jiang, Y. Zheng, Learning to see moving objects in the dark, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 7324-7333.

[44]

W. Wang, X. Chen, C. Yang, X. Li, X. Hu, T. Yue, Enhancing low light videos by exploring high sensitivity camera noise, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019, pp. 4111–4119.

[45]

Wang J., Hu Y., An improved enhancement algorithm based on CNN applicable for weak contrast images, IEEE Access 8 (2020) 8459–8476.

[46]

K. Wei, Y. Fu, J. Yang, H. Huang, A physics-based noise formation model for extreme low-light raw denoising, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2758–2767.

[47]

Y.-S. Chen, Y.-C. Wang, M.-H. Kao, Y.-Y. Chuang, Deep photo enhancer: unpaired learning for image enhancement from photographs with GANs, in: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6306–6314.

[48]

A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, L. Van Gool, WESPE: Weakly supervised photo enhancer for digital cameras, in: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 804–813.

[49]

V. Bychkovsky, S. Paris, E. Chan, F. Durand, Learning photographic global tonal adjustment with a database of input/output image pairs, in: Proceedings of the 2011 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 97–104.

[50]

Yu F., Chen H., Wang X., Xian W., Chen Y., Liu F., Madhavan V., Darrell T., BDD100K: A Diverse driving dataset for heterogeneous multitask learning, 2020, arXiv:1805.04687.

[51]

Zhang H., Cisse M., Dauphin Y.N., Lopez-Paz D., Mixup: Beyond empirical risk minimization, 2018, arXiv:1710.09412.

[52]

A. Bevilacqua, P. Azzari, A high performance exact histogram specification algorithm, in: Proceedings of the 14th International Conference on Image Analysis and Processing (ICIAP 2007), 2007, pp. 623–628.

[53]

Howard A.G., Zhu M., Chen B., Kalenichenko D., Wang W., Weyand T., Andreetto M., Adam H., MobileNets: EFficient convolutional neural networks for mobile vision applications, 2017, arXiv:1704.04861.

[54]

F. Chollet, Xception: deep learning with depthwise separable convolutions, in: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1800–1807.

[55]

Ronneberger O., Fischer P., Brox T., U-Net: COnvolutional networks for biomedical image segmentation, 2015, arXiv:1505.04597.

[56]

Y. Wu, K. He, Group normalization, in: Proceedings of the 2018 European Conference on Computer Vision (ECCV), 2018, pp. 3–19.

[57]

Huynh-Thu Q., Ghanbari M., Scope of validity of PSNR in image/video quality assessment, Electron. Lett. 44 (2008) 800–801.

[58]

D. Sadykova, A.P. James, Quality assessment metrics for edge detection and edge-aware filtering: A tutorial review, in: Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2017, pp. 2366–2369.

[59]

Reza A.M., Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement, J. VLSI Signal Process. Syst. Signal Image Video Technol. 38 (2004) 35–44.

[60]

Petro A.B., Sbert C., Morel J.-M., Multiscale retinex, Image Process. Line 4 (2014) 71–88.

[61]

Z. Ying, G. Li, Y. Ren, R. Wang, W. Wang, A new image contrast enhancement algorithm using exposure fusion framework, in: Proceedings of the 2017 Computer Analysis of Images and Patterns (CAIP), 2017, pp. 36–46.

[62]

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the 2018 European Conference on Computer Vision (ECCV), 2018, pp. 801–818.

[63]

Bosse S., Maniry D., Muller K.-R., Wiegand T., Samek W., Deep neural networks for no-reference and full-reference image quality assessment, IEEE Trans. Image Process. 27 (2018) 206–219.

[64]

Panetta K., Gao C., Agaian S., No reference color image contrast and quality measures, IEEE Trans. Consum. Electron. 59 (2013) 643–651.

[65]

Yan J., Li J., Fu X., No-reference quality assessment of contrast-distorted images using contrast enhancement, 2019, arXiv:1904.08879.

[66]

Braun M., Krebs S., Flohr F., Gavrila D.M., The eurocity persons dataset: A novel benchmark for object detection, IEEE Trans. Pattern Anal. Mach. Intell. 41 (2019) 1844–1861.

[67]

Fan H., Zheng L., Yan C., Yang Y., Unsupervised person re-identification: Clustering and fine-tuning, ACM Trans. Multimed. Comput. Commun. Appl. 14 (4) (2018) 1–18.

Digital Library

[68]

Y. Huang, Z.J. Zha, X. Fu, R. Hong, L. Li, Real-world person re-identification via degradation invariance learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 14084–14094.

Cited By

Wang YLu YQiu Y(2025)Gated image-adaptive network for driving-scene object detection under nighttime conditionsMultimedia Systems10.1007/s00530-024-01589-131:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s00530-024-01589-1
Wang JChen YJi XDong ZGao MHe Z(2024)SpikeTOD: A Biologically Interpretable Spike-Driven Object Detection in Challenging Traffic ScenariosIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.346803825:12(21297-21314)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1109/TITS.2024.3468038
Xu HLiu XZhang HWu XZuo W(2024)Degraded Structure and Hue Guided Auxiliary Learning for low-light image enhancementKnowledge-Based Systems10.1016/j.knosys.2024.111779295:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111779
Show More Cited By

Index Terms

A deep learning based image enhancement approach for autonomous driving at night
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

An Image Enhancement Approach Using Retinex and YIQ
ITCS '09: Proceedings of the 2009 International Conference on Information Technology and Computer Science - Volume 01

On a basis of Retinex and YIQ color space, a new approach of color image enhancement is presented in this paper. The original image in the RGB color space is converted to the YIQ color space, and the multi-scale Retinex (MSR) algorithm is introduced to ...
Autonomous intersection driving with neuro-evolution
GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Recently there has been increasing research attention focused on producing adaptive control systems for autonomous vehicles. To accommodate such autonomous vehicles there have been proposals that current road and highway infrastructure undergo ...
Low Illumination Color Image Enhancement Based on Improved Retinex Theory
EITCE '20: Proceedings of the 2020 4th International Conference on Electronic Information Technology and Computer Engineering

In view of the defects of the classical Retinex image enhancement algorithm, such as poor texture detail retention, halo and over enhancement, and abrupt tone, this paper proposes a low illumination color image enhancement algorithm based on Gabor filter ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 213, Issue C

Feb 2021

732 pages

ISSN:0950-7051

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 15 February 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLu YQiu Y(2025)Gated image-adaptive network for driving-scene object detection under nighttime conditionsMultimedia Systems10.1007/s00530-024-01589-131:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s00530-024-01589-1
Wang JChen YJi XDong ZGao MHe Z(2024)SpikeTOD: A Biologically Interpretable Spike-Driven Object Detection in Challenging Traffic ScenariosIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.346803825:12(21297-21314)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1109/TITS.2024.3468038
Xu HLiu XZhang HWu XZuo W(2024)Degraded Structure and Hue Guided Auxiliary Learning for low-light image enhancementKnowledge-Based Systems10.1016/j.knosys.2024.111779295:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111779
Xiang KDong XWang WDong X(2024)Edge-guided oceanic scene element detectionKnowledge-Based Systems10.1016/j.knosys.2023.111322284:COnline publication date: 25-Jan-2024
https://dl.acm.org/doi/10.1016/j.knosys.2023.111322
Zhang HYang KLi YChan L(2024)Self-supervised network for low-light traffic image enhancement based on deep noise and artifacts removalComputer Vision and Image Understanding10.1016/j.cviu.2024.104063246:COnline publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.cviu.2024.104063
Naseer MPrabakaran BHasan OShafique M(2024)UnbiasedNets: a dataset diversification framework for robustness bias alleviation in neural networksMachine Language10.1007/s10994-023-06314-z113:5(2499-2526)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s10994-023-06314-z
Archana RJeevaraj P(2024)Deep learning models for digital image processing: a reviewArtificial Intelligence Review10.1007/s10462-023-10631-z57:1Online publication date: 7-Jan-2024
https://dl.acm.org/doi/10.1007/s10462-023-10631-z
Wan FXu BPan WLiu H(2024)PSC diffusion: patch-based simplified conditional diffusion model for low-light image enhancementMultimedia Systems10.1007/s00530-024-01391-z30:4Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1007/s00530-024-01391-z
Mi ALuo WHuo Z(2024)Dual-band low-light image enhancementMultimedia Systems10.1007/s00530-024-01298-930:2Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1007/s00530-024-01298-9
Rivera-Aguilar BCuevas EPérez MCamarena ORodríguez A(2024)A new histogram equalization technique for contrast enhancement of grayscale images using the differential evolution algorithmNeural Computing and Applications10.1007/s00521-024-09739-236:20(12029-12045)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00521-024-09739-2
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents