article

A new thermal infrared and visible spectrum images-based pedestrian detection system

Authors:

Redouan Lahmyed,

Mohamed El Ansari,

Ayoub EllahyaniAuthors Info & Claims

Multimedia Tools and Applications, Volume 78, Issue 12

Pages 15861 - 15885

https://doi.org/10.1007/s11042-018-6974-5

Published: 01 June 2019 Publication History

Abstract

In this paper, we propose a hybrid system for pedestrian detection, in which both thermal and visible images of the same scene are used. The proposed method is achieved in two basic steps: (1) Hypotheses generation (HG) where the locations of possible pedestrians in an image are determined and (2) hypotheses verification (HV), where tests are done to check the presence of pedestrians in the generated hypotheses. HG step segments the thermal image using a modified version of OTSU thresholding technique. The segmentation results are mapped into the corresponding visible image to obtain the regions of interests (possible pedestrians). A post-processing is done on the resulting regions of interests to keep only significant ones. HV is performed using random forest as classifier and a color-based histogram of oriented gradients (HOG) together with the histograms of oriented optical flow (HOOF) as features. The proposed approach has been tested on OSU Color-Thermal, INO Video Analytics and LITIV data sets and the results justify its effectiveness.

References

[1]

Akhloufi MA, Porcher C, Bendada A (2013) Fusion of thermal infrared and visible spectrum images for robust pedestrian tracking. In: Proceedings of SPIE, volume, 9076

[2]

Ansari M El, Lahmyed R, Tremeau A (2018) A hybrid pedestrian detection system based on visible images and lidar data. In: Proceedings of the 13th international joint conference on computer vision, imaging and computer graphics theory and applications - volume 5: VISAPP, pages 325---334. INSTICC, SciTePress

[3]

Bay H, Ess A, Tuytelaars T, Gool LV (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346---359

Digital Library

[4]

Breiman L (2001) Random forests. Mach Learn 45(1):5---32

Digital Library

[5]

Castillo JC, Serrano-Cuerda J, Sokolova MV, Costa A, Novais P (2012) Multispectrum video for proactive response in intelligent environments. In: 2012 8th international conference on intelligent environments (IE). IEEE, pp 178---185

Digital Library

[6]

Charfi S, Ansari M El (2018) Computer-aided diagnosis system for colon abnormalities detection in wireless capsule endoscopy images. Multimedia Tools and Applications 77(3):4047---4064

Digital Library

[7]

Choi S, Kim E, Lee K, Oh S (2017) Real-time nonparametric reactive navigation of mobile robots in dynamic environments. Robot Auton Syst 91:11---24

Digital Library

[8]

Christianini N, Shawe-Taylor J (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, United Kingdom

Digital Library

[9]

Cuntoor N, Kale A, Chellappa R (2003) Combining multiple evidences for gait recognition. In: 2003 international conference on multimedia and Expo, 2003. ICME'03. Proceedings, vol 3. IEEE, pp III---113

Digital Library

[10]

Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, volume 1. IEEE, pp 886---893

Digital Library

[11]

Davis JW, Sharma V (2007) Background-subtraction using contour-based fusion of thermal and visible imagery. Comput Vis Image Underst 106(2):162---182

Digital Library

[12]

Dollár P, Appel R, Belongie S, Perona P (2014) Fast feature pyramids for object detection. IEEE Trans Pattern Anal Mach Intell 36(8):1532---1545

Digital Library

[13]

Elguebaly T, Bouguila N (2011) A nonparametric bayesian approach for enhanced pedestrian detection and foreground segmentation. In: 2011 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 21---26

[14]

Elguebaly T, Bouguila N (2013) Finite asymmetric generalized gaussian mixture models learning for infrared object detection. Comput Vis Image Underst 117(12):1659---1671

Digital Library

[15]

Ellahyani A, Ansari M El, Jaafari I El (2016) Traffic sign detection and recognition based on random forests. Appl Soft Comput 46:805---815

Digital Library

[16]

Fendri E, Boukhriss RR, Hammami M (2017) Fusion of thermal infrared and visible spectra for robust moving object detection. Pattern Anal Applic 20(4):907---926

Digital Library

[17]

Foster JP, Nixon MS, Prügel-Bennett A (2003) Automatic gait recognition using area-based metrics. Pattern Recogn Lett 24(14):2489---2497

Digital Library

[18]

Gascuena JM, Serrano-Cuerda J, Castillo JC, Fernández-Caballero A, López MT (2014) A multi-agent system for infrared and color video fusion. In: Trends in practical applications of heterogeneous multi-agent systems. The PAAMS collection. Springer, pp 131---138

[19]

Gavrila DM, Munder S (2007) Multi-cue pedestrian detection and tracking from a moving vehicle. Int J Comput Vis 73(1):41---59

Digital Library

[20]

Ge J, Luo Y, Tei G (2009) Real-time pedestrian detection and tracking at nighttime for driver-assistance systems. IEEE Trans Intell Transp Syst 10(2):283---298

Digital Library

[21]

Guo L, Ge P-S, Zhang M-H, Li L-H, Zhao Y-B (2012) Pedestrian detection for intelligent transportation systems combining adaboost algorithm and support vector machine. Expert Systems with Applications 39(4):4274---4286

Digital Library

[22]

Herrmann C, Müller T, Willersinn D, Beyerer J (2016) Real-time person detection in low-resolution thermal infrared imagery with mser and cnns. In: SPIE security+ defence, pp 99870I---99870I. International society for optics and photonics

[23]

Huang D-Y, Wang C-H (2009) Optimal multi-level thresholding using a two-stage otsu optimization approach. Pattern Recogn Lett 30(3):275---284

Digital Library

[24]

Hwang S, Park J, Kim N, Choi Y, Kweon IS (2015) Multispectral pedestrian detection benchmark dataset and baseline. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1037---1045

[25]

Ino, "ino video analytics dataset." https://www.ino.ca/en/video-analytics-dataset/. Accessed 6 Sept 2017

[26]

John V, Mita S, Liu Z, Qi B (2015) Pedestrian detection in thermal images using adaptive fuzzy c-means clustering and convolutional neural networks. In: 2015 14th IAPR international conference on machine vision applications (MVA). IEEE, pp 246---249

[27]

Jungling K, Arens M (2009) Feature based person detection beyond the visible spectrum. In: IEEE computer society conference on computer vision and pattern recognition workshops, 2009. CVPR Workshops 2009. IEEE, pp 30---37

[28]

Källhammer J-E, Smith K, Matsangas P (2016) Modeling ratings of in-vehicle alerts to pedestrian by leveraging field operational tests data in a controlled laboratory study. Transportation Research Part F: Traffic Psychology and Behaviour

[29]

Kassani PH, Teoh ABJ (2016) A new sparse model for traffic sign classification using soft histogram of oriented gradients. Appl Soft Comput

Digital Library

[30]

Kim K, Chalidabhongse TH, Harwood D, Davis L (2005) Real-time foreground---background segmentation using codebook model. Real-Time Image 11(3):172---185

Digital Library

[31]

Lahmyed R, Ansari M El (2016) Multisensors-based pedestrian detection system. In: 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). IEEE, pp 1---4

[32]

Lee L, Dalley G, Tieu K (2003) Learning pedestrian models for silhouette refinement. In: ICCV, vol 1, pp 663---670

Digital Library

[33]

Level Otsu N (1979) A threshold selection method from gray-level histogram. IEEE Trans Syst Man Cybern B Cybern 9(1):62---66

[34]

Li H, Zhu J, Tao D (2018) Asymmetric projection and dictionary learning with listwise and identity consistency constraints for person re-identification. IEEE Access 6:37977---37990

[35]

Li Jianfu, Gong Weiguo, Li W, Liu X (2010) Robust pedestrian detection in thermal infrared imagery using the wavelet transform. Infrared Phys Technol 53(4):267---273

[36]

Li Z, Bo Wu, Nevatia S (2007) Pedestrian detection in infrared images based on local shape features. In: 2007 IEEE conference on computer vision and pattern recognition CVPR'07. IEEE, pp 1---8

[37]

Liang C-W, Juang C-F (2015) Moving object classification using local shape and hog features in wavelet-transformed space with hierarchical svm classifiers. Appl Soft Comput 28:483---497

Digital Library

[38]

Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91---110

Digital Library

[39]

Lucas BD, Kanade T, et al (1981) An iterative image registration technique with an application to stereo vision

Digital Library

[40]

Morales Y, Miyashita T, Hagita N (2017) Social robotic wheelchair centered on passenger and pedestrian comfort. Robot Auton Syst 87:355---362

[41]

Nanda H, Davis L (2002) Probabilistic template based pedestrian detection in infrared videos. In: Intelligent vehicle symposium 2002. IEEE, volume 1, pp 15---20

[42]

Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51---59

[43]

Ouloul IM, Moutakki Z, Afdel K, Amghar A (2018) Improvement of age estimation using an efficient wrinkles descriptor. Multimedia Tools and Applications, pp 1---35

Digital Library

[44]

Perš J, Kristan M, Perše M, Kovaă?iă? S (2007) Motion based human identification using histograms of optical flow. na

[45]

Perš J, Sulić V, Kristan M, Perše M, Polanec K, Kovaă?iă? S (2010) Histograms of optical flow for efficient representation of body motion. Pattern Recog Lett 31(11):1369---1376

Digital Library

[46]

Premebida C, Ludwig O, Nunes U (2009) Lidar and vision-based pedestrian detection system. J Field Rob 26(9):696---711

Digital Library

[47]

Qingbo J, Enze Z, Xinqi Y, Yu X, Yun L (2016) Face recognition method based on hog and dmma from single training sample. Multimedia Tools and Applications, 75(21):13163---13177

Digital Library

[48]

Radman A, Suandi SA (2018) Robust face pseudo-sketch synthesis and recognition using morphological-arithmetic operations and hog-pca. Multimedia Tools and Applications. pp 1---22

Digital Library

[49]

San-Biagio M, Crocco M, Cristani M (2012) Recursive segmentation based on higher order statistics in thermal imaging pedestrian detection. In: 2012 5th international symposium on communications control and signal processing (ISCCSP). IEEE, pp 1---4

[50]

Serrano-Cuerda J (2014) Robust human detection through fusion of color and infrared video. ELCVIA: Electronic Letters On Computer Vision And Image Analysis 13(2):0017---18

[51]

Souaidi M, Abdelouahad AA, Ansari M El (2017) A fully automated ulcer detection system for wireless capsule endoscopy images. In: 2017 International Conference on Advanced Technologies for Signal and Image Processing (ATSIP). IEEE, pP 1---6

[52]

Souaidi M, Abdelouahed AA, Ansari M El (2018) Multi-scale completed local binary patterns for ulcer detection in wireless capsule endoscopy images. Multimedia Tools and Applications, pp 1---18

Digital Library

[53]

Souaidi M, Charfi S, Abdelouahad AA, Ansari M El (2018) New features for wireless capsule endoscopy polyp detection. In: 2018 international conference on intelligent systems and computer vision (ISCV). IEEE, pp 1---6

[54]

Sun Hao, Wang Cheng, Wang B, El-Sheimy N (2011) Pyramid binary pattern features for real-time pedestrian detection from infrared videos. Neurocomputing 74 (5):797---804

Digital Library

[55]

Tao D, Guo Y, Li Y, Gao X (2018) Tensor rank preserving discriminant analysis for facial recognition. IEEE Trans on Image Process 27(1):325---334

[56]

Torabi A, Massé G, Bilodeau G-A (2012) An iterative integrated framework for thermal---visible image registration, sensor fusion, and people tracking for video surveillance applications. Comput Vis Image Underst 116(2):210---221

Digital Library

[57]

Usher JM, McCool R, Strawderman L, Carruth DW, Bethel CL, May DC (2017) Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul Model Pract Theory 75:96---112

[58]

Vapnik VN, Vapnik V (1998) Statistical learning theory, volume 1. Wiley, New York

[59]

Wagner J, Fischer V, Herman M, networks SB (2016) Multispectral pedestrian detection using deep fusion convolutional neural. In: 24th European symposium on artificial neural networks computational intelligence and machine learning (ESANN), pp 509---514

[60]

Wang Y, Wang Z, Tao D, Zhuo S, Xu X, Pu S, Song M (2017) Allfocus: patch-based video out-of-focus blur reconstruction. IEEE Trans Circuits Syst Video Technol 27(9):1895---1908

[61]

Yang T, Fu D, Pan S (2017) Pedestrian tracking for infrared image sequence based on trajectory manifold of spatio-temporal slice. Multimedia Tools and Applications 76(8):11021---11035

Digital Library

[62]

Zhang J, Li F-W, Nie W-Z, Li W-H, Su Y-T (2016) Visual attribute detction for pedestrian detection. Multimedia Tools and Applications, pp 1---18

[63]

Zin TT, Takahashi H, Hama H, Toriu T (2011) Fusion of infrared and visible images for robust person detection. INTECH Open Access Publisher

Cited By

Xie QCheng TDai ZTran VTrigoni NMarkham A(2024)Illumination-Aware Hallucination-Based Domain Adaptation for Thermal Pedestrian DetectionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.330716725:1(315-326)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TITS.2023.3307167
Xue TZhang ZMa WLi YYang AJi T(2022)Nighttime Pedestrian and Vehicle Detection Based on a Fast Saliency and Multifeature Fusion Algorithm for Infrared ImagesIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.319308623:9(16741-16751)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1109/TITS.2022.3193086
Lahmyed REl Ansari MKerkaou Z(2022)A novel visible spectrum images-based pedestrian detection and tracking system for surveillance in non-controlled environmentsMultimedia Tools and Applications10.1007/s11042-022-13026-481:27(39275-39309)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s11042-022-13026-4
Show More Cited By

A new thermal infrared and visible spectrum images-based pedestrian detection system
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning

Recommendations

A novel visible spectrum images-based pedestrian detection and tracking system for surveillance in non-controlled environments
Abstract
For many vision applications, robust detection and tracking of pedestrians in image sequences are essential. In this paper, a hybrid system for pedestrian detection and tracking is presented. The proposed method is achieved in four major stages. ...
SCHOG Feature for Pedestrian Detection
ICPRAM 2014: Proceedings of the 3rd International Conference on Pattern Recognition Applications and Methods

Co-occurrence Histograms of Oriented Gradients(CoHOG) has succeeded in describing the detailed shape of the object by using a co-occurrence of features. However, unlike HOG, it does not consider the difference of gradient magnitude between the ...
G2P: a new descriptor for pedestrian detection
Abstract
Pedestrian detection plays an important role in many applications. Since its birth 13 years ago, Histogram Of Gradient (HOG) descriptor has become a popular descriptor for pedestrian detection. Besides its original instantiation, the HOG also ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 78, Issue 12

June 2019

1578 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © Copyright © 2019 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie QCheng TDai ZTran VTrigoni NMarkham A(2024)Illumination-Aware Hallucination-Based Domain Adaptation for Thermal Pedestrian DetectionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.330716725:1(315-326)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TITS.2023.3307167
Xue TZhang ZMa WLi YYang AJi T(2022)Nighttime Pedestrian and Vehicle Detection Based on a Fast Saliency and Multifeature Fusion Algorithm for Infrared ImagesIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.319308623:9(16741-16751)Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1109/TITS.2022.3193086
Lahmyed REl Ansari MKerkaou Z(2022)A novel visible spectrum images-based pedestrian detection and tracking system for surveillance in non-controlled environmentsMultimedia Tools and Applications10.1007/s11042-022-13026-481:27(39275-39309)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s11042-022-13026-4
Lahmyed RAnsari MKerkaou Z(2022)Automatic road sign detection and recognition based on neural networkSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-021-06726-w26:4(1743-1764)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1007/s00500-021-06726-w
Zahid MKhan MAzam FSharif MKadry SMohanty J(2021)RETRACTED ARTICLE: Pedestrian identification using motion-controlled deep neural network in real-time visual surveillanceSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-021-05701-927:1(453-469)Online publication date: 12-Mar-2021
https://dl.acm.org/doi/10.1007/s00500-021-05701-9
Kerkaou ZEl Ansari M(2020)Support vector machines based stereo matching method for advanced driver assistance systemsMultimedia Tools and Applications10.1007/s11042-020-09260-379:37-38(27039-27055)Online publication date: 21-Jul-2020
https://dl.acm.org/doi/10.1007/s11042-020-09260-3

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents