Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Influence of the Distribution of Tag IDs on RFID Memoryless Anti-Collision Protocols
Next Article in Special Issue
Dual Quaternions as Constraints in 4D-DPM Models for Pose Estimation
Previous Article in Journal
A Vision-Based Wayfinding System for Visually Impaired People Using Situation Awareness and Activity-Based Instructions
Previous Article in Special Issue
Monocular Stereo Measurement Using High-Speed Catadioptric Tracking
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 17
Issue 8
10.3390/s17081890
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Detecting Traversable Area and Water Hazards for the Visually Impaired with a pRGB-D Sensor

by
Kailun Yang
,
Kaiwei Wang
*,
Ruiqi Cheng
,
Weijian Hu
,
Xiao Huang
and
Jian Bai
College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China
*
Author to whom correspondence should be addressed.
Sensors 2017, 17(8), 1890; https://doi.org/10.3390/s17081890
Submission received: 9 July 2017 / Revised: 6 August 2017 / Accepted: 11 August 2017 / Published: 17 August 2017
(This article belongs to the Special Issue Imaging Depth Sensors—Sensors, Algorithms and Applications)
Browse Figures
Graphical abstract
"> Figure 1
<p>The flow chart of the approach.</p> "> Figure 2
<p>The light reflection and refraction for water hazards with particles and ground bottom.</p> "> Figure 3
<p>Reflection and refraction energy components from water as function of degree and direction of polarization. (<b>a</b>) Unpolarized light; (<b>b</b>) Light is three quarters polarized in perpendicular direction with the water plane; (<b>c</b>) Light is a quarter polarized in perpendicular direction with the water plane.</p> "> Figure 4
<p>Traversable area detection in different scenarios.</p> "> Figure 5
<p>Flow chart of the water hazard detection.</p> "> Figure 6
<p>Water hazard detection with traversable area detection. (<b>a</b>,<b>d</b>) Left color images; (<b>b</b>,<b>e</b>) Right color images; (<b>c</b>,<b>f</b>) Detection results with the pRGB-D sensor.</p> "> Figure 7
<p>Farthest traversable lines with or without water hazard detection. (<b>a</b>,<b>d</b>,<b>g</b>) Traversable area and water hazard detection results; (<b>b</b>,<b>e</b>,<b>h</b>) Farthest traversable lines correspond to the traversable area detection without water hazard detection; (<b>c</b>,<b>f</b>,<b>i</b>) Farthest traversable lines obtained with both traversable area detection and water hazard detection.</p> "> Figure 8
<p>Stereo sound feedback of the assisting system. Sounds of five directions of traversable area with or without water hazard detection are presented by five musical instruments in the virtual 3D space, including trumpet, piano, gong, violin and xylophone.</p> "> Figure 9
<p>The relationship between the average traversable distance and instrument loudness.</p> "> Figure 10
<p>The relationship between the proportion of overlapping areas and instrument pitch.</p> "> Figure 11
<p>Results of traversable area and water hazard detection in indoor and outdoor environments. (<b>a</b>,<b>e</b>,<b>i</b>,<b>m</b>) Left color images; (<b>b</b>,<b>f</b>,<b>j</b>,<b>n</b>) Depth images; (<b>c</b>,<b>g</b>,<b>k</b>,<b>o</b>) Traversable area and water hazard detection results denoted respectively by the green and magenta masks on the left color images; (<b>d</b>,<b>h</b>,<b>l</b>,<b>p</b>) Farthest traversable lines as stereo sound feedback interface.</p> "> Figure 12
<p>The detection results of different approaches. (<b>a</b>–<b>c</b>) Results of the proposed approach; (<b>d</b>–<b>f</b>) Results of traversable area detection based on RANSAC plus filtering techniques; (<b>g</b>–<b>i</b>) Results of water hazard detection based on sky light effect modelling.</p> "> Figure 13
<p>Detection limitations. (<b>a</b>–<b>c</b>) Detection with defective depth information; (<b>d</b>–<b>f</b>) Small obstacles would be detected as traversable area.</p> "> Figure 14
<p>(<b>a</b>,<b>b</b>) The voluntary participants wearing the assisting prototype including a pRGB-D sensor, a laptop in the backpack and a set of bone-conduction headphone; (<b>c</b>) The route around a building and an empty field in the real-world environment while the red labels represent the locations during navigation.</p> ">
Versions Notes

Abstract

:
The use of RGB-Depth (RGB-D) sensors for assisting visually impaired people (VIP) has been widely reported as they offer portability, function-diversity and cost-effectiveness. However, polarization cues to assist traversability awareness without precautions against stepping into water areas are weak. In this paper, a polarized RGB-Depth (pRGB-D) framework is proposed to detect traversable area and water hazards simultaneously with polarization-color-depth-attitude information to enhance safety during navigation. The approach has been tested on a pRGB-D dataset, which is built for tuning parameters and evaluating the performance. Moreover, the approach has been integrated into a wearable prototype which generates a stereo sound feedback to guide visually impaired people (VIP) follow the prioritized direction to avoid obstacles and water hazards. Furthermore, a preliminary study with ten blindfolded participants suggests its effectivity and reliability.

Graphical Abstract">

Graphical Abstract

1. Introduction

According to the World Health Organization, 285 million people around the world are estimated to be visually impaired and 39 million of them are blind [1]. It is rather challenging for visually impaired people (VIP) to naturally navigate through obstacles and avoid water hazards in unknown environments. The invention of RGB-D sensors had a prodigious influence to the research field of visually impaired people assistance with the emergence of light-coding sensor-based approaches [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17] and stereo camera [18,19,20,21,22,23,24,25,26,27,28,29]-based approaches as well as the proof-of-concepts they provided.
The appearance of RGB-D sensors across a wide range of visual assistance contexts is not only due to their low-power-consumption and cost-efficiency, but also its robustness and good performance as they are able to simultaneously perceive color and depth information at video framerates. Apart from color and depth, polarization and its imaging extend information dimension to be used for material discrimination and target detection because polarization parameters reflect physical properties of materials [30]. Meanwhile, given that light with different polarization state behaves differently at the interface of object surface, polarization has been applied to many surface measurement techniques. However, typical commercial RGB-D sensors including light-coding sensors and stereo cameras simply rely on intensity information whereas their polarization cues are generally unavailable or weak.
Light-coding sensors project near-IR speckles and capture the patterns with a calibrated camera, the structured light pattern being coded with spatial or temporary projecting methods [31]. These sensors correlate observed pixels with projected pixels so as to derive depth information through triangulating algorithms. However, many consumer-grade sensors are coded in intensity rather than polarization regardless of whether they use spatial or temporary coding methods. Thereby, as projected speckles are easily lost in sunlight, approaches for VIP with light-coding sensors focus mainly on obstacle avoidance [2,3,9,11,13], traversability awareness [5,6,7,8,12,14] and stairs detection [4,10,15,16,17].
A stereo camera estimates depth maps through stereo matching of images from two lenses. There is a large body of work on stereo matching, some of which are local stereo matching algorithms [32,33,34] while others approximate global optimization [35,36,37]. Local stereo algorithms are generally faster as they calculate the depth of pixels based simply on the correlation of local image patches. Most commercial stereo cameras include local algorithms to achieve speedy refresh rates but return sparse depth information in texture-less scenes, such as a blank wall. Different from conventional stereo cameras, the RGB-D sensor of the RealSense R200 and RS400 [38] devices employed active stereo [39] to combine a pair of cameras with fixed structured light patterns to realize extraordinary environmental adaptability. ZED [40] implemented GPU-accelerated global stereo algorithms to achieve dense and large-scale depth perception, but these commercial stereo cameras comprised IR or RGB lenses and none of them use polarization modulation. As a result, most approaches for VIP with stereo cameras assist navigation in large outdoor spaces [18,19,20,21,22,26,27,28,29] and some schemes have also addressed localization [23,24,25], but none of them detect water hazards to prevent misleading VIP to step into water areas.
In this paper, we proposed to use a polarized a RGB-D sensor for detecting traversable areas and water hazards by adequately considering polarization effects. We equipped a stereo camera with an attitude angle sensor, a pair of horizontal and vertical polarizers to form a pRGB-D sensor. The point cloud in 3D space relative to the sensor is computed with the depth image and intrinsic parameters. The points are adjusted to the world coordinate system with the attitude angles of the sensor. The traversable area is determined by building a stochastic polar occupancy grid, estimating the ground pose by fitting a B-spline surface and using dynamic programming to segment the ground area and obstacles. The water areas are identified by using polarization effects as the primary cue and the polarization difference image is computed with disparity information from the pRGB-D sensor.
This work extends what was presented in [11,12,29], where we addressed the traversable area detection and scene segmentation with a wearable RGB-D sensor. In this paper, we have attached the filters to add polarization information, and generated a stereo sound feedback system to take into account both traversable regions and water areas to guide VIP to follow the prioritized direction in order to avoid obstacles and water hazards, as well as set up a pRGB-D navigation framework which has been proved to be useful and reliable by a field test with ten blindfolded participants.
This paper is organized as follows: in Section 2, related works that have addressed traversability awareness or water hazard detection are reviewed; in Section 3, the presented approach is elaborated in detail; in Section 4, the effectiveness and robustness in terms of the detection and the stereo sound interface are demonstrated with experiments and a user study; and in Section 5, relevant conclusions are drawn and future works are expected.

2. Related Work

Several related works have been dedicated to traversable area detection, which can be divided into two categories: traversable area detection based on scene segmentation and traversable area detection based on surface normal vector estimation. Dakopoulos presented a comparative survey of wearable obstacle avoidance systems to inform the research community and users about the capabilities of the progress in assistive technology for VIP [41]. As for segmenting ground areas from hazardous obstacles, Wang adopted a mean-shift algorithm to regard the local point cloud to be a traversable area if the angle between the fitting plane and horizontal plane in the camera coordinate system is less than a threshold [42]. The approach achieves good robustness under certain environmental conditions with the reliance on thresholds and assumptions. Cheng put forward a seeded region growing algorithm to detect ground areas by adequately considering the depth boundaries instead of attaching importance to setting thresholds [11]. The approach withstands the fluctuations between frames but confuses VIP with the unstable results as the seeded pixels are randomly elected. Rodríguez estimated ground planes based on RANdom SAmple Consensus (RANSAC) and used a polar grid representation to account for the potential obstacles [27]. The approach involved a field test to verify the usability but it yielded a detection error in more than ten percent of the frames. Badino proposed to represent the 3D situation with a set of rectangular sticks for autonomous systems by taking into account the fact that the traversable area in front of vehicles is limited by objects with almost vertical surfaces and estimated the road by fitting a B-spline surface instead of assuming a planar road [43,44]. Elleuch developed a segmentation approach based on possibility modeling theory by imposing a crucial starting point, namely considering reference area placed in the bottom of the image to be traversable [45,46]. As for estimating the surface normal vector to determine traversable areas, Koester detected accessible sections by calculating the gradients and estimating the surface normal vector directions of real-world scene patches. The approach delivers fast and effective detection even in crowded scenes, but it prevents practical applications for user studies due to its overreliance on the quality of 3D reconstruction process and adherence to constraints [47]. For example, it assumes that area directly in front of the user is accessible. Bellone defined a descriptor to measure the unevenness of a local surface based on the estimation of normal vectors [48]. The index gives an enhanced description of the traversable area to perform obstacle avoidance and terrain navigability analysis simultaneously. However, it computes both the inclination and roughness at a low framerate which remains unpromising for assisting at normal walking speed. Ni proposed to extract safe directions by compressing the depth image from a light-coding sensor in the walking assistive robotic system which also comprises ultrasonic sensors to detect the evenness of road surfaces [49]. Cui predicted traversability based on support vector machine (SVM), and planned the optimal path for field robot with consideration of the travelling smoothness [50].
Several related work have been devoted to water hazard detection by taking multi-cue approaches with color, texture, depth and polarization information. Rankin detected water areas by analyzing brightness and saturation values, and searched for the reflections of ground cover with range information, and also discussed the possibility to specifically use the zero disparity to encode water area [51]. Yao extracted the features of brightness and texture from the non-reflective regions with color cameras, and detected the reflected regions where negative height exists and disparity sudden changes with stereo cues [52]. Xie detected water hazards by calculating the polarization degree image and used a self-adaptive segmentation algorithm and a morphology filtering technique to label water regions [53]. The method has good performance in complex natural backgrounds, but it requires mechanically rotating the linear polarizing filter and taking images at different angles, which results in a tradeoff between convenience and effectivity. Yao combined traditional machine learning and mean-shift segmentation, which accurately detects not only common water hazards, but also identifies special water hazards that may have lots of ripples or low brightness [54]. Rankin firstly found the horizon line and constrained the search for water bodies to the region below the horizon. Thereby, this approach decreases the computational cost of water detection as well as reduces the probability of false water detection [55]. Shao used a line-structured light sensor to achieve all-day water puddle detection by measuring the deformation of active light strip [56]. Kim presented a methodology using a stereo camera and treated the detection of wet areas and puddles differently [57]. For the detection of wet areas, the polarization difference, graininess as well as gradient magnitude are combined using hypothesis verification. For the detection of puddles, the depth map obtained by the stereo camera is used to check whether depth changes abruptly around the puddles. Nguyen set up a ZED stereo camera on top of a car with polarizing films on the camera lenses and tracked water hazards based on the polarization and color variation of reflected light with consideration of the effect of polarized light from the sky [58].
Almost all reported works were conducted under ideal scenarios or cause intolerable side effects in navigational assistance for VIP. To the best of our knowledge, no previous work has addressed the detection of water hazards to help VIP to avoid stepping in water areas and no previous work has fully combined the detection of traversable area and the detection of water hazard. Our approach builds on the prior work and makes the breakthrough while the superiority is apparent in the following aspects:
  • The approach is able to detect traversable area and water hazard simultaneously, which prevents colliding into obstacles and stepping in water areas during navigation.
  • The approach takes into account of the attitude angles of the sensor, which enables accurate detection with the continuous movement of the wearable prototype during navigation.
  • The approach combines the detection of water hazard with the detection of traversable area, which decreases the computing cost and reduces water area detection error in scenarios such as sky regions and edges of buildings.

3. Approach

In this section, the approach to detect traversable area and water hazard is elaborated in detail. As shown in Figure 1, the approach is described in terms of the pRGB-D sensor, the traversable area detection, the water hazard detection and the stereo sound feedback accordingly.

3.1. pRGB-D Sensor

As shown in Figure 1, the pRGB-D sensor is comprised of a ZED stereo camera which is equipped with horizontal and vertical polarization filters on the left and right camera, respectively, as well as a MPU6050 attitude sensor [59]. The stereo camera delivers 720 p left-right color video pairs and a depth image at 60 frames per second with the resolution same as the color video. The wide-angle all-glasses dual lenses share a diagonal field of view of 110 degrees. The sensor ranges from 0.5 m to 20 m which is quite suitable to detect traversable area and water hazard.
For the traversability awareness, it not only refers to long-range ground plane detection but also involves the early warning when a VIP approaches obstacles at close-range. For obstacle detection, the minimum range of 0.5 m, equivalent to the length of an arm, is enough for the avoidance of most obstacles given the normal walking speed of a VIP during navigation.
For water hazard detection, the primary cue is the polarization effect as specular reflection on water is known to polarize light [58]. The specular reflection from the water surface R r e f l e c t is the sum of two polarization components R r e f l e c t , ⊥ and R r e f l e c t , / / , perpendicular and parallel respectively to the plane formed by the incident and reflected rays as given in Equations (1) and (2). As shown in Figure 2, n a i r and n w a t e r are the refractive indices of air and water respectively and θ is the reflection angle at the water surface.
R r e f l e c t , ⊥ ( n a i r , n w a t e r , θ ) = [ n a i r c o s θ − n w a t e r 1 − ( n a i r / n w a t e r ) 2 s i n 2 θ n a i r c o s θ + n w a t e r 1 − ( n a i r / n w a t e r ) 2 s i n 2 θ ] 2
R r e f l e c t , / / ( n a i r , n w a t e r , θ ) = [ − n w a t e r c o s θ + n a i r 1 − ( n a i r / n w a t e r ) 2 s i n 2 θ n w a t e r c o s θ + n a i r 1 − ( n a i r / n w a t e r ) 2 s i n 2 θ ] 2
Suppose the polarized light from the air comprises energy components E ⊥ S ( θ ) and E / / S ( θ ) for perpendicular and parallel components respectively as functions of reflection angle. The total energy entering the water can be calculated with Equation (3):
F S = E ⊥ S ( θ ) [ 1 − R r e f l e c t , ⊥ ( n a i r , n w a t e r , θ ) ] + E / / S ( θ ) [ 1 − R r e f l e c t , / / ( n a i r , n w a t e r , θ ) ]
However, part of the energy F S is scattered by suspended particles and ground bottom while the rest is absorbed by both particles and the ground as shown in Equation (4) where μ p a r t i c l e s and μ b o t t o m are the scattering coefficients of particles and the ground bottom respectively, and μ a b s o r p t i o n is the absorption coefficient:
μ p a r t i c l e s + μ b o t t o m + μ a b s o r p t i o n = 1
Light in water can be considered as highly unpolarized light with random scattering and internal reflection. With part of the scattered light coming out of the water through refraction, the total light energy component coming out of the water is the summation of reflection and refraction for each polarization component, which can be calculated with Equations (5) and (6) where θ ′ is the refraction angle from water to air:
E ⊥ R ( θ ) = E ⊥ S ( θ ) R r e f l e c t , ⊥ ( n a i r , n w a t e r , θ ) + 0.5 F S [ μ p a r t i c l e s + μ b o t t o m ] R r e f r a c t , ⊥ ( n a i r , n w a t e r , θ ′ = θ )
E / / R ( θ ) = E / / S ( θ ) R r e f l e c t , / / ( n a i r , n w a t e r , θ ) + 0.5 F S [ μ p a r t i c l e s + μ b o t t o m ] R r e f r a c t , / / ( n a i r , n w a t e r , θ ′ = θ )
For illustrative purposes, in Figure 2, the water refractive index n w a t e r is set to 1.33 and the absorption coefficient μ a b s o r p t i o n is set to 60%. Figure 3 shows that the polarization difference between perpendicular and parallel components is large enough to provide a strong cue for water hazards at reflection angles above 70 degrees or at distances above the minimum detection range. Thereby, it is able to detect water hazards by setting the threshold of polarization difference with point correspondence from left color image to right color image.
If the wearing height of the pRGB-D sensor is H , the minimum water hazard detection range D is calculated with Equation (7), where φ is the vertical field view of the stereo camera, which equals 55.75 degrees:
D = H × tan ( θ − φ / 2 )
Thereby, given the average value of user height, the wearing height H is set to 1.6 m, and the reflection angle θ is set to 70 degrees. Thus, the minimum range D is around 1.45 m which is enough to provide immediate feedback to avoid stepping in water areas and quite matched with the working range of the depth sensor. However, with the continuous movement of the user head during navigation, the threshold to segment potential water hazards should be varying as the polarization direction of reflected light would not be strictly parallel or perpendicular with the polarizer in this circumstances. Given the rolling angle of the pRGB-D sensor equaling to η , which is obtained with an attitude sensor, the dynamic threshold is set to δ = δ 0 cos 2 η , where δ 0 is the threshold if there is no rolling. Additionally, notwithstanding different brightness of water area in left and right image due to the specular reflection, depth information of water areas on the road is available as the stereo camera in the pRGB-D sensor implemented a global stereo matching method to approximate smoothness constraint for compensating local brightness differences and radiometric transformations [35]. As a result, the pRGB-D sensor is quite suitable for navigational assistance thanks to its wearable computing technology and its compatibility to support traversable area and water hazard detection.

3.2. Traversable Area Detection

In order to detect traversable area, a simple and effective technique is presented. First, 3D coordinates of the point cloud are calculated. Given the depth Z of pixel ( u , v ) in the depth image, the calibrated focal length f , the radial distortion coefficients c 1 , c 2 , c 5 , the tangential distortion coefficients c 3 , c 4 , as well as ( u 0 , v 0 ) the principal point, the point ( X , Y , Z ) in the camera coordinate system can be derived with Equations (8)−(13):
x = u − u 0 f
y = v − v 0 f
r 2 = x 2 + y 2
t = 1 + c 1 r 2 + c 2 r 4 + c 5 r 6
X = Z × [ x t + 2 c 3 x y + c 4 ( r 2 + 2 x 2 ) ]
Y = Z × [ y t + 2 c 4 x y + c 3 ( r 2 + 2 y 2 ) ]
With the help of the attitude sensor, X , Y and Z coordinates in the camera coordinate system are adjusted to world coordinates. Assume a point in the camera coordinate system is ( X , Y , Z ) and the attitude angles acquired from the attitude sensor are ( a , b , c ) . This means the point ( X , Y , Z ) rotates about the x-axis by α = a , then rotates about the y-axis by β = b and rotates about z-axis by γ = c in the end. Multiplying the point ( X , Y , Z ) by the rotation matrix given by Equation (14), the point ( X w , Y w , Z w ) in world coordinate system is obtained:
[ X w Y w Z w ] = [ cos γ − sin γ 0 sin γ cos γ 0 0 0 1 ] [ cos β 0 sin β 0 1 0 − sin β 0 cos β ] [ 1 0 0 0 cos α − sin α 0 sin α cos α ] [ X Y W ]
After the point cloud adjustment, occupancy grids are computed with the stereo disparities transferred from the adjusted point cloud coordinates, and the method proposed in [60] is used to propagate the uncertainty of the stereo disparities onto the grid. In the polar occupancy grid, the image column is used to represent the angular coordinate and the adjusted disparity is used to represent the range. 3D points lying above the ground area are registered as obstacles in the occupancy grid and extremely high 3D points are dismissed such as ceilings which will not influence the travelling of VIP. Moreover, the ground pose is fitted by a B-Spline surface with consideration of two facts: real-world ground areas are not always planar surfaces [44] and the lenses share a distortion given a diagonal field of view of 110 degrees. Our system builds on the work of Stixel World [43], which represents the scene using a few upright objects on the ground plane. The computation includes dividing the input image into a set of vertical columns, and searching for regions that could be immediately reached without collision. In this sense, the task of traversable area detection is to find the first visible relevant obstacle in the positive direction of depth. In this work, instead of using a single threshold to segment obstacles and traversable areas independently, dynamic programming is used to seek the optimal path cutting the polar grid from left to right by imposing a spatial smoothness and a temporal smoothness to penalize both jumps in depth and deviations of the current frame from the previous frame. Figure 4 depicts frames of traversable area detection results by producing a green mask to overlay on the left color image acquired with the pRGB-D sensor.

3.3. Water Hazard Detection and Stereo Sound Feedback

As shown with the flowchart in Figure 5, in order to detect water hazards out of the traversable area, we first generate the disparity image from the depth image produced by the pRGB-D sensor.
Considering pixel G ( u , v ) in the depth image which ranges d , the calibrated focal length f , the baseline length B , the disparity value d i s of pixel G can be calculated with Equation (15):
d i s = B × f / d
After that, we warp the right image to the left image to produce a point correspondence for water hazard detection. With the stereo pair, we are able to calculate a brightness difference image which signifies the polarization difference. Considering pixel G whose image coordinate is ( u , v ) in left image and corresponds to the pixel G ′ ( u + d i s , v ) in right image. If the brightness at G ( u , v ) in left image is V l e f t and the brightness in the warped right image is V r i g h t at G ′ ( u + d i s , v ) . The left image is captured by the left color camera with the horizontal polarizer while the right image is captured by the right color camera with the vertical polarizer. Thereby, the brightness difference of pixel G equals | V l e f t − V r i g h t | . After extraction of the brightness difference image, it depends on the following conditions whether pixel G can be classified as water hazard:
  • Pixel G is classified as traversable in the process as presented in Section 3.2, where traversable area is detected with depth and attitude information.
  • | V l e f t − V r i g h t | > δ , where δ is the dynamic threshold of polarization difference set with the derivation in Section 3.1 and varies with the rolling angle of the pRGB-D sensor.
Figure 6 depicts frames of water hazard detection results by producing magenta masks on the green traversable area detection results as presented in Section 3.2.
After traversable area detection and water hazard detection, a farthest traversable line to represent the traversable distances in different directions is proposed to generate the stereo sound interface. In each direction, the farthest distance for navigating is determined by both the traversable area and water hazard detection results. In each direction, we first locate the farthest traversable pixel with the biggest depth value Z t and then search for the nearest pixel denoted as water area whose depth value is Z w . The farthest traversable depth value Z f equals to Z w if a water area is detected and equals to Z t if no water areas are detected in this direction. Gaussian filtering is applied to smooth the farthest traversable line to reduce noises before generating feedback to VIP. After the filtering stage, the green farthest traversable lines with or without water hazard detection are obtained, as shown in Figure 7 where the red triangles represent the suggested navigation directions to take to avoid obstacles or water hazards.
This paper uses a variant of the non-semantic audio interface presented in our previous work [29] to transfer the traversable area and water hazard detection results to VIP by synthesizing stereo sound from the farthest traversable line. In this work, the mapping of traversable distances to the different sonification of instruments is elaborated.
As shown in Figure 8, the directions of traversable areas are differentiated not only by sound source locations in the virtual 3D space and the directions of stereo sound, but also by musical instruments, whose timbre differs from each other.
The generation of the stereo sound follows rules below to guide and attract VIP to take the prioritized direction to detour around hazardous regions:
  • Divide the farthest traversable line into five sections which correspond to the five different musical instruments. We only use five instruments to make sure that it is easy to understand and would not sound confusing after the familiarization period. Five instruments, including trumpet, piano, gong, violin and xylophone, produce sounds simultaneously.
  • The horizontal field view of the pRGB-D sensor is 86.5°, so each musical instrument corresponds to the traversable line with a range of 17.3°.
  • Each direction of traversable area and water hazard is represented by a musical instrument in the virtual 3D space.
  • For each musical instrument, the bigger the sum of height in the corresponding section of the traversable line, the louder the sound from the instrument. As the relationship shown in Figure 9, the instrument loudness increases exponentially with the average traversable distance in the corresponding section of the traversable line so as to guide or attract the user to navigate the traversable direction.
  • For each musical instrument and the corresponding section, the bigger the areas that overlap with the red triangle sections which denote suggested directions to take on navigation, the higher the pitch of the instrument. As the relationship shown in Figure 10, the instrument pitch increases linearly with the proportion of the overlapping areas with the red triangle sections.

4. Experiments

In this section, the proposed approach is verified with a series of experiments including tests on a score of indoor and outdoor scenarios, comparisons with other works in the literature and a field test in the real-world environment involving ten voluntary blindfolded users.

4.1. Indoor and Outdoor Detection Results

Figure 11 shows a number of traversable area detection and water area detection results in indoor and outdoor environments. The approach detects traversable areas and generates farthest traversable lines correctly, whose red triangles suggest the prioritized directions to avoid close obstacles and water hazards.

4.2. Comparison of Detection Results

To compare the performance of traversable area and water hazard detection with respect to other works, the results of several approaches are shown in Figure 8. Figure 12a−c are the results of the proposed approach which detects ground areas and water areas correctly. Comparatively, Figure 12d–f show the indoor and outdoor results of the approach proposed by Rodríguez which estimated the ground plane based on RANSAC plus filtering techniques [27]. Figure 12f is a correct result of detecting ground with some noises and holes, but the ground is partly detected in Figure 12e which is a type of sample error as the inclination angle of the plane is not considered.
We removed this kind of error in our work with adequate use of the attitude angles of the pRGB-D sensor. The approach in [58] set up a stereo camera to track water areas by modeling the sky light effect with reflection and azimuth angles. That approach is able to detect water hazards robustly, but mistakenly detects sky regions or edges of buildings as water areas as shown in Figure 12g−i. As the approach proposed in this paper combines the detection of water hazards and traversable areas, the aerial detection errors which would not hinder the navigation of VIP are reduced in number. Thus, the aerial errors would not be transferred to mislead VIP to avoid the fake water hazards. The total computation time of a single frame is 70.5 ms, while the image acquisition from the pRGB-D sensor takes 3 ms, and the time cost for the segmentation of traversable area and the detection of water hazard are 48 ms and 19.5 ms, respectively. Thereby, the computation cost is saved to maintain a reasonably qualified refresh rate of 14.2 frames per second on a 4.0 GHz Intel Core 7 processor with the resolution empirically set at 640 × 360, which is enough to feed results back in time to assist VIP at normal walking speed.
In order to provide a quantitative evaluation of the approach, the performance of the algorithm is thoroughly analyzed by calculating traversable area detection rate ( T A D R ), water area detection rate ( W A D R ) and expansion error ( E E ). Given Equations (16)−(18), traversable area detection rate ( T A D R ) is defined as the number of frames which ground has been detected correctly ( G D ) divided by the number of frames with ground ( G ). Water area detection rate ( W A D R ) is defined as the number of frames which water area has been detected correctly ( W A D ) divided by the number of frames with water area ( W A ). Meanwhile, expansion error ( E E ) is defined as the number of frames which traversable area has been expanded to non-ground areas ( E N G ) including water areas divided by the number of frames with ground ( G ):
T A D R = G D D
W A D R = W A D W A
E E = E N G G
For quantitative analysis we used a sequence of 1285 frames from the pRGB-D dataset [61], which is built for tuning parameters and testing. The dataset is captured by the wearable sensor whose images of a single frame are respectively the depth image, left and right color images acquired with different polarizers whose polarization directions are perpendicular to each other after calibrations using a polarization analyzer. As shown in Table 1, the approach has a high traversable area detection rate of 94.4% compared with respect to other works [27,29]. Moreover, the expansion error has been decreased to 7.3% after the combination of traversable area detection and water area detection, illustrating the reliability of the approach.

4.3. Detection Limitations

The proposed pipeline effectively combines polarization-color-depth-attitude information and improves the detection performance in indoor and outdoor environments, which is able to provide real-time assistances for VIP to avoid colliding into obstacles or stepping into water areas. However, it still has certain potential limitations. Since the proposed traversable area detection is based on the pRGB-D sensor whose depth information is acquired with stereo matching between left and right RGB images, the approach will have a slight problem when the depth information is defective as shown in Figure 13. As the textures of glass door and the overexposed region are poor, the depth information are invalid or filled with wrong values offered by the stereo type of RGB-D sensor in our prototype as shown in Figure 13a−c. Thereupon, the approach wrongly detects the region of glass door as a traversable area, and merely part of the ground area close to the overexposed wall is detected. Besides, as shown in Figure 13d–f, the approach is insensitive to small obstacles on the ground due to the depth filtering and dynamic programming strategy. To solve this problem, in the future time we aim to implement a small obstacle discovery algorithm [62] or provide both physical support and navigational indication for tiny threat awareness [49].

4.4. User Study

To evaluate the ability of our system to help visually impaired users follow a prioritized route without losing their orientation or colliding into obstacles or stepping into water puddles, we conducted a user study in a real-world outdoor space. Before the participants were blindfolded, the stereo sound feedback of the system when wearing a bone-conduction headphone was explained as shown in Figure 14a,b. As we know, VIP or blindfolded users rely on cues from the environment a lot. For example, they use the sounds from their ears to understand the orientations of streets. Thereby, the assisting prototype is not only wearable but also ears-free, because the bone-conducting interface will not block the ears of participants from hearing environmental sounds. This familiarization period lasted for about seven minutes, during which they were able to adapt to hearing the stereo sound to keep away from close hazards including obstacles and water areas.
After the learning and familiarization stage, ten voluntary participants were blindfolded and were required to travel around a building and an empty field without colliding into obstacles or stepping into water areas. As a comparison task, the ten volunteers were divided into two groups randomly and while one group travelled with only traversable area detection, the other group travelled with both traversable area and water hazard detection. To provide the participants with a sense of orientation, the user would get an extra hint to turn at the bends or the road intersections. For a single test, three measures were recorded including the time needed by a participant to walk the route as shown in Figure 14c, the number of collisions into obstacles and the number of times they stepped into water areas. The route was acquired through recording the real-time locations during navigation with the red labels on the map provided by AMAP [63]. The timer starts when a participant was sent to the start region and stops when the participant completed navigating the route. As shown in Table 2, average number of times stepping into water areas with water hazard detection is significantly less than that without water hazard detection. It is the detection of water hazard in advance that endows participants to take the prioritized direction to avoid water hazards compared with the condition with only traversable area detection. Moreover, the average number of collisions into obstacles is few and most of the collisions occurred when the participants swerved and hit a curb which was outside the horizontal field view of the pRGB-D sensor. Comparatively, times of stepping in water areas are very few but not none, which is due to the minimum range of water hazard detection. If the user did not turn when hearing the stereo sound and kept moving forward, the water area would be out of the vertical field view of the pRGB-D sensor or the reflection angles would not be large enough to provide a strong cue for water hazards. Nonetheless, explicit feedback about directions to walk to avoid obstacles and water hazards were found helpful for assisting VIP. The results suggest that participants were aware of obstacles and water areas with our system and could make use of the stereo sound to keep away from or detour around the hazards. In other word, the safety and robustness of navigating VIP has been dramatically enhanced with the traversable area and water hazard detection based on the proposed pRGB-D framework.
After the test, the ten participants were asked two simple questions including whether the prototype was easy to wear and whether the stereo sound feedback could be used to find traversable directions. As shown in the questionnaire results (Table 3), all users answered that the system is useful and could help them to find traversable areas and avoid obstacles or water hazards.
Most participants were able to understand the changing process of loudness and pitch, and follow the stereo sound to find the traversable directions and detour around the hazards, while a small group of participants only used the direction of the stereo sound to turn left or turn right to avoid collisions. This demonstrated the usefulness and effectivity of the approach in terms of traversable area detection and stereo sound feedback. In addition, some users provided some advice on adding functions, such as the detection of curbs and negative obstacles. Moreover, the users were optimistic about the pRGB-D system and would like to have a more profound experience.

5. Conclusions

RGB-D sensors are a ubiquitous choice to assist navigation for the visually impaired. However, most solutions are confined to using intensity information whereas the polarization cues are weak. The proposed pRGB-D framework enhances the safety and robustness of navigation by combining traversable area detection and water hazard detection with polarization-color-depth-attitude information. Indoor and outdoor empirical evidences, quantitative analysis of the detection as well as a user study with ten blindfolded participants demonstrated its usefulness and reliability.
In the future, we will improve our navigation assistance approach to reach a higher level of perception and offer more independence to visually impaired individuals. Specifically, we look forward to including polarization for ranging and further investigating discovery schemes for the detection of small obstacles such as curbs and transparent objects. Additionally, it is necessary to run a larger study with visually impaired participants to test this approach, and different audio output settings could be compared. The use of portable processors and sensors with larger field of view would also benefit the system in terms of wearable mobility and detectable performance.

Acknowledgments

We acknowledge I. C. Khoo for the heuristic discussions on liquid crystals photonics during the research. We acknowledge Hangzhou KR-VISION Technology for the organization of the user study. We acknowledge the anonymous reviewers for the insightful comments.

Author Contributions

Kailun Yang conceived the method, performed the traversable area and water hazard detection, as well as wrote the paper. Kaiwei Wang designed the experiments. Ruiqi Cheng captured the pRGB-D dataset. Weijian Hu developed the stereo sound interface. Xiao Huang calibrated the pRGB-D sensor. Jian Bai analyzed the experimental results. All authors participated in the user study and approved the paper.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. World Health Organization. Visual Impairment and Blindness. Available online: http://www.who.int/mediacentre/factsheets/fs282/en/ (accessed on 6 June 2017).
  2. Zollner, M.; Huber, S.; Jetter, H.; Reiterer, H.C.; Reiterer, H. NAVI—A Proof-of-Concept of a Mobile Navigational Aid for Visually Impaired Based on the Microsoft Kinect. In Human—Computer Interaction—INTERACT 2011; Graham, P.C.N., Nunes, J.J.N., Winckler, P.M.M., Eds.; Springer: Berlin/Heidelberg, Germany, 2007; pp. 584–587. [Google Scholar]
  3. Takizawa, H.; Yamaguchi, S.; Aoyagi, M.; Ezaki, N.; Mizuno, S. Kinect cane: An assistive system for the visually impaired based on three-dimensional object recognition. Pers. Ubiquitous Comput. 2012, 19, 740–745. [Google Scholar]
  4. Filipe, V.; Fernandes, F.; Fernandes, H.; Sounsa, A.; Paredes, H.; Barroso, J. Assisted Guidance for the Blind Using the Kinect Device. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, UTAD, Vila Real, Portugal, 1–3 December 2016; pp. 13–19. [Google Scholar]
  5. Park, C.H.; Howard, A.M. Real-time haptic rendering and haptic telepresence robotic system for the visually impaired. In Proceedings of the World Haptics Conference (WHC), Daejeon, Korea, 14–18 April 2013; pp. 229–234. [Google Scholar]
  6. Hicks, S.L.; Wilson, I.; Muhammed, L.; Worsfold, J.; Downes, S.M.; Kennard, C. A depth-based head-mounted visual display to aid navigation in partially sighted individuals. PLoS ONE 2013, 8, e67695. [Google Scholar] [CrossRef] [PubMed]
  7. Wang, Z.; Liu, H.; Wang, X.; Qian, Y. Segment and label indoor scene based on RGB-D for the visually impaired. In Proceedings of the International Conference on Multimedia Modeling, Dublin, Ireland, 6–10 January 2014; pp. 449–460. [Google Scholar]
  8. Aladren, A.; Lopez-Nicolas, G.; Puig, L.; Guerrero, J.J. Navigational assistance for the visually impaired using rgb-d sensor with range expansion. IEEE Syst. J. 2014, 99, 1–11. [Google Scholar]
  9. Hsieh, C.T.; Lai, W.M.; Yeh, C.H.; Huang, H.C. An Obstacle Detection System Using Depth Information and Region Growing for Blind. Res. Notes Inf. Sci. 2013, 465–470. [Google Scholar]
  10. Guerrero, J.J.; Pérez-Yus, A.; Gutiérrez-Gómez, D.; Rituerto, A.; López-Nicolás, G. Human navigation assistance with a RGB-D sensor. In Proceedings of the ACTAS V Congreso Internacional de Turismo para Todos: VI Congreso Internacional de Diseño, Redes de Investigación y Tecnología para todos DRT4ALL, Madrid, Spain, 23–24 September 2015; pp. 285–312. [Google Scholar]
  11. Cheng, R.; Wang, K.; Yang, K.; Zhao, X. A ground and obstacle detection algorithm for the visually impaired. In Proceedings of the IET International Conference on Biomedical Image and Signal Processing (ICBISP 2015), Beijing, China, 19 November 2015. [Google Scholar] [CrossRef]
  12. Yang, K.; Wang, K.; Cheng, R.; Zhu, X. A new approach of point cloud processing and scene segmentation for guiding the visually impaired. In Proceedings of the IET International Conference on Biomedical Image and Signal Processing (ICBISP 2015), Beijing, China, 19 November 2015. [Google Scholar] [CrossRef]
  13. Saputra, M.R.U.; Santosa, P.I. Obstacle Avoidance for Visually Impaired Using Auto-Adaptive Thresholding on Kinect’s Depth Image. In Proceedings of the 2014 IEEE 11th Intl Conf on Ubiquitous Intelligence and Computing and 2014 IEEE 11th Intl Conf on Autonomic and Trusted Computing and 2014 IEEE 14th International Conference on Scalable Computing and Communications and Its Associated Workshops, Bali, Indonesia, 9–12 December 2014. [Google Scholar] [CrossRef]
  14. Blessenohl, S.; Morrison, C.; Criminisi, A.; Shotton, J. Improving Indoor Mobility of the Visually Impaired with Depth-Based Spatial Sound. In Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile, 7–13 December 2015; pp. 26–34. [Google Scholar]
  15. Perez-Yus, A.; Lopez-Nicolas, G.; Guerrero, J.J. Detection and modelling of staircases using a wearable depth sensor. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 449–463. [Google Scholar]
  16. Munoz, R.; Rong, X.; Tian, Y. Depth-aware indoor staircase detection and recognition for the visually impaired. In Proceedings of the 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Seattle, WA, USA, 11–15 July 2016; pp. 1–6. [Google Scholar]
  17. Perez-Yuz, A.; Gutierrez-Gomez, D.; Lopez-Nicolas, G.; Guerrero, J.J. Stairs detection with odometry-aided traversal from a wearable RGB-D camera. Comput. Vis. Image Underst. 2017, 154, 192–205. [Google Scholar] [CrossRef]
  18. Wong, F.; Nagarajan, R.; Yaacob, S. Application of stereovision in a navigation aid for blind people. In Proceedings of the 2003 IEEE Joint Conference of the Four Information Communications and Signal Processing, and Fourth Pacific Rim Conference on Multimedia, Singapore, 15–18 December 2003; pp. 734–737. [Google Scholar]
  19. Johnson, L.A.; Higgins, C.M. A navigation aid for the blind using tactile-visual sensory substitution. In Proceedings of the 28th Annual International Conference of the IEEE on Engineering in Medicine and Biology Society (EMBS’06), New York, NY, USA, 31 August–3 September 2006; pp. 6289–6292. [Google Scholar]
  20. Rodriguez, A.; Bergasa, L.M.; Alcantarilla, P.F.; Yebes, J.; Cela, A. Obstacle avoidance system for assisting visually impaired people. In Proceedings of the IEEE Intelligent Vehicles Symposium Workshops, Madrid, Spain, 3–7 June 2012; p. 16. [Google Scholar]
  21. Martinez, J.M.S.; Ruiz, F.E. Stereo-based aerial obstacle detection for the visually impaired. In Proceedings of the Workshop on Computer Vision Applications for the Visually Impaired, Marseille, France, 18 October 2008. [Google Scholar]
  22. Pradeep, V.; Medioni, G.; Weiland, J. Robot vision for the visually impaired. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA, 13–18 June 2010; pp. 15–22. [Google Scholar]
  23. Brilhault, A.; Kammoun, S.; Gutierrez, O.; Truillet, P.; Jouffrais, C. Fusion of artificial vision and GPS to improve blind pedestrian positioning. In Proceedings of the 2011 4th IFIP International Conference on New Technologies, Mobility and Security (NTMS), Paris, France, 7–10 February 2011; pp. 1–5. [Google Scholar]
  24. Lee, Y.H.; Medioni, G. RGB-D camera based navigation for the visually impaired. In Proceedings of the RSS RGBD Advanced Reasoning with Depth Camera Wproorkshop, Los Angeles, CA, USA, 27 June 2011; pp. 1–5. [Google Scholar]
  25. Alcantarilla, P.F.; Yebes, J.J.; Almazan, J.; Bergasa, L.M. On Combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation (ICRA), Saint Paul, MN, USA, 14–18 May 2012; pp. 1290–1297. [Google Scholar]
  26. Lin, K.W.; Lau, T.K.; Cheuk, C.M.; Liu, Y. A wearable stereo vision system for visually impaired. In Proceedings of the 2012 International Conference on Mechatronics and Automation (ICMA), Chengdu, China, 5–8 August 2012; pp. 1423–1428. [Google Scholar]
  27. Rodriguez, A.; Yebes, J.J.; Alcantarilla, P.F.; Bergasa, L.M.; Almazan, J.; Cele, A. Assisting the visually impaired: Obstacle detection and warning system by acoustic feedback. Sensors 2011, 12, 17476–17496. [Google Scholar] [CrossRef] [PubMed]
  28. Miksik, O.; Vineet, V.; Lidegaard, M.; Prasaath, R.; Nebner, M.; Golodetz, S.; Hicks, S.L.; Pérez, P.; Izadi, S.; Torr, P.H.S. The semantic paintbrush: Interactive 3d mapping and recognition in large outdoor spaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea, 18–23 April 2015; pp. 3317–3326. [Google Scholar]
  29. Yang, K.; Wang, K.; Hu, W.; Bai, J. Expanding the Detection of Traversable Area with RealSense for the Visually Impaired. Sensors 2016, 16, 1954. [Google Scholar] [CrossRef] [PubMed]
  30. Huang, X.; Bai, J.; Wang, K.; Liu, Q.; Luo, Y.; Yang, K.; Zhang, X. Target enhanced 3D reconstruction based on polarization-coded structured light. Opt. Express 2017, 25, 1173–1184. [Google Scholar] [CrossRef] [PubMed]
  31. Ryan Fanello, S.; Rhemann, C.; Tankovich, V.; Kowdle, A.; Orts Escolano, S.; Kim, D.; Izadi, S. Hyperdepth: Learning depth from structured light without matching. In Proceedings of the IEEE Conference on Computer Vision and Patten Recognition, Seattle, WA, USA, 27–30 June 2016; pp. 5441–5450. [Google Scholar]
  32. Einecke, N.; Eggert, J. A two-stage correlation method for stereoscopic depth estimation. In Proceedings of the 2010 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia, 1–3 December 2010; pp. 227–234. [Google Scholar]
  33. Einecke, N.; Eggert, J. Block-matching stereo with relaxed fronto-parallel assumption. In Proceedings of the Intelligent Vehicles Symposium, Ypsilanti, MI, USA, 8–11 June 2014; pp. 700–705. [Google Scholar]
  34. Einecke, N.; Eggert, J. A multi-block matching approach for stereo. In Proceedings of the 2015 IEEE Intelligent Vehicles Symposium (IV), Seoul, Korea, 28 June–1 July 2015; pp. 585–592. [Google Scholar]
  35. Hirschmuller, H. Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 328–341. [Google Scholar] [CrossRef] [PubMed]
  36. Xie, Y.; Zeng, S.; Chen, L. A Novel Disparity Refinement Method Based on Semi-Global Matching Algorithm. In Proceedings of the 2014 IEEE International Conference on Data Mining Workshop (ICDMW), Shenzhen, China, 14 December 2014; pp. 1135–1142. [Google Scholar]
  37. Hernandez-Juarez, D.; Chacon, A.; Espinosa, A.; Vazquez, D.; Moure, J.C.; Lopez, A.M. Embedded real-time stereo-estimation via Semi-Global Matching on the GPU. Procedia Comput. Sci. 2016, 80, 143–153. [Google Scholar] [CrossRef]
  38. Keselman, L.; Woodfill, J.I.; Grunnet-Jepsen, A.; Bhowmik, A. Intel RealSense Stereoscopic Depth Cameras. arXiv 2017, arXiv:1705.05548. [Google Scholar]
  39. Konolige, K. Projected texture stereo. In Proceedings of the 2010 IEEE International Conference on Robotics and Automation(ICRA), Anchorage, AK, USA, 4–8 May 2010; pp. 148–155. [Google Scholar]
  40. Stereolabs. Available online: Http://www.stereolabs.com (accessed on 6 June 2017).
  41. Dakopoulos, D.; Bourbakis, N.G. Wearable obstacle avoidance electronic travel aids for blind: A survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2015, 40, 25–35. [Google Scholar] [CrossRef]
  42. Wang, T.; Bu, L.; Huang, Z. A new method for obstacle detection based on Kinect depth image. In Proceedings of the 2015 Chinese Automation Congress (CAC), Wuhan, China, 27–29 November 2015; pp. 537–541. [Google Scholar]
  43. Badino, H.; Franke, U.; Pfeiffer, D. The stixel world-a compact medium level representation of the 3D-world. In Proceedings of the 31st DAGM Symposium, Jena, Germany, 9–11 September 2009; pp. 51–60. [Google Scholar]
  44. Wedel, A.; Franke, U.; Badino, H.; Cremers, D. B-spline modelling of road surfaces for freespace estimation. In Proceedings of the 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands, 4–6 June 2008; pp. 808–833. [Google Scholar]
  45. Elleuch, J.F.; Bellaaj, M.; Sellami, D.; Kallel, I.K. Travesable area segmentation approach at indoor environment for visually impaired people. In Proceedings of the 13th International Conference on Advances in Mobile Computing and Multimedia, Brussels, Belgium, 11–13 December 2015; pp. 48–53. [Google Scholar]
  46. Frikha, J.; Sellami, D.; Kallel, I.K. Indoor/outdoor navigation system based on possibilistic traversable area segmentation for visually impaired people. ELCVIA Electron. Lett. Comput. Vis. Imag. Anal. 2016, 15, 60–76. [Google Scholar] [CrossRef]
  47. Koester, D.; Schauerte, B.; Stiefelhagen, R. Accessible section detection for visual guidance. In Proceedings of the IEEE International Conference on Multimedia and Expo Workshops, San Jose, CA, USA, 15–19 July 2013; pp. 1–6. [Google Scholar]
  48. Bellone, M.; Messina, A.; Reina, G. A new approach for terrain analysis in mobile robot applications. In Proceedings of the IEEE International Conference on Mechatronics, Wollongong, Australia, 9–12 July 2013; pp. 225–230. [Google Scholar]
  49. Ni, D.; Song, A.; Tian, L.; Xu, X.; Chen, D. A walking assistant robotic system for the visually impaired based on computer vision and tactile perception. Int. J. Soc. Robot. 2015, 7, 617–628. [Google Scholar] [CrossRef]
  50. Cui, J.; Guo, Y.; Zhang, H.; Qian, K.; Bao, J.; Song, A. Support Vector Machine Based Robotic Traversability Prediction with Vision Features. Int. J. Comput. Intell. Syst. 2013, 6, 596–608. [Google Scholar] [CrossRef]
  51. Rankin, A.L.; Matthies, L.H.; Huertas, A. Daytime water detection by fusing multiple cues for autonomous off-road navigation. In Transformational Science and Technology for the Current and Future Force; Wei, T.K., Ed.; World Scientific Publishing Co. Pte. Ltd.: Singapore, 2006; pp. 177–183. [Google Scholar]
  52. Yao, T.; Xiang, Z.; Liu, J.; Xu, D. Multi-feature fusion based outdoor water hazards detection. In Proceedings of the 2007 IEEE International Conference on Mechatronics and Automation (ICMA), Harbin, China, 5–8 August 2007; pp. 652–656. [Google Scholar]
  53. Xie, B.; Pan, H.; Xiang, Z.; Liu, J. Polarization-based water hazards detection for autonomous off-road navigation. In Proceedings of the International Conference on Mechatronics and Automation (ICMA), Harbin, China, 5–8 August 2007; pp. 1666–1670. [Google Scholar]
  54. Yao, T.Z.; Xiang, Z.Y.; Liu, J.L. Robust water hazard detection for autonomous off-road navigation. J. Zhejiang Univ. 2009, 10, 786–793. [Google Scholar] [CrossRef]
  55. Rankin, A.L.; Matthies, L.H.; Bellutta, P. Daytime water detection based on sky reflections. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, 9–13 May 2011; pp. 5329–5336. [Google Scholar]
  56. Shao, H.; Zhang, Z.; Li, K. Research on water hazard detection based on line structured light sensor for long-distance all day. In Proceedings of the 2015 IEEE International Conference on Mechatronics and Automation (ICMA), Beijing, China, 2–5 August 2015; pp. 1785–1789. [Google Scholar]
  57. Kim, J.; Baek, J.; Choi, H.; Kim, E. Wet area and puddle detection for Advanced Driver Assistance Systems (ADAS) using a stereo camera. Int. J. Control Autom. Syst. 2016, 14, 263. [Google Scholar] [CrossRef]
  58. Nguyen, C.V.; Milford, M.; Mahony, R. 3D tracking of water hazards with polarized stereo cameras. arXiv 2017, arXiv:1701.04175. [Google Scholar]
  59. InvenSense MPU-6050. Available online: https://playground.arduino.cc/Main/MPU-6050 (accessed on 6 June 2017).
  60. Badino, H.; Franke, U.; Mester, R. Free space computation using stochastic occupancy grids and dynamic programming. In Proceedings of the 2007 ICCV Workshop on Dynamical Vision, Rio de Janeiro, Brazil, 20 October 2007; Available online: http://www.lelaps.de/papers/badino_wdv2007.pdf (accessed on 12 August 2017).
  61. Kaiwei Wang Team. Available online: wangkaiwei.org (accessed on 6 June 2017).
  62. Kumar, S.; Karthik, M.S.; Krishna, K.M. Markov Random Field based small obstacle discovery over images. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 494–500. [Google Scholar]
  63. AMAP. Available online: http://ditu.amap.com/ (accessed on 6 June 2017).
Sensors 17 01890 g001 550
Figure 1. The flow chart of the approach.
Figure 1. The flow chart of the approach.
Sensors 17 01890 g001
Sensors 17 01890 g002 550
Figure 2. The light reflection and refraction for water hazards with particles and ground bottom.
Figure 2. The light reflection and refraction for water hazards with particles and ground bottom.
Sensors 17 01890 g002
Sensors 17 01890 g003 550
Figure 3. Reflection and refraction energy components from water as function of degree and direction of polarization. (a) Unpolarized light; (b) Light is three quarters polarized in perpendicular direction with the water plane; (c) Light is a quarter polarized in perpendicular direction with the water plane.
Figure 3. Reflection and refraction energy components from water as function of degree and direction of polarization. (a) Unpolarized light; (b) Light is three quarters polarized in perpendicular direction with the water plane; (c) Light is a quarter polarized in perpendicular direction with the water plane.
Sensors 17 01890 g003
Sensors 17 01890 g004 550
Figure 4. Traversable area detection in different scenarios.
Figure 4. Traversable area detection in different scenarios.
Sensors 17 01890 g004
Sensors 17 01890 g005 550
Figure 5. Flow chart of the water hazard detection.
Figure 5. Flow chart of the water hazard detection.
Sensors 17 01890 g005
Sensors 17 01890 g006 550
Figure 6. Water hazard detection with traversable area detection. (a,d) Left color images; (b,e) Right color images; (c,f) Detection results with the pRGB-D sensor.
Figure 6. Water hazard detection with traversable area detection. (a,d) Left color images; (b,e) Right color images; (c,f) Detection results with the pRGB-D sensor.
Sensors 17 01890 g006
Sensors 17 01890 g007 550
Figure 7. Farthest traversable lines with or without water hazard detection. (a,d,g) Traversable area and water hazard detection results; (b,e,h) Farthest traversable lines correspond to the traversable area detection without water hazard detection; (c,f,i) Farthest traversable lines obtained with both traversable area detection and water hazard detection.
Figure 7. Farthest traversable lines with or without water hazard detection. (a,d,g) Traversable area and water hazard detection results; (b,e,h) Farthest traversable lines correspond to the traversable area detection without water hazard detection; (c,f,i) Farthest traversable lines obtained with both traversable area detection and water hazard detection.
Sensors 17 01890 g007
Sensors 17 01890 g008 550
Figure 8. Stereo sound feedback of the assisting system. Sounds of five directions of traversable area with or without water hazard detection are presented by five musical instruments in the virtual 3D space, including trumpet, piano, gong, violin and xylophone.
Figure 8. Stereo sound feedback of the assisting system. Sounds of five directions of traversable area with or without water hazard detection are presented by five musical instruments in the virtual 3D space, including trumpet, piano, gong, violin and xylophone.
Sensors 17 01890 g008
Sensors 17 01890 g009 550
Figure 9. The relationship between the average traversable distance and instrument loudness.
Figure 9. The relationship between the average traversable distance and instrument loudness.
Sensors 17 01890 g009
Sensors 17 01890 g010 550
Figure 10. The relationship between the proportion of overlapping areas and instrument pitch.
Figure 10. The relationship between the proportion of overlapping areas and instrument pitch.
Sensors 17 01890 g010
Sensors 17 01890 g011 550
Figure 11. Results of traversable area and water hazard detection in indoor and outdoor environments. (a,e,i,m) Left color images; (b,f,j,n) Depth images; (c,g,k,o) Traversable area and water hazard detection results denoted respectively by the green and magenta masks on the left color images; (d,h,l,p) Farthest traversable lines as stereo sound feedback interface.
Figure 11. Results of traversable area and water hazard detection in indoor and outdoor environments. (a,e,i,m) Left color images; (b,f,j,n) Depth images; (c,g,k,o) Traversable area and water hazard detection results denoted respectively by the green and magenta masks on the left color images; (d,h,l,p) Farthest traversable lines as stereo sound feedback interface.
Sensors 17 01890 g011
Sensors 17 01890 g012 550
Figure 12. The detection results of different approaches. (a–c) Results of the proposed approach; (d–f) Results of traversable area detection based on RANSAC plus filtering techniques; (g–i) Results of water hazard detection based on sky light effect modelling.
Figure 12. The detection results of different approaches. (a–c) Results of the proposed approach; (d–f) Results of traversable area detection based on RANSAC plus filtering techniques; (g–i) Results of water hazard detection based on sky light effect modelling.
Sensors 17 01890 g012
Sensors 17 01890 g013 550
Figure 13. Detection limitations. (a–c) Detection with defective depth information; (d–f) Small obstacles would be detected as traversable area.
Figure 13. Detection limitations. (a–c) Detection with defective depth information; (d–f) Small obstacles would be detected as traversable area.
Sensors 17 01890 g013
Sensors 17 01890 g014 550
Figure 14. (a,b) The voluntary participants wearing the assisting prototype including a pRGB-D sensor, a laptop in the backpack and a set of bone-conduction headphone; (c) The route around a building and an empty field in the real-world environment while the red labels represent the locations during navigation.
Figure 14. (a,b) The voluntary participants wearing the assisting prototype including a pRGB-D sensor, a laptop in the backpack and a set of bone-conduction headphone; (c) The route around a building and an empty field in the real-world environment while the red labels represent the locations during navigation.
Sensors 17 01890 g014
Table
Table 1. Quantitative analysis.
Table 1. Quantitative analysis.
Approaches T A D R % W A D R % E E %
Proposed approach without water area detection94.4%N/A60.6%
Proposed approach with water area detection94.4%89.2%7.3%
RANSAC plus filtering techniques [27]79.6%N/A41.5%
RANSAC plus expanding techniques [29]93.8%N/A65.2%
3D tracking of water areas [58]N/A86.5%N/A
Table
Table 2. Average time, number of collisions and number of times stepping in water areas in two conditions: the stereo sound interface transferred to the participants is generated according to the traversable area detection result with/without water hazard detection.
Table 2. Average time, number of collisions and number of times stepping in water areas in two conditions: the stereo sound interface transferred to the participants is generated according to the traversable area detection result with/without water hazard detection.
Transferred Detection ResultAverage Time to CompleteAverage Number of Collisions into ObstaclesAverage Number of Times Stepping in Water Areas
traversable area detection24 min 32 s3.030.6
traversable area and water hazard detection26 min 11 s2.62.8
Table
Table 3. Questionaire. After the field test, ten participants were asked two simple questions.
Table 3. Questionaire. After the field test, ten participants were asked two simple questions.
UserEasy to Wear?Could You Use the Stereo Sound Feedback to Find Traversable Directions?
User 1YesYes
User 2YesYes
User 3NoYes
User 4YesYes
User 5YesYes
User 6YesYes
User 7NoYes
User 8NoYes
User 9YesYes
User 10YesYes

Share and Cite

MDPI and ACS Style

Yang, K.; Wang, K.; Cheng, R.; Hu, W.; Huang, X.; Bai, J. Detecting Traversable Area and Water Hazards for the Visually Impaired with a pRGB-D Sensor. Sensors 2017, 17, 1890. https://doi.org/10.3390/s17081890

AMA Style

Yang K, Wang K, Cheng R, Hu W, Huang X, Bai J. Detecting Traversable Area and Water Hazards for the Visually Impaired with a pRGB-D Sensor. Sensors. 2017; 17(8):1890. https://doi.org/10.3390/s17081890

Chicago/Turabian Style

Yang, Kailun, Kaiwei Wang, Ruiqi Cheng, Weijian Hu, Xiao Huang, and Jian Bai. 2017. "Detecting Traversable Area and Water Hazards for the Visually Impaired with a pRGB-D Sensor" Sensors 17, no. 8: 1890. https://doi.org/10.3390/s17081890

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop