An Approach to Detect Vehicles in Multiple Climatic Conditions Using the Corner Point Approach

3.1 Steps

The proposed methodology comprises the following steps:

1. Detection of corner points in every 16th frame using the Harris corner detector.

2. Removal of corner points that do not belong to the vehicle.

3. Grouping of the extracted corner points.

4. Tracking corner points with the Lucas-Kanade algorithm to stabilize the result.

The flowchart of the proposed algorithm is shown in Figure 1.

Figure 1:

Flow Diagram of the Proposed Algorithm.

The frames are extracted from the video sequence captured from the moving camera. For every extracted frame, the proposed algorithm is applied to detect vehicles.

3.2 Corner Detection

The Harris corner detector, proposed by Harris and Stephen [2], is used to detect the corner points of the vehicle. The algorithm is invariant to rotation, translation, and scaling of an image.

A local window in an image is considered, and the change in intensity that results after shifting the window by a small amount in different directions is calculated. The window patch in the image is flat iff the change in intensity is small in all directions. The window patch in the image is an edge iff there is a small change in a direction and relatively large change in other directions. The window patch is a corner iff the change in intensity is high in all directions. A brief insight about the Harris corner detection algorithm is explained.

Harris and Stephen measured the variation of intensity at position I(x,y) by considering a small square window, e.g. (3*3, 5*5), in the image, and this window is shifted by one pixel in all the directions. The sum of the squared intensity difference of the corresponding position in two windows multiplied by a Gaussian window defines the intensity variations [refer to Eq. (1)]. Consider a 6*6 image as shown in Figure 2. Consider the red color 3*3 be the original window centered at location (x,y) with intensity I(x,y) (A5). The purple color 3*3 is the shifted window patch with the shift (u, v), i.e. one pixel along the diagonal with the intensity I(x+u, y+v).

Figure 2:

Calculations of Intensity Variation for a 3×3 Window.

The mathematical equation for intensity variation is given as

where (u,v) is the arbitrary shift in the x and y direction, respectively; w(x, y) is the weight of the Gaussian window at position (x, y); and I(x,y) and I(x+u, y+v) are the intensities of the image at position (x, y) and the moved window, respectively.

I(x+u, y+v) can be rewritten using Taylor series expansion as shown in Eq. (2):

I(x+u, y+v)=I(x, y)+∂I∂xu+∂I∂yv.

Substituting I(x+u, y+v) in Eq. (1):

Eu,v=∑x,yw(x,y)|I(x, y)+∂I∂xu+∂I∂yv−I(x, y)|2,

Eu,v=∑x,yw(x,y)|∂I∂xu+∂I∂yv|2,

Eu,v=∑x,yw(x,y)|Ixu+Iyv|2,

Eu,v=∑x,yw(x,y)(Ix2u2+Iy2v2+2uvIxIy).

The above equation can be rewritten in matrix form as below:

E(u, v)=[uv]M[uv],

where M is

M=∑x,yw(x, y)[Ix2IxIyIxIyIy2].

Here, I_x and I_y are image derivatives in the x and y directions, respectively. The response function is given by

R=det(M)−k(trace(M)).

The above equation will determine if a window contains an edge, a corner, or a flat region.

• where k is constant;

• where det(M)=λ₁λ₂;

• trace(M)=λ₁+λ₂;

• λ₁ and λ₂ are eigenvalues of M.

A few points need to be noted to define edge, corner, and flat region in the window:

1. If λ₁ and λ₂ are large, i.e. R is large, then the window has a corner.

2. If either of λ₁ or λ₂ is large, with R>0, then the window has an edge.

3. If R is small, then the window has a flat region.

The corner points are not extracted for every frame of the video but for every 16th frame in the video to reduce the time complexity.

3.3 Background Subtraction

Not all the corner points extracted from the image belong to the vehicles because of the background environment. Hence, the points that do not belong to vehicles need to be removed from the image. In order to remove these points, a background subtraction technique is performed. Frame subtraction is carried out to remove non-vehicle corner points. The frame difference is a pixel level difference between two adjacent frames. Only those points that belong to the vehicle are retained, as shown in Figure 4A.

3.4 Grouping of Objects

Once the extraction of the interest points from each object is done, grouping of these interest points is carried out. Assuming that the interest points extracted from each vehicle or object are highly dense in that area, i.e. closely packed together, we take advantage of these closely packed points and carry out grouping, as shown in Figure 4B. The grouping of points should be such that this group of points should correlate with that of the vehicle or object. In order to perform grouping, a minimum value of distance is considered between points, and this value is called the threshold value. If the distance between the points satisfies the threshold value, then the points are grouped as one. The distance between the points is calculated by the Euclidean formula [10]. Let P₁(x₁, y₁) and P₂(x₂, y₂) be the two points on the plane; then, the Euclidean distance measurement between these points is as shown below:

For every point extracted, its distance from its neighbor is calculated. If the distance between the point and its neighbor is less than the threshold, then these two points are grouped as one. In this way, grouping is done for all the extracted points. The threshold value is nothing but the minimum distance value set between the points.

3.5 Tracking

The interest points that are extracted and grouped may not be reliable from frame to frame. We may lose some interest points from frame to frame because of environmental conditions. Thus, the output will vary drastically. In order to solve this problem, the interest points that are extracted are tracked from frame to frame. Tracking avoids loss of interest points in the video sequence. Tracking of these points is done by using the Lucas-Kanade algorithm [5]. The Lucas-Kanade algorithm is applied to estimate the position of the motion of a pixel from one image to another.

4.1 Detection in Rainy Climate

The algorithm was tested with videos of different climate conditions. One of the weather conditions tested in this section is rainy. The proposed algorithm was tested on this video, and it worked exceptionally well as shown in Figure 5A. The frame rate of the video was 10 fps, and the detection rate was 15 fps for this video. All of the vehicles were detected in the video. A few of the resultant frames of this video are shown in Figure 5A and B. Only the main lane was considered for the vehicle detection.

4.2 Detection in Sunny Day with Low Resolution

The video was considered and tested on a sunny day with different scenes with a low-resolution camera. Undoubtedly, the algorithm worked well on the video with a much faster detection rate. The frame rate of the video was 30 fps, and the detection rate was 15 fps for this video. The corners of the vehicle were extracted accurately. Moreover, the corner points extracted from the vehicle were not densely packed when compared with a high-resolution video. All of the vehicles were detected in the video, and there were no missed detections. A few of the resultant frames of this video are shown in Figure 6A and B.

4.3 Detection in Sunny Day with High Resolution

The next video considered was a sunny day video taken with a high-resolution camera. Undoubtedly, the algorithm worked exceptionally well on the video. The corners of vehicles were extracted more accurately and the corner points extracted from the vehicle were more densely packed, making the grouping of the points and identification of the vehicles easy. The frame rate of the video was 30 fps, and the detection rate was 10 fps for this video. All of the vehicles were detected in the video, and there were no missed detections. A few of the resultant frames of this video are shown in Figure 7A and B.

4.4 Detection in Fog Video

To detect vehicles in the fog condition, an experiment was conducted by using a fog video. The proposed algorithm worked well on the fog video and detected all the vehicles present in the video, and the corner points extracted from the vehicle were not densely packed when compared with a low-resolution video. The frame rate of the video was 30 fps, and the detection rate was 15 fps for this video. All of the vehicles were detected in the video, and there were no missed detections. A few of the resultant frames of this video are shown in Figure 8A–D.

4.5 Rear-View Vehicle Detection with Moving Camera

The algorithm was tested with the rear view of the vehicle video. The proposed algorithm worked exceptionally well not only on the top-view videos but also on the rear view of the vehicle. It was tested on the rear view of the single vehicle. The corners of the vehicle were extracted accurately, and the corner points extracted from the vehicle were densely packed; however, the corner points of the surrounding environmental conditions were also extracted. This gave rise to noise; hence, thresholding should be effectively taken care of. The frame rate of the video was 30 fps, and the detection rate was 8 fps for this video. The vehicle was detected in the video, and there were no missed detections. A few of the resultant frames of this video are shown in Figure 9A–D.

Figure 9:

Results of Rear-View Video Sequence of Single-Vehicle Detection.

(A) Vehicle detection at curving point with moving camera capturing rear view of the vehicle. (B) Vehicle detection with no false detection at curving in presence of pedestrian. (C) Vehicle detection with moving camera. (D) Vehicle detection with moving camera in presence of noise (pedestrian and building).

4.6 Rear-View Multiple Vehicle Detection with a Moving Camera

The algorithm was tested with respect to detection and tracking of multiple vehicles in the rear-view video of the vehicle. The proposed algorithm not only worked exceptionally well on the single-vehicle video but also on the detection of the multiple vehicles in the rear view of the vehicle. The algorithm was first tested on the rear view of a single vehicle, and then on multiple vehicles. Working on the detection of vehicles would give rise to noise in the rear view of the video; hence, thresholding should be effectively taken care of. The frame rate of the video was 30 fps, and the detection rate was 8 fps for this video. The vehicle was detected in the video, and there were no missed detections. A few of the resultant frames of this video are shown in Figure 10A–D.

Figure 10:

Results of the Rear View of Multiple-Vehicle Detection.

(A) Multiple vehicle detection with high speed moving camera. (B) Multiple vehicle detection with moving camera in presence of noise (building).

4.7 Performance Evaluation

The experimental results of the proposed algorithm are shown in Figure 11. The tabular representation of all the results and their accuracy, time taken, view, and fps are discussed in Table 1. The parameters are defined to evaluate the detection results of the proposed algorithm [9, 11], as follows: correctness, completeness, and quality, as shown in Eqs. (4) to (6).

Figure 11:

Experimental Results.

(A) Complex background. (B) Complex background with camera shaking. (C) Complex background with more number of pedestrians at the intersection point. (D) Complex background at the intersection point. (E) Detection in a snowfield. (F) Detection in fog.

where TP is true positive: the number of correctly detected true vehicles; FP is false positive: the number of correctly detected false vehicles; and FN is false negative: the number of vehicles that are not detected.

The total overall correctness, completeness, and quality of vehicle detection were 99.32%, 94.63%, and 93.95% for the set of videos.

The computational complexity of the proposed algorithm is O(N²), where N is the dimension of the image. The time complexity varies with respect to the resolution of the videos. To reduce time, the corner detection algorithm is applied on every 16th frame and the Lucas-Kande algorithm is used to make the proposed algorithm stable by tracking the corner points to the first 15 frames. Tracking the corner points takes fewer milliseconds. For example, the average time complexity of the proposed algorithm for an image size of 320*240 is 92.36/61.2 ms, which means that the average time taken to detect vehicles by detecting the corner points of the vehicles is 92.36 ms. The average time taken to detect vehicles while tracking corners is 61.2 ms. Tracking is performed for the next 15 frames out of 16 frames. The average time taken for every video is shown in Table 1.

4.8 Comparative Studies

Different techniques have been used to detect and track the vehicles in aerial (top) and rear views. Table 2 explores the comparative analysis of four different techniques based on accuracy, time, and view. None of the techniques that have been implemented works on both top and rear views of the vehicles, as discussed. The proposed algorithm works on both top and rear views of the vehicles. The overall quality of the proposed algorithm for both the top and rear views of the vehicles in the video was 93.95% for the set of data. For the top-view videos with different climate conditions, the quality of the algorithm was 98% after testing it with 10 different videos. The graphical representation in Figure 12 shows the comparative study of different algorithms and their accuracy. We can see that the proposed system performs outstandingly well when compared to the accuracy of the top and rear views.

Figure 12:

Comparative Analysis Graph.