Nothing Special   »   [go: up one dir, main page]
Next Article in Journal
Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features
Previous Article in Journal
A Smart Band for Automatic Supervision of Restrained Patients in a Hospital Environment
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 20
Issue 18
10.3390/s20185210
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Parallel Three-Branch Correlation Filters for Complex Marine Environmental Object Tracking Based on a Confidence Mechanism

by
Yihong Zhang
*,
Shuai Li
,
Demin Li
,
Wuneng Zhou
,
Yijin Yang
,
Xiaodong Lin
and
Shigao Jiang
College of Information Science and Technology, Engineering Research Center of Digitized Textile & Fashion Technology, Ministry of Education, DongHua University, Shanghai 201620, China
*
Author to whom correspondence should be addressed.
Sensors 2020, 20(18), 5210; https://doi.org/10.3390/s20185210
Submission received: 15 August 2020 / Revised: 4 September 2020 / Accepted: 10 September 2020 / Published: 12 September 2020
(This article belongs to the Section Remote Sensors)
Browse Figures
Figure 1
<p>Comparison of the peak distribution of the PCF tracker and the proposed tracker when occurring occlusion. The red one is the proposed tracker and the green one is the baseline tracker PCF. There are three status: (<b>a</b>) The object has not affected by interence; (<b>b</b>) The object has affected by interence; (<b>c</b>) The object gradually breaks away from the interence.</p> "> Figure 2
<p>Comparison results of the proposed tracker and the baseline tracker PCF in the complex marine environment.</p> "> Figure 3
<p>Qualitative experimental results from marine environmental videos. Five different colors represent representative five trackers, respectively.</p> "> Figure 4
<p>Comparison experimental results of the proposed tracker and the baseline tracker PCF.</p> "> Figure 5
<p>Experimental results show that the proposed tracker performs excellently, particularly in three attributes: (<b>a</b>) The PP of the OPE on attribute BC; (<b>b</b>) The SP of the OPE on attribute BC; (<b>c</b>) The PP of the OPE on attribute MB; (<b>d</b>) The SP of the OPE on attribute MB; (<b>e</b>) The PP of the OPE on attribute OV; (<b>f</b>) The SP of the OPE on attribute OV.</p> "> Figure 6
<p>Experimental results for the representative ten trackers represented by ten different color curves: (<b>a</b>) The PP of the OPE on all sequences; (<b>b</b>) The SP of the OPE on all sequence.</p> "> Figure 7
<p>Qualitative experimental results from four representative videos. Ten different colors represent representative ten trackers, respectively.</p> "> Figure 7 Cont.
<p>Qualitative experimental results from four representative videos. Ten different colors represent representative ten trackers, respectively.</p> "> Figure 8
<p>Failure case of proposed tracker on complex marine environmental objects. The results for representative five trackers are expressed in different colors.</p> ">
Versions Notes

Abstract

:
Marine object tracking is critical for search and rescue activities in the complex marine environment. However, the complex marine environment poses a huge challenge to the effect of tracking, such as the variability of light, the impact of sea waves, the occlusion of other ships, etc. Under these complex marine environmental factors, how to design an efficient dynamic visual tracker to make the results accurate, real time and robust is particularly important. The parallel three-branch correlation filters for complex marine environmental object tracking based on a confidence mechanism is proposed by us. The proposed tracker first detects the appearance change and position change of the object by constructing parallel three-branch correlation filters, which enhances the robustness of the correlation filter model. Through the weighted fusion of response maps, the center position of the object is accurately located. Secondly, the Gaussian-triangle joint distribution is used to replace the original Gaussian distribution in the training phase. Finally, a verification mechanism of confidence metric is embedded in the filter update section to analyze the tracking effect of the current frame, and to update the filter sample from verification result. Thus, a more accurate correlation filter is trained to prevent model drift and achieve a good tracking effect. We found that the effect of various interferences on the filter is effectively reduced by comparing with other trackers. The experiments prove that the proposed tracker can play an outstanding role in the complex marine environment.

1. Introduction

Due to the complexity of the marine environment, it is meaningful to study the tracking of complex marine environmental objects [1]. Owing to changes in appearance, changes in lighting, fast motion, blur and partial occlusion, it is difficult for traditional visual object trackers to achieve accurate and robust tracking of complex marine environmental objects [2]. Therefore, based on the in-depth analysis of the visual object tracking technology, this passage proposes a series of strategies to improve the tracking performance for complex marine environmental objects.
Correlation filtering is a key point of object tracking research, which originated from signal processing [3]. Given the object in the first frame, perform feature extraction on the object and analyze the region of interest. Then, find similar features and regions of interest in subsequent images and make predictions for the position of the next frame. We take the coordinates corresponding to maximum peak as the location of the object. Following this, the earliest tracker, MOSSE [4], was proposed, which exploited the most simple tracking method of related filtering ideas. Subsequently, there were many related improvements based on MOSSE, such as CSK [5], KCF [6], etc. According to introducing nuclear method, they have achieved good results, especially the KCF using the cyclic matrix calculation makes the tracking speed amazing. Subsequently, the DSST [7] tracker innovatively handles scale changes. Two independent correlation filters, the translation filter and scale filter, are used for object positioning and scale evaluation. Due to the problem that all the above-mentioned methods are influenced by the boundary effect. The SRDCF [8] tracker came into being. The tracker takes the image signal as large as possible, retains more real information of the object, and then punishes samples that are farther from the center of the object through a spatial weight coefficient.
Then the ECO_HC [9] tracker was proposed at the top computer vision conference CVPR2017. This method uses efficient convolution operations to extract the feature matrix and performs multi-feature fusion to obtain the feature map. Correlate the response spectrum and analyze it to get the position of the object. The ECO_HC tracker effectively improves the problems of low filter efficiency and the overfitting of related filters.
However, the ECO_HC tracker also has its shortcomings. When the object occurring deformation, light changes, or self-rotation, it is no reason to renew the relevant model with the same learning rate. In addition, the use of a smooth Gaussian distribution as the expected output will make the positioning inaccurate [10]. Subsequently, the PCF [11] tracker came into being. Compared with the ECO_HC tracker on the video sequences, precision accuracy and overlap accuracy of the PCF tracker are better than the former. When the object rotates significantly, the light changes and deforms and the PCF tracker can successfully track the object. However, the above trackers do not effectively track the object in the complex marine environment specifically. So as to promote the precision of tracking performance in the complex marine environment, we put forward parallel three-branch correlation filters for tracking the object, which can efficiently improve the robustness of object tracking [12]. Aiming at the problem that parallel three-branch correlation filters lack effective supervision mechanism, this paper proposes a confidence mechanism to analyze the distribution of related responses to verify whether it is reliable [13], and supervise the update of the sample model with inspection results.
The contributions of this paper are as follows:
First, target samples with three different weights are respectively trained with three parallel correlation filters and, where the values of the weights are determined by the learning rate. Three different learning rates are used to renew the model and perform weighted fusion to effectively improve the robustness of the tracker to overcome bad conditions in the marine environment.
Second, the confidence metric proposed in this paper judges whether to update the filter, that is, to supervise the update of the sample model. Confidence Response ( C R ) aims to evaluate the numerical difference of the largest n responses on the relevant response graph. When the C R value is above the threshold, the result is valid and the sample model can be updated, otherwise it will be not updated.
Third, we have done a lot of experiments in the ocean environment, which fully proves that the proposed tracker can still show excellent results when the background is blurred and the waves interfere. In addition, we also compared nine other representative trackers on the OTB-2015 dataset, showing that the proposed tracker can deal with object tracking efficiently in complex and changing scenarios.

2. Related Work

The PCF tracker is the baseline tracker of this article, and it has a good effect on object tracking in complex scenes. In the initial frame, the PCF tracker uses the shared generated samples and improved expected output to train two parallel correlation filters. Then, in the present frame, according to balancing the response maps of two filters, the PCF tracker is able to detect the location of the object with the Newton method [14]. Subsequently, the PCF tracker uses Gaussian Mixture Model to add a new sample or merge the two most similar samples to generate a new sample. Then, we use the new sample to renew the PCF1 and PCF2 at different learning rates.
PCF tracker uses object samples f 1 k l , f 2 k l , … f m k l with different weights w k p t to train two parallel correlation filters, h 1 t l and h 2 t l , respectively. The value of the weights w 1 k p t and w 2 k p t are changed according to learning rate. The objective functions of PCF1 and PCF2 are shown in Equations (1) and (2), respectively:
ε ( h 1 t ) = ∑ k = 1 n ‖ ∑ l = 1 d h 1 l ∗ ∑ p = 1 m ( w 1 k p t f k p l ) − g s k ‖ 2 + λ ∑ l = 1 d ‖ ω h 1 l ‖ 2
ε ( h 2 t ) = ∑ k = 1 n ‖ ∑ l = 1 d h 2 l ∗ ∑ p = 1 m ( w 2 k p t f k p l ) − g s k ‖ 2 + λ ∑ l = 1 d ‖ ω h 2 l ‖ 2
where p , k represents the amount of samples generated by training data and cyclic shift method, respectively, and g s k represents the improved joint-expected output distribution, which is a linear weighted fusion of Gaussian distribution and triangular distribution. The second term of the two equations are regularization terms, and ω represents spatial regularization parameters, which are used to prevent the model from overfitting and used to solve boundary effects. The λ is a regularization parameter used to prevent the overfitting of the correlation filtering model. The greater the λ, the greater the punishment. The common methods such as the L-curve and the GCV for solving ill-conditioned equations are compared with the analysis of examples. The calculation example shows that the Tikhonov regularization parameter optimization method is a feasible and effective method to get the optimal regularization parameter λ.
The fast Fourier transform can effectively transform Equations (1) and (2) into the frequency domain for efficient calculation. Since the regularization parameters ω break the closed solution of the function and the solution cannot be directly obtained for the objective equation, the conjugate gradient method can obtain the solution sum of the above two objective functions. Then, the output response score of the relevant filter is calculated by Equation (3), and finally the location of the object is obtained by the Newton iteration method.
y t , p o s = F − 1 { ∑ l = 1 d ( α H 1 t − 1 l Z t , p o s l + ( 1 − α ) H 2 t − 1 l Z t , p o s l ) }
where α is the fusion factor, and H 1 t − 1 l and H 2 t − 1 l are two filters in the frequency domain.
After object positioning is completed, the image features of the object are collected from the positioning area as new samples, and then the similarity between the new samples and the existing training set samples is calculated using the Gaussian mixture model. If new samples and the training samples have a particularly big feature difference, the new sample is added and the old sample added previous is removed to make sure that the number of samples remains unchanged. If the similarity between the new sample and the training sample is high, the two most similar samples are merged. The weight of each training sample is updated by Equation (4):
w k p t = ( 1 − η ) w k p t − 1
where η is the learning rate and t represents the t -th frame. It can be drawn from the above formula that different learning rates determine different sample weights.
To judge the scale of the object, the DCF tracker is used to extract the scale features under different scale factors [15], and then the relevant output response score of each scale factor is calculated using Equation (5). Finally, the scale corresponding of the max output response score is the scale of the object.
y t , s c a l e = F − 1 { ∑ l = 1 d A ¯ t − 1 , s c a l e l Z t , s c a l e l B t − 1 , s c a l e + λ }
where A ¯ t − 1 , s c a l e l and B t − 1 , s c a l e represent the scale correlation filter of the previous frame, and λ ( λ ≥ 0 ) is the regularization parameter.
The expected output of the traditional DCF method generally follows a two-dimensional Gaussian distribution [16], and the peak value is relatively smooth. However, because the DCF model uses a cyclic shift window to generate lots of synthetic training samples, the only real sample is in the center position, so the sharp expected output peak is more reasonable and the drift of the relevant filtering model can be avoided. In the expected output response graph, the areas farther away from the center represent negative sample labels, so the values of these locations should be set to tend to zero, while the triangular distribution graph has a smaller decline rate at the valley. Therefore, Gaussian distribution (as shown in Equation (6)) and triangular distribution (as shown in Equation (7)) are not suitable for DCF tracker. The joint expected output response distribution is used to replace the original Gaussian distribution graph. As shown in Equation (8), the Gaussian distribution and the triangular distribution are organically fused together by a multiplication operator [17], which can promote the robustness of the tracker.
g ( x , y ) = 1 2 π σ 2 e − ( x 2 + y 2 ) 2 σ 2 , ( − a 2 ≤ x ≤ a 2 , − b 2 ≤ y ≤ b 2 )
s ( x , y ) = ( 1 − 2 a | x | ) ( 1 − 2 b | y | ) , ( − a 2 ≤ x ≤ a 2 , − b 2 ≤ y ≤ b 2 )
m ( x , y ) = 1 2 π σ 2 ( 1 − 2 a | x | ) ( 1 − 2 b | y | ) e − ( x 2 + y 2 ) 2 σ 2 , ( − a 2 ≤ x ≤ a 2 , − b 2 ≤ y ≤ b 2 )
where σ represents the standard covariance of the distribution map, x , y represent the axis of the distribution map coordinates, a , b , respectively, the width and height of the distribution map, and the origin of the coordinates of the distribution map is ( a 2 , b 2 ) .

3. Proposed Approach

3.1. Parallel Three-Branch Correlation Filters

The DCF tracking algorithm can effectively train the discriminant related filter model in the frequency domain. The difference between the object and the background is effectively used. When the object is in a more complex scene, the algorithm has better tracking and discrimination capabilities. After the object is located, the target image feature is extracted, and the related filter model is updated online. The formula is as follows:
min H * = ‖ ∑ l = 1 d h l f 1 l − f 2 ‖ 2 + λ ∑ l = 1 d ‖ h l ‖ 2
where l is the dimension, and h is an improved correlation filter. λ is the adjustment parameter to control the influence of regularization [18]. Calculate the partial derivative after FFT and simplify to get:
H l = G ¯ F t l ∑ k = 1 d F ¯ t k F t k + λ
where G, F are after Fourier transform. Calculate the numerator and denominator, respectively, in the process of resolving:
y = F − 1 { ∑ l = 1 d A ¯ l Z l B   +   λ }
where F − 1 is the inverse Fourier transform, A, B represent the numerator and denominator, respectively, and z is the calculated feature map. The value of y is changed by changing the value of z. When y is the largest, it is the time when the correlation filter is most suitable.
A t l = ( 1 − η ) A t − 1 l +   η F t + 1 F t l
B t = ( 1 − η ) B t − 1 +   η ∑ k = 1 d F ¯ t k F t k
where η is the learning rate, and t represents the number of frames. From the first frame, neither the numerator nor the denominator has the previous item, and the initial numerator and denominator are well obtained.
Then, the tracker with parallel three-branch correlation filters is proposed by us [19]. The tracker uses object samples f 1 k l , f 2 k l , … f m k l with different weights w k p t to train three parallel correlation filters, h 1 t l , h 2 t l and h 3 t l , respectively. The value of the weights w 1 k p t , w 2 k p t and w 3 k p t are decided by the learning rate. The functions of PCF1, PCF2 and PCF3 can be shown as Equations (14), (15) and (16), respectively:
ε ( h 1 t ) = ∑ k = 1 n ‖ ∑ l = 1 d h 1 l ∗ ∑ p = 1 m ( w 1 k p t f k p l ) − g s k ‖ 2 + λ ∑ l = 1 d ‖ ω h 1 l ‖ 2
ε ( h 2 t ) = ∑ k = 1 n ‖ ∑ l = 1 d h 2 l ∗ ∑ p = 1 m ( w 2 k p t f k p l ) − g s k ‖ 2 + λ ∑ l = 1 d ‖ ω h 2 l ‖ 2
ε ( h 3 t ) = ∑ k = 1 n ‖ ∑ l = 1 d h 3 l ∗ ∑ p = 1 m ( w 3 k p t f k p l ) − g s k ‖ 2 + λ ∑ l = 1 d ‖ ω h 3 l ‖ 2
The conjugate gradient method is used to obtain the sum of the solutions of the above three objective functions [20]. The output response score of the relevant filter is calculated by Equation (17), and finally the position of the object in the present frame is obtained by the Newton iteration method. Three correlation filters use three different learning rates to determine the location of the object.
y t , p o s = F − 1 { ∑ l = 1 d ( α H 1 t − 1 l Z t , p o s l + β H 2 t − 1 l Z t , p o s l + ( 1 − α − β ) H 3 t − 1 l Z t , p o s l ) }
where α , β are fusion factors, and H 1 t − 1 l , H 2 t − 1 l and H 3 t − 1 l represent three parallel correlation filters. Three different learning rates are used to renew the model and perform weighted fusion [21] to effectively improve the performance of the tracker for complex marine environmental objects.
The weight of each training sample is updated by Equation (18):
w k p t = ( 1 − η ) w k p t − 1
Then, the relevant output response score of each scale factor is calculated using Equation (19).
y t , s c a l e = F − 1 { ∑ l = 1 d A ¯ t − 1 , s c a l e l Z t , s c a l e l B t − 1 , s c a l e + λ }

3.2. Confidence Machanism

Without an effective supervision mechanism during the tracking of the parallel three-branch correlation filters method, errors are easily generated when the object is interfered with by external environmental factors such as occlusion and illumination changes, and then the errors are brought into the update of the sample model. The accumulated error will cause the tracking frame to drift, which will eventually cause the object to be lost. Aiming at the problem that the method of parallel three-branch correlation filters lacks effective supervision mechanism, this paper proposes a verification mechanism [22] to analyze the reliability of current results by evaluating the distribution of related responses, and supervise the update of the sample model based on the inspection results. Before inputting the tracking object area into the relevant filter, a verification mechanism of confidence measurement is added to analyze and verify the distribution of the relevant response and tracking results, and judge whether to update the sample model according to the verification mechanism [23]. When the tracking result meets the set conditions, it is used to update the sample model, otherwise it is not adopted. The confidence metric in the update stage of the filter acts a pivotal part in tracking the quality of the tracking effect. The traditional tracking method compares the value of the related responses and takes the object area with the highest related response as the correct tracking result. In practical applications, this evaluation method has a large error. When there is an interference area with a high similarity, the object may be lost, but the value of its related response is still high, as shown in Figure 1. When the form of object changes, the relevant response value may be low, which will affect the judgment of the tracking effect.
In most cases, we usually take the largest point of the peak as the object location, which is not reliable [24]. This paper introduces a new confidence metric C R (Confidence Response) and embeds it in the filter update module to analyze the peak distribution of the relevant response and evaluate the tracking results. The solution process is as follows: select n related responses from the current frame image and arrange them in order from largest to smallest. Use R ( i ) to indicate the response value in the i-th position, p ( R ( i ) ) is the probability of the response value appearing. Find the expected μ of n response values:
μ = ∑ i = 1 n R ( i ) × p ( R ( i ) )
Then, introduce the concept of standard deviation to measure the response value R ( i ) . The available C R formula is as follows:
C R = ∑ i = 1 n ( R ( i ) − μ ) 2
C R aims to evaluate the numerical difference of the largest n responses on the relevant response graph [25]. The larger the C R value, the more dispersed the distribution between the interference area and the true area in the frame, and the more reliable the tracking effect; otherwise, the more concentrated the distribution between the interference area and the true area in the frame, the more unreliable the tracking effect.
As shown in Figure 1, the peak response is generally regarded as the location of the object [26]. However, in reality, when occurring occlusion, deformation, etc., the peak response does not represent the location of the actual object. As shown in Figure 1a, when there is no interference, the response peak is single, which can track the object well. As shown in Figure 1b, when there is interference, the response peak gradually increases. At this time, the highest peak is not the actual object position. As shown in Figure 1c, taking the highest peak position as the object position will cause tracking failure such as object drift [27]. Therefore, the confidence metric proposed in this paper judges whether to update the filter, that is, to supervise the update of the sample model. When the C R value is above the set threshold, the result is valid and the sample model can be updated, otherwise it is not updated. As shown in Figure 1c, when the C R value is lower than the threshold we set, the model update is stopped [28].
When the object is occluded, the baseline method PCF continuously updates the sample model, resulting in the accumulation of errors gradually and the occurrence of tracking drift [29]. The proposed method judges whether to update the sample model with the comparison result of the confidence metric C R value and the threshold value, and the update is no longer allowed when the object is occluded. Therefore, the improved sample update strategy can effectively solve the problem that the baseline PCF method easily causes tracking frame drift and object loss. According to the confidence mechanism, we can effectively improve the performance of the tracker.

3.3. Algorithm of the Proposed Tracker

As shown in Algorithm 1, we use three parallel correlation filters with different learning rates to make the tracking more robust. Firstly, if we use more branches, the speed will drop drastically, which makes tracking not time sensitive. Secondly, too many branches will cause overfitting. We combine the confidence mechanism and three-branch correlation filtering to be almost the same as the baseline tracker in terms of rate, but greatly improve the accuracy. The specific algorithm is shown below, which specifically explains the working principle step by step. In the initial video frame, the algorithm uses the prior information of the object to initialize three parallel correlation filters. Firstly, update the position by fusing the values of the three-branch parallel correlation filter, then use the scale filter to calculate the scale of the object, and finally extract the features of the object based on the new position and scale, and use the GMM to generate a new sample set to train the relevant filter model online. Finally, use the confidence mechanism to determine whether the model is updated. In subsequent frames, repeat the above steps.
Algorithm 1: Parallel Three-Branch Correlation Filters Tracking Algorithm Based on a Confidence Mechanism
Input:
  1: Image I t .
  2: Detected target position P t − 1 and scale S t − 1 .
Output:
  1: Detected target position P t and scale S t .
Loop:
  1: Initialize H 1 1 l , H 2 1 l , H 3 1 l and A 1 , s c a l e , B 1 , s c a l e in the first frame by Equations (14),(15),(16) and (12),(13).
  2: for t ∈ [ 2 , t f ] do.
  3:  Position detection:
  4:    Extract position features Z t , p o s from I t at P t − 1 and S t − 1 by a search region.
  5:    Compute three parallel correlation scores y 1 t , p o s , y 2 t , p o s , y 3 t , p o s .
  6:    Merge the three correlation scores to y t , p o s by Equation (17).
  7:    Set P t to the target position by Newton iterative method.
  8:  Scale detection:
  9:    Extract scale feature Z t , s c a l e from I t at P t − 1 and S t − 1 by a search region.
  10:   Compute correlation scores y t , s c a l e by Equation (19).
  11:   Set S t to the target scale that maximizes y t , s c a l e .
  12: Model update:
  13:   Extract new sample features F t , p o s and F t , s c a l e from I t and S t .
  14:   Generate new training set by the G M M method.
  15:   Compute response scores C R by Equation (21).
  16:   If C R > threshold, return step 2; else continue.
  17:   Update the model H 1 t l , H 2 t l , H 3 t l by the learning rate η t 1 , η t 2 , η t 3 .
  18:   Update the scale model A t , s c a l e , B t , s c a l e by the learning rate η 0 .
  19:  Return P t , S t .
  20: end for.

4. Results

4.1. Experimental Results and Analysis in the Complex Marine Environment

To prove that our tracker is superior to the baseline tracker, Figure 2 shows the comparison between the baseline tracker PCF and the tracker proposed in this paper in different complex marine environments. The experiments clearly show that the proposed tracker can effectively suppress the drift caused by tracking failure, which makes the tracking effect more robust.
For the purpose of verifying that the proposed tracker can track the complex marine environmental objects with better performance, we conduct a comparative experiment on the complex marine environmental objects to compare the proposed tracker with the current representative trackers, including the PCF tracker [11], ECO-HC tracker [9], Staple tracker [30] and CSK tracker [5].
Figure 3 shows that the proposed tracker behaves significantly better than that of several other trackers when tracking complex marine environmental objects with motion blur and rotation deformation; the proposed tracker can effectively learn changes in appearance and accurately locate the object. The following three video paragraphs are representative, highlighting the target tracking in the three most common marine environments, especially the tracking of small targets in the ocean, the interference of ocean waves, and the interference of other ships. We select a few representative frames of each data, so the selection is meaningful. We can see from several sets of pictures that when other trackers have already drifted (once they drift, they will continue to drift), our trackers can still perform well and robustly, and this is an intuitive experience. We have a lot of data later to verify the excellent performance of our tracker.

4.2. Comparison of the Proposed Tracker and the Baseline Tracker on OTB-2013

We have done experiments on the OTB-2013 dataset [31] to compare the benchmark tracker with the proposed tracker. Table 1 and Table 2 show the center position accuracy and overlap accuracy of the tracker under 11 different attributes, the attributes include scale variation of the object(SV), illumination variation of the environment (IV), rotation of the object out of plane (OPR), the object is occlusion (OCC), the background of the object appears cluttered (BC), the object deforms (DEF), the object produces motion blur (MB), the fast motion of the object (FM), the rotation of the object in the plane (IPR), the movement of the object out of view (OV), and the low resolution of the camera (LR). Compared with the baseline tracker, the tracking accuracy of the proposed tracker has been improved in nine attributes. The results are shown in Figure 4, which further verified that the tracker in this paper can performs better than baseline tracker in complex environment, especially for the following two attributes: the illumination variation of the environment and the background of the object appears cluttered. These attributes determine whether the proposed tracker can perform excellently in the complex marine environment.

4.3. Results and Analysis on the Dataset OTB-2015

The OTB-2015 dataset [32] is a benchmark dataset commonly used for object tracking experiments. Compared with the OTB-2013 dataset, the OTB-2015 dataset is a more challenging dataset that includes 100 videos, and also has eleven various attributes. On this very challenging dataset, we do the experiments with other nine trackers: ECO-HC [9], BACF [33], LMCF [34], DSST [7], LCT [35], SAMF [36], Staple [30], SRDCF [8] and KCF [6].
Figure 5 shows the tracking accuracy of each tracker under the three attributes of BC (contains 31 video sequences), MB (contains 29 video sequences) and OV (contains 14 videos). According to the figure, the proposed tracker achieved the highest tracking accuracy among the three attributes, the AP and the AUC under these three attributes were 88.6% and 83.7%, 85.3% and 81.0%, 85.0% and 73.4%. The experiments further verify that the proposed tracker can solve the problem of blurred background and rapid change.
Figure 6 shows the average accuracy of ten trackers on the OTB-2015 dataset. From the figure, we can see that, among these 10 trackers, the proposed tracker performs best with the AP of 87.8% and the AUC of 80.9% respectively. Besides, this section also evaluates the 10 trackers in 11 attributes. From the Table 3, the proposed tracker has the highest AP among 10 attributes. From Table 4, the proposed tracker achieved the highest AUC among all 10 attributes. In summary, the proposed tracker has significantly improved tracking accuracy, robustness and real-time performance. Besides, we show a qualitative experiment results on the OTB-2015 dataset with four representative video selected sequences. The results are shown in Figure 7. We select a few representative frames of each data so the selection is meaningful. The following video paragraphs are representative, highlighting the object tracking for changes in form, interference from obstructions, changes in light. These factors are particularly important in the marine environment. The proposed tracker has more excellent performance than other nine trackers in the perspective of position estimation and scale estimation, which can deal with changes in the complex marine environment and make the tracking effect outstanding for marine objects.

4.4. Failure Case

Although the proposed tracker can achieve outstanding results in most marine environments, it is easy to lose an object in cases of low resolution and large movements at the same time. We can see from the experimental results in Figure 8 that, in this case, many trackers, including the tracker proposed in this paper, have lost the target, resulting in continuous drift. As part of the next step, we will add a deep-learning neural network, such as the siamese network, to improve the tracker, and effectively improve the effect when adapting to scenarios in various complex environments. The siamese neural network breaks the limitation that the tracker based on the deep neural network cannot be real time. At the same time, the tracker based on the siamese neural network also has high robustness. By calculating the degree of correlation between the candidate domain to be detected and the target area, it is determined that the position with the highest similarity value is the predicted position of the object to be tracked. So, we will add the siamese network to our tracker to make the tracking effect more robust.

5. Conclusions

The baseline tracker PCF is easily interfered with by occlusion, strong light and other factors, which leads to tracking failure. In order to track the complex marine environmental objects efficiently, the parallel three-branch correlation filters based on a confidence mechanism is proposed. Three different parallel correlation filters with three different learning rates are applied to improve the robustness of the tracking effect. In addition, a new tracking performance evaluation index is proposed, and the measurement result is used as a reference index for the filter update. The proposed tracker effectively reduces the interference caused by marine environmental factors, which leads to excellent performance of object tracking. Through the experimental results, we can conclude that the tracker proposed in this paper has outstanding attributes in light changes, interference occlusion, fast motion and scale changes. It can effectively solve the target tracking loss caused by interference from waves and ships, light changes, and fast movement in the complex marine environment. Compared with other trackers, the proposed tracker shows excellent performance with accuracy and success rate. It shows good robustness under the conditions of background changes and its own non-rigid transformation, verifying that the proposed tracker is outstanding in the complex marine environment. However, when the object is blocked and there is a large displacement at the same time, it cannot perform well. Based on the end-to-end feature fusion framework of the siamese network, the framework can effectively fuse CNN features and hand-designed features, solve the problem of parameter learning in feature fusion, and improve the versatility of the target tracker. So, we will try to improve the tracker with the siamese network in the future.

Author Contributions

Y.Z. and S.L. conceived the main idea, designed the main algorithm and wrote the manuscript. S.L. designed the main experiments under the supervision of Y.Z., D.L., W.Z. and Y.Y. The experimental results were analyzed by Y.Z., S.L., W.Z., Y.Y., X.L., S.J. and D.L. provided suggestions for the proposed algorithm. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by [the Fundamental Research Funding for the Central Universities of Ministry of Education of China] grant number [18D110408], [the Special Project Funding for the Shanghai Municipal Commission of Economy and Information Civil-Military Inosculation Project “Big Data Management System of UAVs”] grant number [JMRH-2018-1042], and [the National Natural Science Foundation of China (NSFC)] grant number [18K10454]. The APC was funded by [the Fundamental Research Funding for the Central Universities of Ministry of Education of China].

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Chen, X.; Xu, X.; Yang, Y.; Wu, H.; Tang, J.; Zhao, J. Augmented Ship Tracking Under Occlusion Conditions From Maritime Surveillance Videos. IEEE Access 2020, 8, 42884–42897. [Google Scholar] [CrossRef]
  2. Singh, M.; Khare, S.; Kaushik, B.K. Performance Improvement of Electro-Optic Search and Track System for Maritime Surveillance. Def. Sci. J. 2020, 70, 66–71. [Google Scholar] [CrossRef]
  3. Kalal, Z.; Mikolajczyk, K.; Matas, J. Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 1409–1422. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Bolme, D.S.; Beveridge, J.R.; Draper, B.A.; Lui, Y.M. Visual Object Tracking Using Adaptive Correlation Filters. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010; pp. 2544–2550. [Google Scholar]
  5. Henriques, J.F.; Caseiro, R.; Martins, P.; Batista, J. Exploiting the Circulant Structure of Tracking-by-Detection with Kernels. Lect Notes Comput Sc 2012, 7575, 70715. [Google Scholar]
  6. Henriques, J.F.; Caseiro, R.; Martins, P.; Batista, J. High-Speed Tracking with Kernelized Correlation Filters. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 37, 583–596. [Google Scholar] [CrossRef] [Green Version]
  7. Danelljan, M.; Hger, G.; Khan, F.S.; Felsberg, M. Discriminative Scale Space Tracking. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1561–1575. [Google Scholar] [CrossRef] [Green Version]
  8. Danelljan, M.; Hager, G.; Khan, F.S.; Felsberg, M. Learning Spatially Regularized Correlation Filters for Visual Tracking. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 4310–4318. [Google Scholar]
  9. Danelljan, M.; Bhat, G.; Khan, F.S.; Felsberg, M. ECO: Efficient Convolution Operators for Tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 6931–6939. [Google Scholar]
  10. Aftab, W.; Hostettler, R.; De Freitas, A.; Arvaneh, M.; Mihaylova, L. Spatio-Temporal Gaussian Process Models for Extended and Group Object Tracking With Irregular Shapes. IEEE Trans. Veh. Technol. 2019, 68, 2137–2151. [Google Scholar] [CrossRef]
  11. Yang, Y.; Zhang, Y.; Li, D.; Wang, Z. Parallel Correlation Filters for Real-Time Visual Tracking. Sensors 2019, 19, 2362. [Google Scholar] [CrossRef] [Green Version]
  12. Lu, X.; Li, J.; He, Z.; Liu, W.; You, L. Visual object tracking via collaborative correlation filters. Signal Image Video Process. 2020, 14, 177–185. [Google Scholar] [CrossRef]
  13. Fang, Y.; Ko, S.; Jo, G.S. Robust visual tracking based on global-and-local search with confidence reliability estimation. Neurocomputing 2019, 367, 273–286. [Google Scholar] [CrossRef]
  14. Tjaden, H.; Schwanecke, U.; Schömer, E.; Cremers, D. A Region-Based Gauss-Newton Approach to Real-Time Monocular Multiple Object Tracking. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 41, 1797–1812. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Yuan, Y.; Chu, J.; Leng, L.; Miao, J.; Kim, B.-G. A scale-adaptive object-tracker with occlusion detection. Eurasip J. Image & Video Process. 2020, 1, 1–15. [Google Scholar]
  16. Guo, Y.; Li, Y.; Xue, A.; Tharmarasa, R.; Kirubarajan, T. Simultaneous tracking of a maneuvering ship and its wake using Gaussian processes. Signal Process. 2020, 172, 107547. [Google Scholar] [CrossRef]
  17. Zheng, H.; Tang, Y. A novel failure mode and effects analysis model using triangular distribution-based basic probability assignment in the evidence theory. IEEE Access 2020, 8, 66813–66827. [Google Scholar] [CrossRef]
  18. Zhang, Q. Using Wavelet Network in Nonparametric Estimation. IEEE Trans. Neural Netw. 1997, 8, 227–236. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  19. Mercorelli, P. Biorthogonal wavelet trees in the classification of embedded signal classes for intelligent sensors using machine learning applications. J. Frankl. Inst. 2007, 344, 813–829. [Google Scholar] [CrossRef]
  20. Mahdavi-Amiri, N.; Shaeiri, M. A conjugate gradient sampling method for nonsmooth optimization. A Q. J. Oper. Res. 2020, 18, 73–90. [Google Scholar] [CrossRef]
  21. Gao, J.; Wang, Q.; Xing, J.L.; Ling, H.B.; Hu, W.M.; Maybank, S. Tracking-by-Fusion via Gaussian Process Regression Extended to Transfer Learning. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 939–955. [Google Scholar] [CrossRef] [Green Version]
  22. Gao, Y.; Zhao, J.; Luo, J.; Zhou, H. Adaptive feature fusion with the confidence region of a response map as a correlation filter tracker. Opt. Precis. Eng. 2019, 27, 1178–1187. [Google Scholar]
  23. Vojir, T.; Noskova, J.; Matas, J. Robust Scale-Adaptive Mean-Shift for Tracking. Pattern Recognit. Lett. 2014, 49, 250–258. [Google Scholar] [CrossRef]
  24. Sun, H.; Chen, X.; Xiao, H. A deep object tracker with outline response map. CAAI Trans. Intell. Syst. 2019, 14, 725–732. [Google Scholar]
  25. Li, X.; Zhou, J.; Hou, J.; Zhao, L.; Tian, N. Research on improved moving object tracking method based on ECO-HC. J. Nanjing Univ. Nat. Sci. 2020, 56, 216–226. [Google Scholar]
  26. Oron, S.; Bar-Hillel, A.; Levi, D.; Avidan, S. Locally Orderless Tracking. Int. J. Comput. Vis. 2014, 111, 213–228. [Google Scholar] [CrossRef]
  27. Han, Y.; Deng, C.; Zhao, B.; Tao, D. State-Aware Anti-Drift Object Tracking. IEEE Trans Image Process. 2019, 28, 4075–4086. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Li, B.; Xie, W.; Zeng, W.; Liu, W. Learning to Update for Object Tracking With Recurrent Meta-Learner. IEEE Trans. Image Process. 2019, 28, 3624–3635. [Google Scholar] [CrossRef] [PubMed]
  29. Lin, Y.; Chen, J.; Lin, K. Integration of Texture and Depth Information for Robust Object Tracking. In Proceedings of the 2014 IEEE International Conference on Granular Computing (GrC), Noboribetsu, Japan, 22–24 October 2014; pp. 170–174. [Google Scholar]
  30. Bertinetto, L.; Valmadre, J.; Golodetz, S.; Miksik, O.; Torr, P.H. Staple: Complementary Learners for Real-Time Tracking. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 1401–1409. [Google Scholar]
  31. Wu, Y.; Lim, J.; Yang, M.H. Online Object Tracking: A Benchmark. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 2411–2418. [Google Scholar]
  32. Wu, Y.; Lim, J.; Yang, M.-H. Object Tracking Benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 1834–1848. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Galoogahi, H.; Fagg, A.; Lucey, S. Learning background-aware correlation filters for visual tracking. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 1144–1152. [Google Scholar]
  34. Wang, M.; Liu, Y.; Huang, Z. Large Margin Object Tracking with Circulant Feature Maps. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 4800–4808. [Google Scholar]
  35. Ma, C.; Yang, X.; Zhang, C.; Yang, M.H. Long-Term Correlation Tracking. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 5388–5396. [Google Scholar]
  36. Li, Y.; Zhu, J. A Scale Adaptive Kernel Correlation Filter Tracker with Feature Integration. In Proceedings of the IEEE European Conference on Computer Vision Workshops (ECCVW), Zurich, Switzerland, 6–12 September 2014; pp. 254–265. [Google Scholar]
Sensors 20 05210 g001 550
Figure 1. Comparison of the peak distribution of the PCF tracker and the proposed tracker when occurring occlusion. The red one is the proposed tracker and the green one is the baseline tracker PCF. There are three status: (a) The object has not affected by interence; (b) The object has affected by interence; (c) The object gradually breaks away from the interence.
Figure 1. Comparison of the peak distribution of the PCF tracker and the proposed tracker when occurring occlusion. The red one is the proposed tracker and the green one is the baseline tracker PCF. There are three status: (a) The object has not affected by interence; (b) The object has affected by interence; (c) The object gradually breaks away from the interence.
Sensors 20 05210 g001
Sensors 20 05210 g002 550
Figure 2. Comparison results of the proposed tracker and the baseline tracker PCF in the complex marine environment.
Figure 2. Comparison results of the proposed tracker and the baseline tracker PCF in the complex marine environment.
Sensors 20 05210 g002
Sensors 20 05210 g003 550
Figure 3. Qualitative experimental results from marine environmental videos. Five different colors represent representative five trackers, respectively.
Figure 3. Qualitative experimental results from marine environmental videos. Five different colors represent representative five trackers, respectively.
Sensors 20 05210 g003
Sensors 20 05210 g004 550
Figure 4. Comparison experimental results of the proposed tracker and the baseline tracker PCF.
Figure 4. Comparison experimental results of the proposed tracker and the baseline tracker PCF.
Sensors 20 05210 g004
Sensors 20 05210 g005 550
Figure 5. Experimental results show that the proposed tracker performs excellently, particularly in three attributes: (a) The PP of the OPE on attribute BC; (b) The SP of the OPE on attribute BC; (c) The PP of the OPE on attribute MB; (d) The SP of the OPE on attribute MB; (e) The PP of the OPE on attribute OV; (f) The SP of the OPE on attribute OV.
Figure 5. Experimental results show that the proposed tracker performs excellently, particularly in three attributes: (a) The PP of the OPE on attribute BC; (b) The SP of the OPE on attribute BC; (c) The PP of the OPE on attribute MB; (d) The SP of the OPE on attribute MB; (e) The PP of the OPE on attribute OV; (f) The SP of the OPE on attribute OV.
Sensors 20 05210 g005
Sensors 20 05210 g006 550
Figure 6. Experimental results for the representative ten trackers represented by ten different color curves: (a) The PP of the OPE on all sequences; (b) The SP of the OPE on all sequence.
Figure 6. Experimental results for the representative ten trackers represented by ten different color curves: (a) The PP of the OPE on all sequences; (b) The SP of the OPE on all sequence.
Sensors 20 05210 g006
Sensors 20 05210 g007a 550Sensors 20 05210 g007b 550
Figure 7. Qualitative experimental results from four representative videos. Ten different colors represent representative ten trackers, respectively.
Figure 7. Qualitative experimental results from four representative videos. Ten different colors represent representative ten trackers, respectively.
Sensors 20 05210 g007aSensors 20 05210 g007b
Sensors 20 05210 g008 550
Figure 8. Failure case of proposed tracker on complex marine environmental objects. The results for representative five trackers are expressed in different colors.
Figure 8. Failure case of proposed tracker on complex marine environmental objects. The results for representative five trackers are expressed in different colors.
Sensors 20 05210 g008
Table
Table 1. Central position accuracy and average accuracy of the proposed tracker and baseline tracker in 11 different attributes. Red represent the better performance.
Table 1. Central position accuracy and average accuracy of the proposed tracker and baseline tracker in 11 different attributes. Red represent the better performance.
TrackersSVIVOPROCCBCDEFMBFMIPROVLRAP
Baseline83.582.387.090.184.591.776.877.781.281.053.388.0
Ours85.984.988.692.287.891.679.681.483.380.153.989.3
Table
Table 2. Overlap accuracy and average accuracy of the proposed tracker and baseline tracker in 11 different attributes. Red represent the better performance.
Table 2. Overlap accuracy and average accuracy of the proposed tracker and baseline tracker in 11 different attributes. Red represent the better performance.
TrackersSVIVOPROCCBCDEFMBFMIPROVLRAUC
Baseline77.178.579.684.780.591.076.875.372.882.853.982.5
Ours77.779.479.985.381.690.776.875.673.281.955.382.9
Table
Table 3. Central position accuracy and average accuracy of representative ten trackers in 11 different attributes. Red, green, and blue, respectively, represent the top three trackers.
Table 3. Central position accuracy and average accuracy of representative ten trackers in 11 different attributes. Red, green, and blue, respectively, represent the top three trackers.
TrackersSVIVOPROCCBCDEFMBFMIPROVLRAP
Ours84.384.086.584.888.684.385.382.582.385.079.087.8
ECO-HC80.579.281.180.682.481.878.079.278.373.779.884.5
BACF76.780.377.973.080.176.474.178.779.274.874.181.6
SRDCF74.178.674.273.077.572.876.776.974.559.765.579.0
LMCF72.379.576.073.682.272.973.073.075.569.367.978.4
Staple71.578.773.072.176.674.370.769.777.066.163.177.9
LCT67.874.374.667.973.468.566.968.178.259.253.776.2
SAMF70.170.873.972.268.968.065.565.472.162.868.575.1
KCF63.572.467.763.271.361.960.162.170.150.156.069.6
DSST63.371.564.458.970.453.356.755.269.148.156.767.9
Table
Table 4. Overlap accuracy and average accuracy of representative ten trackers in 11 different attributes. Red, green, and blue, respectively, represent the top three trackers.
Table 4. Overlap accuracy and average accuracy of representative ten trackers in 11 different attributes. Red, green, and blue, respectively, represent the top three trackers.
TrackersSVIVOPROCCBCDEFMBFMIPROVLRAUC
Ours75.481.077.878.383.777.681.076.072.273.465.480.9
EOC-HC71.275.472.374.477.273.475.374.067.765.769.477.2
BACF69.878.070.969.276.769.173.575.671.168.966.376.8
SRDCF66.274.066.467.870.165.972.971.766.255.862.672.8
LMCF62.274.567.768.776.465.970.367.465.665.154.671.3
Staple61.072.164.667.270.967.266.163.867.356.049.170.4
LCT58.371.567.663.170.361.665.965.569.453.143.670.1
SAMF58.464.066.066.463.960.664.159.564.155.151.567.4
DSST52.564.955.153.161.347.955.151.758.945.744.260.0
KCF41.555.052.751.260.950.355.052.655.344.229.555.1

Share and Cite

MDPI and ACS Style

Zhang, Y.; Li, S.; Li, D.; Zhou, W.; Yang, Y.; Lin, X.; Jiang, S. Parallel Three-Branch Correlation Filters for Complex Marine Environmental Object Tracking Based on a Confidence Mechanism. Sensors 2020, 20, 5210. https://doi.org/10.3390/s20185210

AMA Style

Zhang Y, Li S, Li D, Zhou W, Yang Y, Lin X, Jiang S. Parallel Three-Branch Correlation Filters for Complex Marine Environmental Object Tracking Based on a Confidence Mechanism. Sensors. 2020; 20(18):5210. https://doi.org/10.3390/s20185210

Chicago/Turabian Style

Zhang, Yihong, Shuai Li, Demin Li, Wuneng Zhou, Yijin Yang, Xiaodong Lin, and Shigao Jiang. 2020. "Parallel Three-Branch Correlation Filters for Complex Marine Environmental Object Tracking Based on a Confidence Mechanism" Sensors 20, no. 18: 5210. https://doi.org/10.3390/s20185210

APA Style

Zhang, Y., Li, S., Li, D., Zhou, W., Yang, Y., Lin, X., & Jiang, S. (2020). Parallel Three-Branch Correlation Filters for Complex Marine Environmental Object Tracking Based on a Confidence Mechanism. Sensors, 20(18), 5210. https://doi.org/10.3390/s20185210

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop