KR20180070258A - Method for detecting and learning of objects simultaneous during vehicle driving - Google Patents

Abstract

The present invention relates to a method for simultaneously recognizing and learning an object in a driving situation, which is able to automatically recognize and learn an object only by repetitive driving by increasing a recognition rate of the object. The method comprises a first step of extracting first and second object candidate areas by an image processing unit; a second step of estimating, by a detection unit, an object from the first and second object candidate areas and comparing the same with a type of an actual object to estimate and store a probability value indicating consistency; and a third step of calculating a type and a probability value of the object and updating the first and second object candidate areas related to the object.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for simultaneous object recognition and learning in a driving situation,

The present invention relates to a simultaneous object recognition and learning method in a running situation, and more particularly, to a simultaneous object recognition and learning method in which the performance of a detector is continuously upgraded in a running situation by performing automatic learning while performing object detection from a video signal And a learning method.

In situations where there are many factors that require attention around the vehicle, the driver will have to slow down. In particular, when the vehicle rotates in a narrow alley or an intersection in the left or right direction, the driver is always nervous because of the possibility that an object approaching the vehicle in a region where the driver's visibility is not sufficiently secured may collide with the vehicle.

General object detection related studies extract the ground truth of an object by hand while offline and learn it and generate a detector. The object can be detected by using this learning device on-line.

In this regard, Korean Unexamined Patent Publication No. 10-2016-0121481 (object recognition system and object recognizing method) extracts left and right feature vectors from a stereo image generated from an object, and extracts left and right feature vectors common to the extracted left and right feature vectors Disclosed is an object recognition system for searching for a feature vector, comparing left and right feature vectors and robust feature vector information with information stored in a database to extract name information of the object to recognize the object, and an object recognition method thereof.

However, in the conventional object recognition system and the object recognition method, recognition and learning are separated and proceeded in off-line. Therefore, the processor operates only when the object is recognized, and the recognition rate of the object is low.

Korean Patent Registration No. 10-2016-0121481

An object of the present invention is to provide a method of automatically performing recognition and learning by integrating object feature extraction and recognizer learning.

Another object of the present invention is to provide a system for automatically recognizing and learning an object by only repeating driving by increasing the recognition rate of the object.

According to an aspect of the present invention, there is provided an object recognition and learning method for recognizing an object from a video signal photographed by a camera in a running situation, the object processing method comprising the steps of: applying an RPN (Region Proposal Network) Extracting a first object candidate region from which the object candidate region can be determined, and tracking the object from the stored video signal to extract a second object candidate region; Estimating an object from the first and second object candidate regions and comparing and comparing the object with a kind of an actual object, and estimating and storing a probability value indicating the degree of matching; And the backwating unit compares the object and probability values estimated by the detection unit with predetermined threshold values for each type of object, calculates the type and probability value of the object when the threshold value is greater than the threshold value, feeds back the type and probability value of the object to the image processing unit And a third step of updating the first and second object candidate regions based on the first object candidate region and the second object candidate region.

Preferably, the first and second object candidate regions associated with the object are updated while recognizing the type of the object in the running situation, thereby increasing the recognition rate of the object.

Preferably, the second object candidate region in the first step may be extracted using KLT (Kanade-Lucas-Tomasi) tracking within a predetermined time.

Preferably, the type of object may include a forward stop or moving object sensed in a running situation.

Preferably, the third step may include: calculating a label matching the object in the first and second object candidate regions; Calculating a label having a maximum probability value among values obtained by taking a softmax function in a label; And a process of feeding the difference between the ground truth label and the label measured in real time to the image processing unit when the label having the maximum probability value is equal to or greater than the threshold value and assuming that the label is the ground truth label.

According to the present invention having the above-described configuration, there is an advantage that performance of the object recognizer is improved through 'simultaneous autonomous learning' using data obtained through continuous driving of a vehicle.

Further, the present invention is advantageous in that the labeling cost is reduced because there is no need to label the object every frame for object learning of the video image.

FIG. 1 shows a simultaneous object recognition and learning method in a running situation in which simultaneous learning is improved compared to the prior art.
2 shows a flowchart according to an embodiment of the present invention.
FIG. 3 shows a data processing view of a simultaneous object recognition and learning method in a running situation according to an embodiment of the present invention.
4 illustrates a method of recognizing an image using a KLT (Kanade-Lucas-Tomasi) tracking result in an image processing unit according to an embodiment of the present invention.
5 shows a detailed configuration of the third step according to the embodiment of the present invention.
FIG. 6 shows an improved result screen for detecting an object by applying a learning method according to an embodiment of the present invention.
FIG. 7 shows the result of improving the object detection rate according to the embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the exemplary embodiments. Like reference numerals in the drawings denote members performing substantially the same function.

The objects and effects of the present invention can be understood or clarified naturally by the following description, and the purpose and effect of the present invention are not limited by the following description. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

FIG. 1 shows a simultaneous object recognition and learning method in a running situation in which simultaneous learning is improved compared to the prior art. Referring to FIG. 1, in the case of existing studies, only the category learning model is updated, and the basic phenotype is not learned. Conventionally, the object recognition system and the object recognition method have separate recognition and learning processes in the off-line, so that only when the object is recognized, the processor operates and the object recognition rate is low.

On the other hand, the present invention provides a method of automatically performing recognition and learning by integrating object feature extraction and recognizer learning through backwashing, and recognizing and learning an object automatically by repetitive running by increasing the recognition rate of the object System method can be provided.

2 shows a flowchart according to an embodiment of the present invention. Referring to FIG. 2, the present invention can be configured in three stages. A first step (S10) of extracting first and second object candidate regions by the image processing unit; Using CNN (Convolutional Neural Network) -LSTM (Long-Short Term Memory) using KLT (Kanade Lucas Tomasi) based tracking results, multiple frames are grouped into one track to perform recognition at one time, and estimation and storage of probability values A second step S30; And a third step (S50 and S70) of updating the first and second object candidate regions related to the object by feeding back the kind and probability value of the object.

In the first step S10, the image processing unit extracts a first object candidate region that can determine an object by applying an RPN (Region Proposal Network) technique to the video signal, tracks the object from the previously stored video signal, This is the process of extracting the region.

The present invention is characterized by recognizing the type of an object in a running situation and updating the first and second object candidate regions related to the object to increase the recognition rate of the object. Or moving objects.

The first object candidate region means an area for extracting an object from a current image signal and the second object candidate region means a learned candidate region detected before a predetermined time interval.

The second step S30 is a process in which the detecting unit estimates an object from the first and second object candidate regions and compares the object with a kind of an actual object to estimate and store a probability value indicating the degree of matching.

In the third step (S50 and S70), the object and the probability value estimated by the detection unit in the backward part are compared with predetermined threshold values for each kind of object, and the type and the probability value of the object are calculated when the threshold value is over, And updating the first and second object candidate regions related to the object by feeding back the probability values.

FIG. 3 shows a data processing view of a simultaneous object recognition and learning method in a running situation according to an embodiment of the present invention. Referring to FIG. 3, an object region is extracted in a first step. In the embodiment of the present invention, an object candidate region is extracted using RPN (Region Proposal Network).

The RPN (Region Proposal Network) is an area-based object recognition system that finds and suggests candidates for objects. It is possible to find candidate regions in the form that there is a candidate group that is estimated as an object that the user wants to find at a certain position of the image. In other words, it means the frontmost network which extracts candidate regions estimated as objects in the whole image. However, since there are too many candidate regions in this process, it is possible to reduce the candidate region by tracking the objects using the past images with respect to time t to identify objects at regular intervals. Finally, CNN can be used to identify the objects, and the confidence of the final result can be used to improve CNN network performance through additional learning of the network.

Therefore, the objects in front are searched based on the images recognized at time t, and the objects extracted from the learned data at time t-1 or earlier learned in the same travel section are tracked to detect the object type. Classifies the types of the detected objects, recognizes the extracted objects in the running situation, and determines whether the probability values of the extracted objects are equal to or larger than a threshold value of the probability of each learned object.

If the probability per object is greater than or equal to the threshold value, the threshold value of the existing learned object can be updated via network backwashing. In addition, it is possible to compare the detected objects, and to estimate the kinds and probabilities of the respective objects.

4 illustrates a method of recognizing an image using a KLT (Kanade-Lucas-Tomasi) tracking result in an image processing unit according to an embodiment of the present invention. Referring to FIG. 4, unlike the conventional Faster R-CNN, a plurality of frames are grouped into one track using a CNN (Convolutional Neural Network) -LSTM (Long-Short Term Memory) Can be performed.

FIG. 4 is a block diagram illustrating a method of combining a plurality of frames into a single track by using a CNN (Convolutional Neural Network) LSTM (Long-Short Term Memory).

CNN (Convolutional Neural Network) LSTM (Long-Short Term Memory) is a kind of neural network structure that allows continuous output of relatively correct output for input patterns. In order to solve the problem of classifying the input pattern into a specific group, in order to solve the problem of classifying the input pattern into a specific group by applying an effective pattern recognition method of a human being to a real computer, I use an algorithm that mimics the ability of learning I have. Through this algorithm, an artificial neural network can generate mapping between input pattern and output pattern, which expresses that artificial neural network has learning ability. Thus, the artificial neural network can generate a relatively correct output for input patterns that were not used for learning based on the learned results.

Referring to FIG. 4, in the embodiment of the present invention, the time t-1 is used. However, the present invention is not limited to this, and an object extracted from the data currently input or previous data do. It is possible to classify the types of detected objects and recognize the objects extracted from the running situation. Alternatively, each input value may be stored in a layer, and each layer may be connected or an existing layer may be upgraded through a regression neural network.

The video signal input at time t-1 is received by the LSTM method and recognized as a track. The probability value (0.9) of the object extracted from the video signal input at time t-1 is upgraded using the video signal input at time t can do. That is, it is possible to recognize an object as a truck in an object area ahead by repeating learning on the same object. Therefore, it is possible to perform recognition of each layer at a time by grouping a plurality of frames into one track by using CNN (Convolutional Neural Network) LSTM (Long-Short Term Memory).

As shown in the figure, the recognition rate can be improved through the information obtained by collecting the results within a predetermined time rather than the recognition at the present time through the LSTM.

5 shows a detailed configuration of the third step according to the embodiment of the present invention. Referring to FIG. 5, in the third step, the third step includes a step S701 of calculating a label matching an object in the first and second object candidate regions; (S702 and S703) of calculating a label having a maximum probability value among values obtained by taking a softmax function in a label; And a step (S704) of, when the label having the maximum probability value is equal to or greater than the threshold, the label as the ground truth label and feeding back the difference between the ground truth label and the label measured in real time to the image processing unit.

The process of calculating the label (S701) is a process of selecting an object having the highest probability value in the softmax output and indexing it. The process of comparing the indexed probability value p with the threshold value S702 is a process of comparing the indexed predicted value p with the threshold value Θ. If Θ <p, learning is performed (Back Propagation) Do Nothing) mode. (Pesudocode) for the forward and backward schemes for thresholds (i. E. Threshold ([theta]) in the case of Tables 1 and 2 below).

The present invention newly implements an unlabeled softmax loss layer. In contrast to the prior art, backwarding based on the difference between an existing ground truth label and a softmax prediction output is performed to obtain loss and the backward based on the difference. The present invention is also applicable to raw data Directional loss is calculated and the backward computation is performed.

In the case of conventional softmax loss, take a loss by the difference between the value obtained by taking the softmax function (probability x = [0,1]) and the ground truth label (probability x '= 1 in this case) Backwash.

)을 선별한 후, 각 레이블마다 독립적으로 정의되어 있는 기준점(임계값<

) 이상인 경우, 레이블 n을 ground truth label이라고 가정하고(

'=1), 그 차이만큼 loss를 취해 백워딩을 진행할 수 있다.On the contrary, the softmax function of the present invention is an unlabeled softmax loss, and the highest probability value obtained by taking a softmax function for the output of a label through a feature

), And then a reference point independently defined for each label (threshold value <

), The label n is assumed to be the ground truth label (

'= 1), the back-warding can proceed by taking the loss as much as the difference.

In the process of calculating the label (S701 to S703) and the process of feeding back to the image processing unit (S704), the object having high confidence in the object region is fed back to the CNN network to detect the object in the image, You can learn.

In most conventional deep learning and machine learning, candidate learning and object detection are performed by taking a classifier or network generated after learning separately in off-line. In this case, since the learning is performed offline, the performance of the classifier or network is not improved unless it is updated separately.

Therefore, the embodiment of the present invention has an advantage that the performance is automatically improved by simultaneously learning the object and learning simultaneously with a simple vehicle driving without off-line learning.

FIG. 6 shows an improved result screen for detecting an object by applying a learning method according to an embodiment of the present invention. Referring to FIG. 6, it can be seen that the result of iterative learning for the same image is improved when the present invention is applied.

In the case where the first and second object areas are continuously improved through repeated detection of the same image, it is confirmed that the recognition rate of the object is increased.

FIG. 7 shows the result of improving the object detection rate according to the embodiment of the present invention. 7 is a numerical representation of the visualized state of Fig. It can be seen that the detection rate is increased through continuous iterative learning on the same image. The x-axis is a step-by-step sequence on the detection screen, and tracking results on the base, and the probability that the y-axis increases as the number of times increases.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. will be. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by all changes or modifications derived from the scope of the appended claims and equivalents of the claims.

Claims (6)

An object recognition and learning method for recognizing an object from a video signal photographed by a camera in a running situation,
A first step of extracting a first object candidate region capable of judging the object from the video signal and extracting a second object candidate region by tracking the object from a previously stored video signal;
A second step of estimating the object from the first and second object candidate regions and comparing and comparing the object with a kind of an actual object to estimate and store a probability value indicating the degree of agreement; And
The backwashing unit compares the object and the probability value estimated by the detection unit with a predetermined threshold value for each type of object, calculates the type and probability value of the object when the threshold value is more than the threshold value, and transmits the type and probability value of the object to the image processing unit And a third step of updating the first and second object candidate regions associated with the object.
The method according to claim 1,
And recognizing the type of the object in the running situation and updating the first and second object candidate regions related to the object to increase the recognition rate of the object.
The method according to claim 1,
Wherein the second object candidate region in the first step includes:
(KLT) tracking within a predetermined period of time. The method for recognizing and learning a simultaneous object in a running situation is characterized in that it is extracted using KLT (Kanade-Lucas-Tomasi) tracking within a predetermined time.
The method according to claim 1,
The type of the object is,
And a forward stop or moving object sensed in the running situation.
The method according to claim 1,
In the third step,
Calculating a label matching the object in the first and second object candidate regions;
Calculating a label having a maximum probability value among values obtained by taking a softmax function in the label; And
When the label having the maximum probability value is greater than or equal to the threshold value, feeding back the difference between the ground truth label and the label measured in real time to the image processing unit Simultaneous object recognition and learning methods.
The method according to claim 1,
And extracting a first object candidate region from the video signal using a Region Proposal Network (RPN) technique.

Images