KR102134902B1 - Frameworking method for violence detection using spatiotemporal characteristic analysis of shading image based on deep learning - Google Patents

Abstract

The present invention relates to a framework for detecting violence using spatial and temporal characteristics analysis of deep learning-based shadow images, and to improve the ability and accuracy of detecting violence in images.
To this end, the present invention is a violent detection framework that detects a violent characteristic of an image by detecting a feature point of violence in an input image composed of video frames provided from a video camera or a video file. Step 1, extracting the 2D-based Y-frame black and white image by excluding the red (R), green (G), and blue (B) from each separated frame image, and the extracted 2D-based Y frame monochrome image A third step of sequentially accumulating a large number of 3D environments and converting them into Y-frame black and white images in a 3D environment, and extracting and accumulating frames of equal layers among the Y-frame monochrome images in the converted 3D environment to perform image convolution. Including the 4th step of deriving the desired detection scene using 3*3*3 filters, network-weighted and time-space-optimized video is created and applied to the algorithm to apply the feature points of violence to specific layers in the video convolution process. By continuously remembering and re-learning, it improves the violent detection ability and accuracy of the image, enables analysis regardless of the length of the analysis frame, and enables analysis of continuous behavior.

Description

Framework for violent detection using spatiotemporal characteristic analysis of shading image based on deep learning}

The present invention relates to a method for detecting and using violent detection framework using spatial and temporal characteristics analysis of deep learning-based shaded images, and more specifically, to detect the violence of an image by improving the part that easily loses the characteristic points that have passed when performing violent image convolution. A dip that improves ability and accuracy, enables analysis regardless of the length of the analysis frame with the use of a small filter, and enables learning of continuous frames by moving the time axis of the filter, enabling analysis of continuous behavior. It relates to a method of detecting and using violent detection framework by analyzing the spatial and temporal characteristics of a running-based shadow image.

In general, when violence, assault, or kidnapping incidents occur in residential areas, buildings, roads, or public facilities, if there are no people around or reports are not made due to indifference, to determine the cause or extent of the occurrence Since it is difficult to collect information, a number of CCTV screens are gathered by installing a number of CCTVs, each of which can be used to shoot images in rainy areas, dark alleys, remote areas, etc. The monitored screen is being monitored.

However, there are dozens of screens for monitoring CCTVs in control centers, etc., but only a few monitor them to confirm this, and in addition, incidents such as violence, assault, and kidnapping incidents and accidents occur in an instant or in a relatively short time. There was a disadvantage that it was not easy to check through the monitoring screen, and in order to compensate for this, a violence detection system through image analysis has recently been developed.

In this regard, existing violent detection frameworks include MoSIFT+HIK (Violence detection in video using computer vision techniques), VIF (Violent flows), MoSIFT+KDE+Sparse Coding (Violent video detection based on mosift feature and sparse coding), Gracia et al (Fast fight detection), Substantial Derivative (Violence detection in crowded scenes using substantial derivative), Bilinski et al (Human violence recognition and detection in surveillance videos), MoIWLD (Discriminative dictionary learning with motion weber local descriptor for violence detection) , ViF+OViF (Violence detection using oriented violent flows), and Three streams + Multi-stream deep networks for person to person violence detection in videos (LSTM).

(1) MoSIFT+HIK: Violence detection in video using computer vision techniques

A. Method: As illustrated in FIG. 1A, it shows image characteristics based on a local feature and a BoW capable of spatio-temporal analysis, and uses SVM (Support Vector Machine) to determine whether or not violence, Space-time interest points (STIP): Since the results of Harris corner detection are analyzed spatiotemporally, the spatiotemporal changes of corner points can be analyzed.

B. Motion SIFT (MoSIFT): As illustrated in FIG. 1B, standard scale-invariant feature transform (SIFT) + optical flow based local motion: combining optical flow-based information with SIFT-based local feature characteristics to determine local feature Change characteristics can be analyzed.

In SIFT, most corners are extracted as points of interest, and the area around the corners is represented by descriptors (histogram of oriented gradients).

C. Bag-of-Words (BOW): As illustrated in FIG. 1C, this is a method of explaining the characteristics of an image with a histogram of a visual word (local descriptor using a combination of specific features), and learning the features based on the visual word It is used as information for classification.

(2) VIF: Violent flows: real-time detection of violent crowd behavior.

A. Method: The change pattern of optical flow vector magnitudes is divided into violence and non-violence using SVM.

B. ViF: Expresses the change pattern of optical flow magnitude over time, and does not consider the value of magnitude itself.

C. Classification: ViF and ViF words are used to display images, and SVM is used to determine whether or not there is violence.

(3) MoSIFT+KDE+Sparse Coding: Violent video detection based on mosift feature and sparse coding

A. Method: As illustrated in FIG. 1D, MoSIFT is selected based on KDE, and feature vectors are generated through sparse coding to determine whether or not there is violence.

B. KDE (Kernel Density Estimation): This method solves the problem of discontinuity in the distribution of histograms and distribution change according to the size and range of bins, generates a kernel function for each observed data, and adds all kernels to the entire data Express the distribution.

(4) Gracia et al: Fast fight detection

A. Method: As illustrated in FIG. 1E, the motion blob using the difference between frames is analyzed to distinguish violence from non-violence.

B. You can analyze the difference between global motion and local motion by analyzing the shape and position of the motion blob (between blobs).

(5) Substantial Derivative: Violence detection in crowded scenes using substantial derivative.

A. Method: As illustrated in FIG. 1F, spatiotemporal characteristics of the optical flow between images are extracted (substantial derivative), and this is expressed as BoW to determine whether there is violence.

(6) Bilinski et al. : Human violence recognition and detection in surveillance videos.

A. Method: Detect violence by reflecting spatiotemporal information in the improved fisher filter.

(7) MoIWLD: Discriminative dictionary learning with motion weber local descriptor for violence detection

(8) ViF+OViF: Violence detection using oriented violent flows

A. Method: As illustrated in FIG. 1G, the image is represented and the violence is detected using OViF applying the concept of ViF to the optical flow direction.

(9) Three streams + LSTM: Multi-stream deep networks for person to person violence detection in videos

A. Method: As illustrated in FIG. 1H, by analyzing an existing single person's behavior (eg, walking, stretching an arm), it is possible to analyze a complex shape when an assault is issued, and to solve this CNN is used to learn the image of people and people to detect violence.

However, in the conventional techniques as described above, since it is common for at least two or more people to get entangled and have complicated movements due to the nature of violence, it is not easy to detect violence in such entangled images. Since there is no detection method considering the parallax difference of the behavior among the methods of, the detection system that does not consider the parallax difference of the behavior has a problem that the performance must be deteriorated.

KR 10-1541272 B1 2015.07.28. Enrollment KR 10-1552344 B1 2015.09.04. Enrollment KR 10-1651410 B1 2016.08.22. Enrollment

Therefore, the present invention was devised to solve the above problems, and a technical problem to be solved by the present invention is to extract a luminance component (Y) image, which is a black and white shaded image excluding color difference components (U, V) from a real-time input image. After that, it makes the network lightweight and optimized for time space, applies it to the algorithm, and continuously remembers and re-learns the feature points of violence on a specific layer during the video convolution process. By improving the discarded part, it is possible to improve the violent detection ability and accuracy of the image, and by using a small filter, analysis is possible regardless of the length of the analysis frame, and continuous frame learning is possible by moving the time axis of the filter. This is to provide a method for detecting and using violent detection framework by analyzing the spatial and temporal characteristics of deep learning-based shadow images that can analyze continuous behavior.

In one embodiment of the present invention for achieving the above object, in the input frame consisting of video frames provided from a video camera or a video file, in the violence detection frameworking method of detecting the violence of the image by detecting the characteristic point of violence, input Extracting the 2D (2D) based luminance component (Y) image by excluding the color difference components (U, V) from the image of one frame included in the image, sequentially 3D (3D) the 2D based Y image ) And extracting only equally spaced frames to obtain a 3D (3D) based Y image group, and performing 3D based Y image group image convolution, 3*3*3 It is a frame detection method for violent detection using a spatial and temporal character analysis of deep learning-based shadow images, including deriving a scene of violence detection using a filter.

According to the present invention, a real-time input image is extracted from a Y-frame black and white shaded image excluding red (R), green (G), and blue (B), and then applied to the algorithm by creating a network-weighted and time-space-optimized image and applying it to the algorithm. By using the re-learning method in the course of a solution, it is possible to continuously remember and re-learn the characteristic points of violence on a specific layer, thereby improving the part that easily loses the characteristic points that have passed when performing violent video convolution, thereby improving the ability to detect violence in the video. It is possible to provide the advantage of improving.

In addition, since the present invention uses a filter (3 x 3 x 3 kernel) smaller than the existing framework (3 x 3 x F), it is possible to provide an advantage that enables analysis regardless of the length of the analysis frame. do.

In addition, the present invention is able to provide an advantage that enables the analysis of the continuous behavior by enabling the learning of more consecutive frames than the existing framework through the time axis movement of the filter.

In addition, the present invention can provide an advantage of improving learning time and detection accuracy by applying residual networks to 3D convolution.

1A to 1H are reference views respectively illustrating various conventional violence detection frameworks.
FIG. 2 is a block diagram illustrating a method for detecting violent frames using spatial and temporal characteristics analysis of a deep learning-based shadow image according to the present invention.
3 is a detailed diagram illustrating a re-learning process during video convolution in the method for detecting violence in a frame according to the present invention.

Hereinafter, the configuration and operation of the violent detection frameworking method utilizing the spatial and temporal characteristics analysis of the deep learning-based shadow image according to the preferred embodiment of the present invention and the effect of the action will be described in detail with reference to the accompanying drawings.

The terms or words used in the specification and claims are not to be construed as being limited to ordinary or lexical meanings, and the inventor is based on the principle that the concept of terms can be properly defined in order to best describe his or her invention It should be interpreted in a sense and concept consistent with the technical idea of the present invention. Therefore, since the embodiments illustrated in the present specification and the configuration illustrated in the drawings are only the most preferred embodiments of the present invention, it is understood that there may be various equivalents and modifications that can replace them at the time of application. shall.

FIG. 2 is a block diagram illustrating a method for detecting violent detection frames using spatial and temporal characteristics analysis of a deep learning-based shadow image according to the present invention, and FIG. 3 is re-learning during video convolution in the method for detecting violent frames according to the present invention As a detailed diagram illustrating the process, as illustrated in the drawing, the violent detection frameworking method using the spatial and temporal characteristics analysis of the deep learning-based shadow image of the present invention includes a color difference component ( Extracting 2D (2D)-based luminance component (Y) images except U,V), sequentially accumulating 2D-based Y images in 3D (3D), and extract only equally spaced frames And obtaining a 3D (3D)-based Y image group, and performing image convolution on the 3D-based Y image group and deriving a desired detection scene using a 3*3*3 filter. The present invention can be installed and operated in the form of software or a platform in an image analysis unit of a violence detection system that detects a feature point of violence in an input image composed of video cameras or video frames given from a video file. In addition, in order to perform the present invention, when the input image is an RGB-based image, the image analysis unit may convert the input image from an RGB-based image to a YUV-based image.

The image analyzer first extracts a two-dimensional (2D)-based luminance component (Y) image by excluding the color difference components (U, V) from the image of one frame included in the input image.

delete

The image analysis unit sequentially accumulates the extracted 2D Y images in 3D (3D), and extracts only equally spaced frames to obtain a 3D (3D) based Y image group. . In the present invention, it is preferable to make a 3D space by stacking 30 frames based on 2D (2D).

The image analysis unit performs image convolution on the obtained 3D-based Y image group and derives a violence detection scene using a 3*3*3 filter.

It is preferable that such an image analysis unit is set to extract only frames of a uniform layer (for example, a predetermined layer) in a Y-frame black-and-white image in a 3D environment in the detection scene derivation step. It is preferably set to perform image convolution.

In addition, the image analysis unit remembers a feature point of violence that has passed in a frame of a predetermined uniform layer of the 10 layers extracted from the extracted scene in the detection scene derivation step, and replays it in the next frame of the frame in which the feature point of violence is stored. By learning and remembering the act of violence, it is possible to improve the problem of losing the characteristic point of violence lost during the video convolution process.

In addition, the image analysis unit performs a first convolution operation using a 3*3*3 filter on an image of the first 224*224 size in the step of deriving a detection scene, and the image of the 224*224 size calculated by the first convolution operation Transforms the first pooling into a 112*112-sized image, performs a second convolution operation on the first-pooled transformed 112*112-sized image, and performs primary re-learning. For a 112-sized image, a third convolution operation equal to the first convolution operation, a quadratic pooling transform with an image of 56*56 size, and a fourth convolution operation are performed. In the 2nd convolution operation, and in the 3rd and 4th convolution operation, the characteristic points of violence are respectively stored in the predetermined equal order layer, and re-learned in the next frame of the corresponding frame in which the characteristic of violence is stored. Therefore, use a re-learning method to remember the act of violence.

In addition, the image analysis unit detects the violence of the image using 8 3x3x3 kernels and 2 2x2x2 kernels in a 3D (3D) space in the detection scene derivation step, so that it is not limited by the length of the analysis frame. Without analysis, it is possible to improve the learning time and detection accuracy by enabling the analysis of continuous behavior by enabling more continuous frames to be learned compared to the existing framework by moving the time axis of the filter. do.

When explaining the effect of the violent detection frameworking method using the spatial and temporal characteristics analysis of the deep learning-based shadow image according to the present invention configured as described above are as follows.

The violent detection frameworking method of the present invention is mounted on the video analysis unit of the violence detection system, and the video analysis unit divides the input image input in real time from the video camera or video file into images per frame and separates the images from each frame. Excluding red (R), green (G), and blue (B), 2D-based Y-frame monochrome images are extracted.

Next, a plurality of the extracted 2D-based Y-frame black and white images (preferably 30 frames) are sequentially accumulated and converted into a Y-frame black and white image in a 3D environment.

By the way. Since the Y-frame black and white images of the 3D environment do not uniformly come in according to the input image device and the image transmission environment, therefore, in the present invention, a frame of the same layer in the Y-frame black-and-white image of the 3D environment (for example, a predetermined first order) As a layer, preferably, only 5 frames) are extracted, only the extracted frames are accumulated again, image convolution is performed using 10 layers, and a desired detection scene is derived using a 3*3*3 filter suitable for space-time. do.

In addition, when 30 convex YD black and white images based on 2D are sequentially accumulated to convolve a violent image to create a Y frame black and white image in a 3D environment, past feature points are easily lost. A method of reusing reused feature points by remembering each feature point of violence that has passed through at least two layers among the extracted 10 layers and re-learning the next layer of the corresponding layer that remembers the feature points of violence, so that the behavior of violence can be remembered. As a result, it is possible to improve the problem of losing the characteristic points of violence lost during the video convolution process.

At this time, the image convolution performs the first convergence operation on the first 224*224 sized image to first transform it into a 112*112 sized image, and the second transformed product on the first transformed 112*112 sized image. Convolution by performing a second transform to an image of size 56*56 by performing the calculation, and storing the characteristic points of violence in layers 3 and 8, respectively, in the first and second convolution operations, The characteristics of the remembered violence are re-learned on the next layer (layers 4 and 9), respectively, to remember the act of violence.

In addition, in the present invention, the violence of an image is detected using 8 3x3x3 kernels and 2 2x2x2 kernels in a 3D (3D) space, so analysis is possible regardless of the length of the analysis frame. In addition, it is possible to learn more continuous frames than the existing framework through the time axis movement of the filter, so it is possible to analyze the continuous behavior, thereby improving the learning time and detection accuracy.

Table 1 below illustrates the results of comparing the accuracy of the violence detection framework according to the present invention with the accuracy of a number of existing violence detection frameworks.

Hockey Dataset Movies Dataset Vioent-Flows Dataset 1)MoSIFT+HIK 90.9% 89.5% 2) VIF 82.9±0.14% 81.3±0.21% 3)MoSIFT+KDE+Sparse coding 94.3±1.68% 4) Gracia et al. 82.4±0.4% 97.8±0.4% 5)Substantial Derivative 96.89±0.2% 85.43±0.21% 6) Bilinski et al. 93.4% 99% 96.4% 7) MoIWLD 96.8±1.04% 93.19±0.1% 8)ViF+OViF 87.5±1.7% 88±2.45% 9)Three streams+LSTM The present invention 97.1±0.23% 99% 95.61±2.76%

According to the present invention, after extracting the Y-frame black and white shaded image except for the red (R), green (G), and blue (B), the real-time input image is applied to the algorithm by creating an image optimized for network weight reduction and time space. By using the re-learning method in the convolution process, it is possible to continuously remember and re-learn the characteristic points of violence on a specific layer, so it improves the part that easily loses the characteristic points passed when performing the violent image convolution and detects the violence of the image It is possible to improve the performance and also use a filter (3 x 3 x 3 kernel) that is smaller than the existing framework (3 x 3 x F), providing the advantage of enabling analysis regardless of the length of the analysis frame. It becomes possible.

In addition, according to the present invention, it is possible to learn more continuous frames than the existing framework through the time axis movement of the filter, so it is possible to analyze the continuous behavior and apply the residual network to 3D convolution. As a result, the learning time and detection accuracy can be improved.

As described above, although the present invention has been described by limited embodiments and drawings, the present invention is not limited to the above embodiments, and various modifications and modifications from these descriptions will be made by those skilled in the art to which the present invention pertains. Deformation is possible. Accordingly, the spirit of the present invention should be understood only by the scope of the claims set forth below, and all equivalent or equivalent modifications thereof will be said to fall within the scope of the spirit of the present invention.

11: Video per separated frame
12: 2D based Y frame black and white video
13: 3D environment Y frame black and white video

Claims (6)

In a violent detection system having a video analysis unit for detecting the violentity of an image by detecting a feature point of violence in an input image composed of video frames provided from a video camera or a video file, in the frame walking method of the violence detection performed by the image analysis unit, ,
Obtaining a two-dimensional (2D) based luminance component (Y) black and white Y image by excluding the color difference components (U, V) from the image of one frame included in the input image;
Sequentially accumulating the 2D-based Y images in 3D (3D), and extracting and accumulating only 30 frames of evenly spaced layers to obtain a 3D (3D)-based Y image group; And
And performing a video convolution on the 3D-based Y image group using a 3*3*3 filter and deriving a desired detection scene.
In the step of deriving the detection scene, the image analysis unit,
The first 224*224 sized image is first-composited using a 3*3*3 filter, and the first 224*224-scaled image is firstly transformed into a 112*112-size image. First re-learning by performing a second convolution operation on an image of the size of 112*112 that has been pooled and transformed by the first pooling. The third convolution operation, the second pooling transform to an image of 56*56 size, and the fourth convolution operation, and the first convolution operation, the second confluence operation, and the third convolution operation And in the fourth convolution operation, each feature point of violence is stored in a predetermined equal layer, and the next frame of the corresponding frame that remembers the feature of violence is re-learned to remember the act of violence,
Ten layers are extracted from Y-frame black and white images in a 3D environment and accumulated again to perform image convolution. Each of the extracted 10 layers is characterized by remembering the characteristic points of violence that have passed through at least two layers, and remembering the characteristic points of violence. Re-learning on the next layer of each layer uses a re-learning method to remember the act of violence,
Violence detection frame using spatial and temporal characteristics analysis of deep learning based shadow image, characterized by detecting the violence of the image using 8 3x3x3 kernels and 2 2x2x2 kernels in 3D space How to walk. delete delete delete delete delete

Images