KR102122859B1 - Method for tracking multi target in traffic image-monitoring-system - Google Patents

Abstract

An embodiment of the present invention relates to a method for tracking multiple targets in a traffic image monitoring system. More specifically, according to one embodiment of the present invention, the method comprises: a step of collecting traffic environment information by each camera device installed in each of a plurality of different traffic monitoring areas during traffic monitoring; a step of extracting object feature metadata from the collected traffic environment information and extracting an object feature from a preset artificial intelligence (AI) format for extracting a feature value of an image patch in accordance with the object feature metadata; and a step of recognizing an object form the extracted object feature by a preset AI format for estimating the size and location of the whole object to track each object, thereby tracking multiple targets. Accordingly, the traffic image monitoring system systematically, effectively, and quickly tracks the multiple targets.

Description

Method for tracking multi target in traffic image-monitoring-system}

The present disclosure relates to the field of data processing technology, and relates to a technology for tracking a plurality of targets in real time in an image surveillance system such as a vehicle or a person on a road.

Unless otherwise indicated herein, the content described in this section is not prior art to the claims of this application and is not admitted to be prior art by inclusion in this section.

In general, a traffic signal control system, for example, is an important system that increases safety and communication efficiency by controlling movement priority of traffic flows that conflict at intersections. However, erroneous status and signal operation deteriorate communication efficiency, increasing congestion and inducing excessive driving patterns of the driver, which is a trigger for danger.

Currently, traffic signal control systems have been established in 21 metropolitan and small and medium-sized cities, including Seoul, but due to the limitations in the management and operation of the loop detector, which is a traffic information collection system, most of the intersections are operated in TOD. have.

In addition, the fixed type plans the signaling time based on the approximate traffic volume of the corresponding intersection, and the TOD method is based on the traffic volume identified by conducting a total survey of the traffic hours during the commuting time and traffic during the daytime and early morning hours when traffic volume is rapidly changing. Signal time is calculated.

In addition, a fixed or TOD signaling method would be advantageous for intersections where traffic patterns are kept constant in terms of maintenance, but the traffic signal pattern is changed due to environmental factors such as changes in the operation method of roads in the city. It may not be able to respond to it and thus operate inefficiently. To this end, it is not possible to monitor traffic at intersections or to inspect them every time.

On the other hand, in recent years, an illegal parking control device, a vehicle speed monitoring device, a vehicle entry/exit system of a building, and a CCTV security system have been used in vehicle number recognition systems. The vehicle number recognition system includes an information processing unit that recognizes a vehicle number by analyzing a camera, which is a photographing apparatus, and an image photographed through the camera. The camera can be installed either stationary or mobile.

Such a license plate is a metal or plastic plate attached to a vehicle for identification purposes. The vehicle license plate has letters, numbers, or a combination thereof, and the shape of the vehicle license plate is determined policy by country or administrative region.

Conventional vehicle license plate detection methods include character-based detection, edge-based detection, color-based detection, and texture-based detection. The character-based detection method has a low processing speed and requires setting a character area, the edge-based detection method is dependent on empirical variables (e.g., aspect ratio, etc.), and the color-based detection method is sensitive to changes in lighting or shadows, The texture-based detection method has a problem that processing speed is slow.

In addition, vehicle license plate recognition technology is a core technology in the field of intelligent traffic systems (ITS) used in traffic control systems, parking control systems, and ADAS systems. Recently, through the deep learning (Deep Learning) technology, the license plate recognition function has been increased, and it is effectively operating in an embedded environment, so it is possible to seek diversification of the license plate recognition technology.

On the other hand, as reported in the recent news, when a fire occurs, there are problems such as secondary damage due to parked vehicles around the fire site and delays in fire extinguishing time due to parked vehicles around the entrance road of the fire engine.

For example, as in the Jecheon Sports Center fire (December 2017), the entrance to a fire engine is delayed by an illegal parking vehicle on the driveway around the building. In this case, the fire truck arrived at the site in 7 minutes at the time of the fire. In addition, even after the fire, there were a large number of illegally parked vehicles on nearby roads, and a related problem occurred due to the lack of safety awareness among citizens.

In this regard, on the other hand, when the damage is caused by typhoons or heavy rains, the flooding of the parking lot in the river highland parking lot, which is difficult to manage, continues to occur.

And, on the other hand, in the event of a disaster, there is no notification service for immediate response of the owners of the surrounding vehicles. Additionally, the current Ministry of Public Administration and Security's Emergency Disaster Message Broadcasting (CBS) is a method of sending a message to all mobile phones belonging to the base station radius of a local area, and thus, disaster messages are transmitted to an unspecified number of people.

Therefore, it is necessary to develop an alarm notification system that combines a portable identification and information transmission terminal using a vehicle plate number recognition technology and a shared administrative information system. In addition, a vehicle number recognition technology suitable for this is also required.

When looking at the related prior art, the prior art literature with such a background is only searched for the prior art from the patent literature and does not search for the related domestic precedent technology directly as the related prior art, but only the following patent literature for reference.

(Patent Document 1) KR1020190143548 A

For reference, the technology of Patent Document 1 improves the license plate recognition rate through vehicle number, type, and color learning, and is a degree of technology regarding a vehicle recognition system capable of real-time learning.

In this regard, in particular, in a traffic image surveillance system, a function of tracking a multi-target is required, and in order to effectively operate this function, it is necessary to secure accurate positions for a plurality of objects and follow real-time tracking.

The disclosed content is to provide a multi-target tracking method of a traffic video surveillance system that enables a system to effectively and quickly track a plurality of objects in a traffic video surveillance system.

The multi-target tracking method of the traffic surveillance system according to the embodiment,

During traffic monitoring, traffic environment information is collected for each camera device installed in each of a number of different traffic monitoring areas.

Then, the object feature metadata is extracted from the collected traffic environment information, and the object feature is extracted from a preset AI format that extracts a feature value of an image patch according to the object feature metadata.

Thus, by recognizing an object by a preset AI format that estimates the size and position of the entire object from the extracted object feature, it is characterized in that it tracks each target and tracks multiple targets.

According to embodiments, in a traffic surveillance system, it is systematically effective and quickly tracks multiple targets.

Additionally, by applying artificial intelligence traffic analysis technology, traffic analysis is systematic and reliable, and appropriate traffic analysis and traffic operation are made according to changes in the traffic environment.

And, in the video surveillance system for recognizing the license plate of the vehicle, by recognizing the vehicle number through the new deep learning-based vehicle number recognition according to this embodiment, the system number is recognized more quickly and systemically more effectively than before.

In addition, it is possible to improve a vehicle specific rate through objectification by extracting features of a vehicle image.

1 is a conceptual diagram to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.
FIG. 2 is a view showing a system to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.
3 is a block diagram showing the configuration of a camera device to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.
Figure 4 is a flow chart showing a multi-target tracking method in sequence in a traffic video surveillance system according to an embodiment
5 is a view for explaining in detail, for example, an object metadata extraction operation according to an embodiment applied to a multi-target tracking method of a traffic image surveillance system according to an embodiment
6 is a view for explaining a CNN-based object classification operation according to an embodiment applied to a multi-target tracking method of a traffic image surveillance system according to an embodiment
7 and 8 are views for explaining in more detail the abnormality detection operation applied to the multi-target tracking method of the traffic image surveillance system according to an embodiment

1 is a conceptual diagram to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.

As shown in FIG. 1, the method according to one embodiment basically collects traffic environment information from the camera device 100 installed on each road and transmits it to the central control center when a vehicle is driving on the road. , Obtain basic information for traffic analysis.

Then, the multi-target tracking method of the traffic image surveillance system according to an embodiment performs traffic analysis by learning such traffic environment information in the central control center.

Therefore, based on the results of the traffic analysis, traffic policies related to the traffic signal policies of the region are derived or traffic abnormal signs are predicted.

In this state, the multi-target tracking method of the traffic image surveillance system tracks multi-targets based on traffic environment information collected by the camera device 100 according to an embodiment.

The camera device 100 is installed for each of a plurality of different traffic monitoring areas, for example, a main intersection, an access road, and the like, and collects traffic environment information of its own traffic monitoring area as basic information for traffic analysis. Is to provide. In such a case, the traffic environment information includes vehicle speed, traffic volume, vehicle occupancy, fine dust status, whether or not the pedestrian crossing traffic light status changes by direct manipulation of the pedestrian, carbon dioxide concentration, fire engines or police cars such as fire departments or police stations, etc. It is environmental information related to comprehensive traffic, such as the stop status of the back, the road occupancy status on the street, etc.

FIG. 2 is a view showing a system to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.

As shown in Figure 2, the system to which the multi-target tracking method of the traffic image surveillance system according to an embodiment is applied is a camera device (100-1, 100-2, ... 100-m, 100-n, .. .) And the central control center 200.

The camera device (100-1, 100-2, ... 100-m, 100-n, ...) is, for example, a major intersection, an access road, a pedestrian crossing, a rainy traffic area, etc. for a number of different traffic monitoring areas Each installed, it collects comprehensive traffic environment information of its traffic monitoring area and provides it to the central control center 200 through its own network as basic information for traffic analysis.

The central control center 200 basically collects and learns such comprehensive traffic environment information from the camera devices 100-1, 100-2, ... 100-m, 100-n, ... Traffic analysis is performed, and related analysis results are provided to the disaster prevention management room or external linkage agencies. At this time, the external linkage agency is a police station information processing device, a traffic agency information processing device, an emergency rescue information processing device, a fire department information processing device, or the like. In this case, the central control center 200 tracks a multi-target according to an embodiment. Specifically, it is as follows. The central control center 200 first collects traffic environment information for each camera device installed in each of a plurality of different traffic monitoring areas during traffic monitoring, and extracts object feature metadata from the traffic environment information. Then, the central control center 200 extracts an object feature from a preset AI format that extracts a feature value of an image patch according to the object feature metadata. Thus, the central control center 200 recognizes objects by a preset AI format that estimates the size and position of the entire object from these object features, and tracks multiple targets by tracking each object.

Additionally, the above-described basic traffic analysis operation is as follows in more detail.

That is, the above-described central control center 200 collects traffic environment information for each camera device, converts it into big data, learns it based on a preset deep learning basis, and applies the results of the deep learning based traffic environment learning to produce a number of different surroundings. Traffic analysis is performed according to environmental conditions, for example, by time zone or by weather and temperature.

In addition, the central control center 200 derives a traffic policy such as a traffic light for each of a number of different traffic elements based on the traffic analysis results for the surrounding environmental conditions that have been analyzed for traffic. Alternatively, traffic anomalies are also predicted based on the traffic analysis results for the traffic conditions.

Additionally, the central control center extracts vehicle feature metadata and vehicle features from the collected traffic environment information according to an embodiment, and recognizes vehicles by estimating the size and position of the entire vehicle from these vehicle features, thereby identifying the vehicle number. Also recognize.

3 is a block diagram showing the configuration of a camera device to which a multi-target tracking method of a traffic image surveillance system according to an embodiment is applied.

As shown in FIG. 3, the camera device 100 to which the multi-target tracking method of the traffic image surveillance system according to an embodiment is applied includes a camera module 110, a DSP unit 120, a sensor unit 130, and a PLC 140.

In this case, the camera device 100 according to an embodiment includes an input module for the PLC 140 to receive signals from the sensor unit, a CPU module as a central control unit, and an output module for outputting control signals to a control target. Includes.

In addition, the PLC 140 is provided with an IOT module that performs an IOT function in the PLC itself, so that the CPU module interoperates with the input/output module by the IOT module to perform an IOT function from a corresponding control logic. do.

The camera module 110 captures an image of a preset traffic monitoring area and delivers it to the DSP unit 120.

The DSP unit 120 digitally processes the photographed image and provides it to the PLC 140.

The sensor unit 130 detects surrounding environment information of the traffic monitoring area. The sensor unit 130 is composed of, for example, an analog sensor that detects fine dust in a traffic monitoring area, a digital switch that detects whether a pedestrian crossing traffic light changes state, or various digital sensors. .

The PLC 140 is configured to provide the result processed by the DSP unit 120 to the central control sensor 200 registered in advance, and to provide information on the surrounding environment of the traffic monitoring area detected by the sensor unit 130. Integrated I/O processing is provided to the central control center 200, and is transmitted to a control target registered in advance. In this case, the PLC 140 has its own IOT module and provides the processed result of the DSP unit provided by the IOT module and the surrounding environment information of the traffic monitoring area to the central control center.

In addition, if this is explained in detail, a module having various functions is required for the existing PLC system used in various environments, and accordingly, the PLC manufacturer provides various modules satisfying user requirements. For example, modules having various functions such as digital input/output modules, analog input/output modules, and communication modules are used in the PLC system, and through these various modules, a system desired by the user is constructed.

For example, the technology of patent document KR101778333 Y1 is a registered invention as such a technology, and specifically, relates to a PLC system having a diagnostic module for diagnosing an operational defect of the output module of the PLC.

The above-described IOT module according to an embodiment uses these points to provide a PLC that provides an IOT function from the IOT module, and furthermore, allows monitoring and control to be easily performed by the user.

Accordingly, according to an embodiment, the PLC 140 includes an input module that receives a signal from a sensor unit, a CPU module that is a central control unit, and an output module that outputs a control signal to a control target.

In addition, the PLC 140 additionally includes an IOT module that performs an IOT function in the PLC itself, and the CPU module performs an IOT function from a corresponding control logic by interworking with the input/output module by the IOT module. Do it.

For reference, the input module receives analog traffic environment information and digital traffic environment information related to traffic environment information. For example, the input module receives, as analog traffic environment information, fine dust information around a camera device from the sensor unit, for example, a fine dust sensor. Or, through the D/I module, the sensor unit, for example, receives pedestrian control information regarding a pedestrian crossing traffic light change around the camera device from the pedestrian traffic light change switch as digital traffic environment information. In this case, the input module is defined to include an I/O card as an input module of the PLC.

In addition, when a signal is input through the input module and a control signal is generated by the CPU module, the output module is a control target, for example, the control signal with a fine dust reduction device or a traffic light installed around the camera device. To pass.

In addition to this, the IOT module performs an IOT function in the PLC itself in this case according to an embodiment. Specifically, when the traffic environment information is collected as described above, the IOT function first has its own wired/wireless communication module. For example, a LoRa module is provided to provide traffic environment information to an external central control center.

In addition, when a signal is input by the input module, the CPU module processes the input signal according to a preset control logic to generate a different control signal. In addition, according to an embodiment, when data is collected by the aforementioned IOT module, the CPU module processes data collected according to a preset control logic to generate different control signals to control differently according to IOT data. .

Additionally, in addition to the above, the PLC 140 issues an alarm or the like with a designated voice for data collected from the I/O card of the PLC.

To this end, the IOT module has its own TTS engine and alarms the voice collected differently according to the voice information for each data set in advance for data collected from the input module of the PLC.

Specifically, the IOT module has its own TTS engine to perform a voice alarm according to the voice information corresponding to the surrounding environment information of the traffic monitoring area.

In this case, the voice message may be registered, for example, in a flash voice memory provided in the IOT module itself.

Thus, through this, an alarm is made with a designated voice for data collected from the PLC's I/O card.

In addition, the PLC 140 additionally performs a noise canceling function for external voice.

Specifically, the IOT module performs an audio IOT function by performing noise canceling by receiving an external voice and outputting an audio.

For example, in this case, an external audio input is received to perform noise canceling, and an audio IOT function is performed by outputting audio through an audio amplifier provided in the IOT module itself.

4 is a flow chart showing a multi-target tracking method in order of a traffic image surveillance system according to an embodiment.

4, the multi-target tracking method of the traffic image surveillance system according to an embodiment is premised on the multi-target tracking method of the traffic image surveillance system by the central control center.

First, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, traffic environment information is collected for each camera device installed in each of a plurality of different traffic surveillance areas in traffic monitoring (S401).

For reference, the traffic environment information includes vehicle speed, traffic volume, vehicle occupancy, fine dust status, whether the pedestrian crossing traffic light is changed by direct manipulation of the pedestrian, carbon dioxide concentration, fire engines such as fire departments or police stations, or This is the environment information related to comprehensive transportation, such as the stop status of a police car, the road occupancy status on the street, and weather information.

Next, object feature metadata is extracted from the collected traffic environment information (S402).

In this case, the object feature metadata refers to an object, for example, metadata indicating a property or characteristic of a vehicle.

Then, according to the extracted object feature metadata, the object feature is extracted from a preset AI format that extracts a feature value of an image patch (S403).

In this case, the AI format is, for example, a fusion machine learning AI format of a Gray Level Co-occurence Matrix (GLCM).

At this time, the GLCM basically has a variety of definitions for a texture image, but represents an image that highlights the spatial characteristics included in the target surveillance image.

In this case, the GLCM calculates basic statistics such as average, contrast, correlation, etc. of brightness values of the current pixel and its neighboring pixels. So, the calculated value is assigned to the central pixel in the kernel as a new brightness value and expressed as a partial texture feature for the input surveillance image.

At this time, the GLCM includes Homogenity, Contrast, and Dissimiliarity.

The Homogenity represents the uniformity of each pixel in the matrix. In addition, Contrast and Dissimiliarity are concepts for distinguishing contrast between pixels, and high weight is applied to pixels far away from the diagonal. ASM and energy measure the uniformity of contrast. There is no change in brightness between pixels, and each pixel has a similar value.

On the other hand, when extracting object features, a deep learning cluster is applied.

The deep learning cluster extracts feature information for object recognition for recognition from an object. The extracted feature information for object recognition is stored in the learned model big data storage unit. The feature information for object recognition extracted by such a deep learning cluster includes, for example, an image name (eg, object name, image frame information and time), an image including a deep vehicle number, and the type of object, for example. , Vehicle, etc., may include the color of the object.

Next, the object is recognized by a preset AI format that estimates the size and position of the entire object from the extracted object feature (S404 ).

In this case, the AI format is, for example, YOLO format.

More specifically, the YOLO format estimates the size and position of the entire object by estimating and combining the total size and position of each object for each of a plurality of different object features.

Thus, the multi-target is tracked by tracking each recognized object (S405).

Accordingly, through this, in the traffic surveillance system, it is systematically effective and quickly tracks multiple targets.

As described above, one embodiment collects traffic environment information for each camera device installed in each of a plurality of different traffic monitoring areas during traffic monitoring.

Then, the object feature metadata is extracted from the collected traffic environment information, and the object feature is extracted from a preset AI format that extracts a feature value of an image patch according to the object feature metadata.

Thus, by recognizing an object by a preset AI format that estimates the size and position of the entire object from the extracted object feature, the multi-target is tracked by tracking each object.

Accordingly, through this, in the traffic surveillance system, it is systematically effective and quickly tracks multiple targets.

Additionally, in this case, the multi-target tracking method of the traffic image surveillance system is performed in parallel with the following traffic analysis operation.

The specific operation is as follows.

First, traffic analysis is performed by collecting comprehensive traffic environment information from camera devices installed for each traffic monitoring area.

To this end, traffic environment information is first collected for each camera device installed in each of a plurality of different traffic monitoring areas during traffic monitoring (S401).

At this time, the traffic environment information includes vehicle speed, traffic volume, vehicle occupancy, fine dust status, whether or not the state of the pedestrian crossing traffic light is changed by direct manipulation of pedestrians, carbon dioxide concentration, fire engines or police cars such as fire departments or police stations, etc. It is the environmental information related to comprehensive traffic, such as the stop status, the road occupancy status on the street, and weather information.

Then, the collected traffic environment information is converted into big data for each camera device (S406), and learning is performed based on a preset deep learning (S407).

At this time, unlike the intelligent used in the existing CCTV expression and alarm, the analysis here focuses on rules, big data patterns, and movement patterns without setting virtual areas, lines, etc. to satisfy specific conditions in the video. do.

This rule analysis is performed according to object information (person, vehicle, etc.), purpose-specific information (child protection, crime prevention, traffic, etc.), location-specific information (specific monitoring area), and time-of-day information (schedule).

In addition, the big data pattern analysis has a lot of information about an object, and creates a data distribution using data such as object classification (person, car, etc.), area, date, and time among the property values of the object. Also, it is characterized by specifying the activity area of the object within the screen to distinguish the area where the object appears and does not appear, and the user sets whether to monitor (normal or abnormal) for the specified data group.

In addition, the movement pattern analysis is performed by tracking the object and analyzing the movement and speed of the normal object using the movement pattern and velocity pattern of the unique object, or by setting whether or not to detect (normal or abnormal).

So, by applying the learned deep learning-based traffic environment learning results, traffic analysis is performed for a number of different environmental conditions, for example, by time zone, weather, temperature, etc. (S408).

Therefore, by applying artificial intelligence traffic analysis technology through this, traffic analysis is systematic and reliable, and appropriate traffic analysis and traffic operation are made according to changes in the traffic environment.

On the other hand, in the case of the multi-target tracking method of the traffic image surveillance system according to an embodiment, in this case, based on the result of the traffic analysis, a traffic policy for a traffic signal policy or the like in a corresponding region may be derived or a traffic anomaly may be predicted. .

1) Derivation of transportation policy

To this end, the multi-target tracking method of the traffic image surveillance system according to an embodiment derives traffic policies for a number of different traffic elements by applying the traffic analysis results according to the traffic conditions to the preset traffic policy establishment format. (S409).

For example, in determining the road fare policy, it is possible to induce efficient road use through methods such as increasing the fare for frequent delays and congestion sections or reducing the fare for sections with good communication.

In addition, if the delay and congestion of the road is concentrated in a specific time period, efficient road use can be applied by applying a variable additional fee in the corresponding time period, or by reducing the fare in a section with better communication compared to other sections in the same time period. It can be used as specific information on the road fare policy to induce. In the case of special transportation periods and weekend holidays, it is possible to grasp the optimal time zone for delivery and outing by receiving the hourly usage of roads in the past relevant period for areas requiring traffic information, thereby helping to make decisions on road use. Gives In addition, through this, the efficiency of road use is increased, thereby improving annual productivity.

2) Traffic anomaly prediction

In addition, the traffic anomaly is predicted by applying the traffic analysis result for each traffic condition to the preset traffic anomaly pattern format (S410).

In this case, for example, by detecting the traffic analysis result by the traffic analysis surrounding environment conditions, for example, by detecting an abnormal pattern by an image distribution diagram of a vehicle, a public institution building, a traffic light, etc. for each object of the traffic abnormal pattern format, traffic Prediction of abnormal signs.

Specifically, the object metadata is extracted from the traffic analysis result for each traffic condition.

Then, the extracted object metadata is classified as an object based on CNN.

Therefore, the abnormality pattern is detected by applying the result of the object classification to the image distribution map for each object in the traffic abnormal pattern format.

FIG. 5 is a diagram for explaining, in detail, an object metadata extraction operation according to an embodiment applied to a multi-target tracking method of a traffic image surveillance system according to an embodiment.

As illustrated in FIG. 5, the object metadata extraction operation according to an embodiment first performs object extraction.

For this, the foreground background is separated first. In more detail, first, successively inputted images are accumulated to generate a background image. Then, a pixel having a difference is calculated by comparing the input image with the background image. Then, a region in which the difference value is greater than or equal to the threshold is output as a foreground image.

Next, noise removal, object extraction, and object combining are performed. Specifically, noise of a foreground image is removed through a morphology filter. Then, the contour of the vehicle feature is extracted from the foreground image from which noise is removed. Then, a vehicle feature is extracted by combining a contour having a minimum size or less with an adjacent contour as a single object.

Meanwhile, in addition to this, multi-object extraction and correction are performed.

Specifically, KCF (Kernelized Correlation Filters) constitutes a new tracking framework that uses the characteristics of the cyclic matrix to improve processing speed. And, in this case, the KCF Tracker tracks the object by extracting the feature of the object robust to the change in brightness using the color feature.

Next, the object extraction result is compared with the prediction result of the KCF Tracker to correct the region and position of the object.

Then, the object is corrected using the correction weight as the object extraction result and the KCF Tracker prediction result, for example, 4 to 1. For reference, then, the operation according to FIG. 6 below is performed.

6 is a view for explaining a CNN-based object classification operation according to an embodiment applied to a multi-target tracking method of a traffic image surveillance system according to an embodiment.

As shown in FIG. 6, the CNN-based object classification operation according to an embodiment is based on CNN as a basic structure of a feature extraction network and a classification network when object metadata is extracted. By performing object classification, object classification based on deep learning is performed.

Specifically, in the CNN-based object classification operation according to an embodiment, when the surveillance image is input as described above, object metadata is extracted from the input surveillance image, thereby obtaining deep learning-based image analysis basic data.

In addition, when the object metadata is extracted, CNN-based objects are classified according to a plurality of different objects from the extracted object metadata, thereby classifying each object through a deep learning basis.

Additionally, the CNN-based object classification operation according to an embodiment classifies an image in which an object appears and performs metadata generation.

Then, by using CNN and constructing and learning with Caffe, which is a deep learning framework, objects such as vehicles are classified. Then, metadata of each object including general information such as color and time of each object of the vehicle is generated.

7 and 8 are diagrams for explaining an abnormality detection operation applied to a multi-target tracking method of a traffic image surveillance system according to an embodiment in more detail.

Specifically, FIG. 7 is a diagram illustrating a rule analysis operation according to the anomaly detection operation, and FIG. 8 is a diagram showing a big data pattern analysis operation according to the anomaly detection operation.

As illustrated in FIGS. 7 and 8, an abnormality detection operation according to an embodiment first tracks a multi-target, and then analyzes images based on a rule-based, big data pattern, and motion pattern.

In other words, unlike the existing intelligent in the CCTV display and alarm, without setting the virtual area, line, etc. to satisfy specific conditions within the video. In the embodiment, analysis is performed mainly on rules, big data patterns, and movement patterns.

For example, whether an object corresponding to the multi-target is abnormal is detected by a preset object abnormal pattern format that detects an abnormal pattern by an image distribution map for each object.

As a specific example, the area of the image in which the object appeared is, for example, divided into 25*25 equal parts (625 blocks). Then, if any part of the region to which the object matches is matched, the appearance region is accumulated. Then, the accumulated data is used to display an overlay based on a frequently occurring heat map. Therefore, it is notified when an object of similar properties appears in a region other than the accumulated region. Then, it is notified when the movement of the object is not normal. For example, if the part where only the person has moved suddenly comes in, the part that the person and the car have moved in is a sudden change.

This rule analysis is based on object information (person, vehicle, etc.), purpose-specific information (child protection, crime prevention, traffic, etc.), location-specific information (specific monitoring area), and time-based information (schedule). to be.

And, in the big data pattern analysis, the metadata of the analyzed image has a lot of information about the object, and data distribution is created using data such as object classification (person, car, etc.), area, date, and time among the property values of the object. do. In addition, the activity area of the object is specified in the screen to distinguish the area where the object appears and does not appear, and the user sets whether to monitor (normal or abnormal) for the specified data group.

In addition, the movement pattern analysis is performed by setting the detection (normal, abnormal) by tracking the object in the image and analyzing the movement and speed of the usual object using the movement pattern and velocity pattern of the unique object.

On the other hand, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, the central control center collects comprehensive traffic environment information from camera devices installed for each traffic surveillance area and recognizes the next turn (FIG. 2).

First, vehicle feature metadata is extracted from traffic environment information collected from a camera device. In this case, the vehicle feature metadata refers to metadata representing a property or characteristic of the vehicle.

Then, when vehicle feature metadata is extracted, vehicle features are extracted from a preset AI format that extracts feature values of a vehicle image patch according to vehicle feature metadata.

In this case, the AI format is, for example, a fusion machine learning AI format of a Gray Level Co-occurence Matrix (GLCM).

At this time, the GLCM basically has a variety of definitions for a texture image, but represents an image that highlights the spatial characteristics included in the target surveillance image.

In this case, the GLCM calculates basic statistics such as average, contrast, correlation, etc. of brightness values of the current pixel and its neighboring pixels. So, the calculated value is assigned to the central pixel in the kernel as a new brightness value and expressed as a partial texture feature for the input surveillance image.

At this time, the GLCM includes Homogenity, Contrast, and Dissimiliarity.

The Homogenity represents the uniformity of each pixel in the matrix. In addition, Contrast and Dissimiliarity are concepts for distinguishing contrast between pixels, and high weight is applied to pixels far away from the diagonal. ASM and energy measure the uniformity of contrast. There is no change in brightness between pixels, and each pixel has a similar value.

On the other hand, when extracting vehicle features, a deep learning cluster is applied. The deep learning cluster extracts feature information for object recognition for recognition from an object. The extracted feature information for object recognition is stored in the learned model big data storage unit. The feature information for object recognition extracted by the running cluster includes, for example, an image name (for example, an object name, image frame information and time), an image including a deep vehicle number, a vehicle type, and a vehicle color. This may be included.

Next, the vehicle is recognized by a preset AI format that estimates the size and position of the entire vehicle from the extracted vehicle features.

In this case, the AI format is, for example, the vehicle YOLO format.

More specifically, the vehicle YOLO format estimates the size and location of the entire vehicle by estimating and combining the overall size and location of each feature for each of a plurality of different vehicle features.

So, when the vehicle is estimated in this way, the vehicle number is recognized by text recognition of the vehicle number and inferring the vehicle number by a preset License Plate Recognition (Lpr) Net.

That is, a vehicle license plate area image (shape) is detected in the entire area of the imaged vehicle.

At this time, the vehicle license plate detection can extract the morphological features of the vehicle, such as the HOG method, and detect the vehicle license plate area image through learning the vehicle/background classifier such as SVM or AdaBoost.

For reference, the LprNet belongs to the prior art, and a detailed description thereof is omitted here.

In this case, when recognizing the vehicle number, deep learning of vehicle number recognition may be applied.

In this case, the vehicle number recognition deep learning collects a vehicle image to derive a vehicle number recognition pattern through a set deep learning. At this time, vehicle number recognition deep learning is deep learning of the vehicle number recognition pattern. When the collected CCTV images are input by frame, the images input for each frame are separated and put into the trained neural network to optimize the images. After extracting the resulting image, it receives the image and performs pre-processing. That is, the pre-processing process converts a color image into a binary image, detects the edges, extracts the edges by applying a threshold, and approaches the arrangement of each contour to erase the noise region.

And after all the pre-processing, the remaining contours are checked in a sorted order from one side and initialized after a certain distance. When the desired contour appears, it is recognized as a license plate. In addition, the vehicle number is recognized by repeatedly performing a process of recognizing the vehicle number as text based on the recognized license plate.

Therefore, deep learning of vehicle number recognition is an iterative and continuous process, and accumulated learning by deep learning is used to extract vehicle numbers from images that are difficult to recognize.

Therefore, through this, in one embodiment, in the video surveillance system that recognizes the license plate of the vehicle, the vehicle number is recognized through the new deep learning-based vehicle number recognition according to this embodiment, thereby being more systematically effective and faster than the existing one. Vehicle number is recognized.

In addition, it is possible to improve a vehicle specific rate through objectification by extracting features of a vehicle image.

As described above, according to an embodiment of the present invention, by extracting vehicle feature metadata from traffic environment information collected by the camera device, image analysis basic data is acquired for each feature of the vehicle based on machine learning.

Then, when the vehicle feature metadata is extracted, vehicle features based on vehicle learning are extracted by extracting vehicle features from a preset AI format that extracts feature values of a vehicle image patch according to vehicle feature metadata.

Next, when the vehicle feature is extracted, the vehicle is recognized by a preset AI format that estimates the size and position of the entire vehicle from the extracted vehicle feature.

So, when the vehicle is estimated in this way, the vehicle number is recognized by text recognition of the vehicle number by the preset LprNet and inference of the vehicle number.

Therefore, through this, in one embodiment, in the video surveillance system that recognizes the license plate of the vehicle, the vehicle number is recognized through the new deep learning-based vehicle number recognition according to this embodiment, thereby being more systematically effective and faster than the existing one. Vehicle number is recognized.

On the other hand, the multi-target tracking method of the traffic image surveillance system according to an embodiment, as described above, by reproducing the vehicle through a three-dimensional model when an object, for example, a vehicle is followed, recognizes the vehicle number more effectively. As much as possible.

To this end, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, when a vehicle is recognized as described above, the vehicle is reproduced by three-dimensional modeling the recognized vehicle by a preset vehicle 3D modeling format. do.

And, at this time, the operation of the 3D modeling is specifically performed as follows.

First, in the operation of the 3D modeling, when a vehicle is recognized as described above, 3D modeling of a vehicle feature by a preset vehicle 3D modeling format is performed to obtain a 3D model for each feature of the vehicle.

Then, by setting a peripheral area other than the feature in the vehicle, a 3D model is obtained by reproducing the peripheral area in 3D by using a preset 3D model format of a peripheral area based on modeling of the curved surface of the vehicle surface. To acquire.

Next, when a 3D model for each feature of the vehicle and a 3D model for a peripheral region other than the vehicle feature are obtained, the 3D model for each feature of the vehicle is obtained from the 3D curved surface for the peripheral region other than the acquired vehicle feature. By matching, the vehicle is reproduced.

Thus, through this, the deep learning-based vehicle number recognition method according to an embodiment, as described above, when the vehicle is followed, reproduces the vehicle through the three-dimensional model according to this embodiment, thereby more effectively Is recognized.

On the other hand, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, when acquiring the three-dimensional model of the vehicle as described above, the surrounding area in the vehicle other than the vehicle feature is reproduced by three-dimensional curvature. By doing so, it is obtained more effectively as a three-dimensional model.

Specifically, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, when a 3D model is obtained for each feature of the vehicle, a peripheral region other than the feature in the vehicle is set by a preset peripheral region curved partitioning format. By segmenting the system systematically, the peripheral area is systematically set.

Thus, when the peripheral area is set as described above, the preset peripheral area is 3D curved and reproduced by a 3D model format based on a preset peripheral area 3D modeling model for the curved surface of the vehicle surface, thereby obtaining a 3D model.

In addition, the curved partitioning method according to an embodiment is to systematically segment a peripheral shape other than a vehicle feature, thereby systematically setting a corresponding portion.

In this case, the curved segmentation method is in a curved segmentation format, and also uses a level set, and this level set is often used in many applications indicated by different names. Implicit curves are level curves and are considered independent of adjacent curves, and these curves are defined by implicit equations.

Likewise, a flat surface is also referred to as an implicit or isosurface or is defined by the name isocontour. This means the contours of the same height, and isocontour used in various applications shows the characteristics of the function value, such as indifference curves. Optimal division is possible by applying this level set function.

Thus, after the division, the peripheral information is 3D modeled by reproducing the peripheral surface information in 3D from a preset 3D model format of surrounding information based on modeling of the surface of the corresponding region.

In addition, in the multi-target tracking method of the traffic image surveillance system of one embodiment, when the vehicle is recognized as described above, the vehicle image is systematically divided from the morphological analysis AI format of the Equalization Contour Model of the preset vehicle. By doing so, a vehicle image is divided corresponding to a plurality of different vehicles.

In this case, the referenced'deformable model' is a deformable spline that minimizes the energy affected by constraints and image forces, and pulls it toward internal forces that resist object contours and deformation. The'deformable model' may be understood as a special case of a general technique of matching a deformable model to an image through energy minimization. In 2D, the active shape model represents a discrete version of this approach and utilizes a point distribution model to limit the explicit area learned from the training range to the shape range.

On the other hand, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, when extracting vehicle feature metadata as described above, by extracting metadata for each feature of the vehicle by meta-model information, it is more efficient. The related metadata is extracted.

To this end, in the multi-target tracking method of the traffic image surveillance system of an embodiment, when the surveillance image is input, by extracting vehicle feature metadata from the input surveillance image from a preset metamodel for each of a plurality of different vehicle features, the machine Based on running, we acquire basic image analysis data for each feature of the vehicle.

On the other hand, additionally, the multi-target tracking method of the traffic image surveillance system according to an embodiment is modeled by a small multi-lane photographing area through a single zoom camera module, a medium multi-lane photographing area modeling, and a large multi-lane photographing area. According to the modeling, while zooming the multi-lane photographing area set on the multi-lane, photographing a plurality of moving objects located in the zoom-adjusted multi-lane photographing area to simultaneously photograph the license plate of the vehicle located in the multi-lane. In addition, it is possible to extract and recognize license plates by combining deep learning, pattern matching, and image processing technology for multi-lane image recording data.

To this end, the multi-target tracking method of the traffic video surveillance system according to an embodiment is located at the bottom of the one-shot camera module, and according to the control signal of the smart control unit, the zoom area camera module, image sensor module, lidar sensor module, multi-lane By pan-tilting the imaging area creation module, a reference imaging area is set on the multi-lane line.

In addition, zoom control is performed while zooming the multi-lane imaging area set on the multi-lane according to the small multi-lane imaging area modeling, the medium multi-lane imaging area modeling, and the large multi-lane imaging area modeling transmitted from the multi-lane imaging area generation module. Multiple moving objects located in the multi-lane imaging area are photographed.

Next, multi-lane image photographing data is generated by converting information of light photographed by the zoom camera module into an electrical signal.

Then, it is located on the lower side of the zoom camera module, outputs a laser pulse of a specific wavelength on a multi-lane, detects a moving object moving in a multi-lane moving object detection section, and then detects the detected multi-lane moving object detection section data Is transferred to the multi-lane imaging area generation module.

Then, after setting the reference photographing area on the multi-lane, based on the transmitted multi-lane moving object detection section data, small multi-lane imaging area modeling, medium-sized multi-lane imaging area modeling, and large multi-lane matching the moving object located in the multi-lane By selecting one of the shooting area modeling, the reference shooting area is changed and transferred to the zoom camera module.

Thus, it is achieved by receiving multi-lane image photographing data transmitted from an image sensor module and controlling to recognize and recognize a vehicle number for a plurality of moving objects located in the multi-lane at the same time.

On the other hand, in addition, in general, the recognition of the vehicle license plate proceeds in a three-step process of acquiring an image, detecting a license plate area, and recognizing a character.

Accordingly, here, in the step of acquiring an image, a method of synthesizing several consecutive input images is used to remove noise generated in the image input step and to increase the clarity.

Then, confirm the degree to which the positions of the parts judged as characters in the input image match the relative positions of the characters specified in the license plate specification. Is detected as the license plate area.

Lastly, by learning the outline pattern of letters in the ART2 neural network, the letters inside the license plate are recognized.

Specifically, the character recognition using the ART2 neural network is different because the license plate of the vehicle is displayed in various colors or the characters inside the license plate are deformed even if the same license plate is affected by the angle, brightness, and ambient lighting of the image. Can adaptively respond to the situation. Then, the character is recognized using a neural network that continuously learns by itself.

Here, the outline shape of the character is recognized as the input pattern of the ART2 neural network. Each character is different from each other when the distance from the left/right sides of the rectangle surrounding the character is changed to a continuous signal pattern.

So, we divide the center of the text in half vertically. Then, the distance between the center line and the point of the left outline of the text and the line of the vertical line is extracted as a continuous pattern from the top of the text. Then, the distance between the points forming the right outline of the character and the center lines is extracted as a continuous pattern. The contours extracted in this way represent analog graphs of different patterns.

Additionally, the multi-target tracking method of the traffic image surveillance system according to an embodiment is different from the aforementioned PLC, and has an external input/output port, that is, a digital input, digital output, and analog input. : 4-20mA input, etc.) It is possible to provide a video surveillance system that can directly handle the communication port of the RS-485 communication port, RS-232 communication port, LAN port, audio port, etc. in the camera unit.

To this end, the multi-target tracking method of the traffic image surveillance system according to an embodiment includes a data collection unit that receives analog data and digital data from external measurement devices, and a commercially available CPU.

The data collection unit is an ADC converter that converts an analog sensor signal into a digital signal, a GPIO port for receiving an electrical input or controlling an external sensor as an output in conjunction with a viewer software, and for receiving digital data from the external measurement device. An MCU that is connected to the RS485 port, the RS232 port for communication with the data processing unit and the viewer software, and the ADC converter, GPIO port, RS485 port, and RS232 port to process data and perform control commands of the viewer software. It may include.

Therefore, through this, an external input/output port, that is, digital input, digital output, analog input (4-20mA input, etc.), RS-485 communication port, RS232 communication port, LAN (LAN ) There is an effect that it is not necessary to negotiate different specifications for each NVR company by directly processing the camera port and audio port.

In addition, data output (D/O (data out)) for controlling external measurement equipment, ports for controlling contact output, alarms, etc., data input (D/I (data in)), analog input (Analog Input: 4-20mA input, etc.) is pre-processed by the MCU and transmitted to the CPU, and puts RS-485 communication port, LAN port, audio port, etc., and receives external input data received from various ports.

By trending data directly from Mera, administrators can easily grasp data such as water level, pressure, and temperature, and this has the effect of providing a method for efficiently controlling external systems.

In this case, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, the data input to the external input/output port is directly database (DATABASE) in the memory of the camera unit, and when it is displayed as an image on the monitor, graph-type data is displayed. Let the trends come together.

To this end, in the multi-target tracking method of the traffic image surveillance system according to an embodiment, analog or digital data such as water level, temperature, pressure, etc. processed together with the collected image of the surveillance area is stored in a database as a database in SD. A can be displayed on the screen of the central control center as a data trend in text or graph format.

The content displayed on the screen is a database for various situations such as water level, temperature, pressure, etc.

Suga is displayed along with the video, and when text is displayed on the video, it can be set to the text of the text, the unit of the text value, the position of the text on the screen, font, color, etc. When the warning, the displayed text blinks in the specified color Distance or text is displayed in either of the flickering cases where the designated color remains in the specified color and the entire screen blinks in the specified color in color or black and white, and the database of various situations is displayed as a data trend in graph format on the image. When expressing, you can express the phrase, unit, and color of the data as set by the administrator. In addition, if the bar of the trend displayed on the image is moved at the desired time, the time value of the trend where the moved bar is located appears, and after checking the data, the data value located on the moving bar disappears automatically. It includes a function that is displayed in either flickering or flickering in which the text remains in a designated color and the entire screen flickers in a specified color in color or black and white.

The method of the program for setting the display method when displaying data text and graph data trends in the image includes general, data analog, digital input, and digital output.

General settings are made in general, and the camera part selects the type of the connected camera part, the address is the network address of the selected camera part, and the protocol is LS Industrial Systems, Modbus, Profibus, etc. It can be selected to match the unit 50, and communication can be selected among RS-232, RS485, and LAN communication, and the communication port (Comm. Port) selects the common port terminal (COM1, COM2) , ..., COM10), IP address is the camera's IP address, History's Trend is selected from Live or History to search for the previous data stored, History selects the date of the previous data search, Period means the search date period

You can choose.

In Analog, Enable is the presence or absence of display data, String is the data name, Measure is the data unit, X axis is the coordinate of the X axis to display characters on the screen, and Y axis is displaying characters on the screen. The coordinates of the Y-axis to be done, the size is the size of the text, the color is the color of the text, the display status is selected from Text or Trend, and the range Min is the minimum number of data. , Maximum range (Range Max) is the highest value of data, and display time (Trend) is displayed as the set time of the time domain X coordinate when displaying Trend.

Enable of Digital Input is the use of display data, String is the data name, X axis is the coordinates of X axis to display characters on the screen, Y axis is the coordinates of Y axis to display characters on the screen. , Size is the size of the text, Color is the color of the text, Effect is the alarm method (Fast flashing of characters, Slow flashing, Screen flashing)

Set, Display Status is selected from Text or Trend, and DisplayTime is displayed at the set time of the time domain X coordinate among Trend coordinates when displaying Trend.

Enable of digital output is enable/disable of display data, string is data name, X axis is the coordinate of X axis to display characters on the screen, Y axis is the coordinate of Y axis to display characters on the screen , Size is the size of the text, Color is the color of the text, Effect is the alarm method (Fast flashing of characters, Slow flashing, Screen flashing), and Control Status Includes a function that allows the administrator to arbitrarily set a program for selecting control on/off (ON/OFF) of the control system.

100: camera device 200: central control center
110: camera module 120: DSP unit
130: sensor unit 140: PLC

Claims (10)

In the multi-target tracking method of the traffic surveillance system by the central control center,
A first step of collecting traffic environment information for each camera device installed in each of a plurality of different traffic monitoring areas during traffic monitoring;
A second step of extracting object feature metadata from the collected traffic environment information;
A third step of extracting an object feature from a preset AI format for extracting a feature value of an image patch according to the extracted object feature metadata;
A fourth step of recognizing an object by a preset AI format that estimates the size and position of the entire object from the extracted object feature; And
A fifth step of tracking multi-targets by tracking each recognized object; Including,
After the fifth step,
A sixth step of detecting whether an object corresponding to the multi-target is abnormal by a preset object abnormal pattern format that detects an abnormal pattern by an image distribution diagram for each object; Further comprising,
After the sixth step,
After the abnormal pattern is detected, a seventh step of notifying an event by assigning different priority to each object by collectively considering object information, information for each purpose, information for each location, and information for each time zone; Multi-target tracking method of the traffic surveillance system, characterized in that it further comprises. The method according to claim 1,
The second step
A 2-1 step of accumulating images continuously input from the collected traffic environment information to generate a background image;
A 2-2 step of comparing the input image and the background image and calculating pixels having a difference by a preset value;
A second 2-3 step of obtaining an area in which the difference value is greater than or equal to a threshold as a foreground image;
2-4 steps of removing noise of the foreground image by deleting it through a morphology filter;
2-5 steps of extracting an outline of an object feature from the noise-free foreground image;
Steps 2-6 of combining a contour having a predetermined minimum size or less with an adjacent contour as a single object;
Steps 2-7 of tracking the single object as a characteristic of an object robust to a change in brightness by using a color feature from a KCF tracker composed of a tracking framework that uses the characteristics of a circular matrix;
Steps 2-8 of correcting the region and position of the object by comparing the object extraction result with the prediction result of the KCF tracker;
The correction comprises the steps 2-9 of correcting the object by setting the correction weight as the object extraction result and the KCF tracker prediction result as a preset N to 1;
2-10 steps of classifying objects by constructing and learning into a deep learning framework using CNN; And
2-11, extracting by generating feature metadata in the classified object; Multi-target tracking method of the traffic surveillance system, characterized in that it comprises a. The method according to claim 2,
Steps 2-10 are
2-10- Analyzing Big Data Patterns by Creating Data Distribution Based on Object Metadata including Object Classification, Area, and Date to Control and Setting Whether to Monitor by Specifying the Object's Activity Area on the Screen Step 1; And
After analyzing the big data pattern, 2-10 to analyze the motion pattern by tracking the object in the CCTV image and setting whether to detect it by analyzing the motion and speed of the object at all times based on the motion pattern and speed pattern of the unique object -2 steps; Multi-target tracking method of the traffic surveillance system, characterized in that it comprises a. The method according to claim 1,
The fourth step
The AI format is obtained by estimating the overall size and position of each feature for a plurality of different vehicle features and estimating and combining the entire vehicle size; Multi-target tracking method of the traffic surveillance system, characterized in that. delete delete delete The method according to claim 1,
The camera device
A camera module for photographing an image of a preset traffic monitoring area;
DSP unit for digitally processing the photographed image;
A sensor unit that detects surrounding environment information of the traffic monitoring area;
The result processed by the DSP unit is provided to a pre-registered central control center, and the surrounding environment information of the traffic monitoring area detected by the sensor unit is integrated and processed to be provided to the central control center. , PLC to be transmitted to a pre-registered control target; It contains,
The PLC
Providing its own IOT module and providing processed results of the DSP unit provided by the IOT module and surrounding environment information of a traffic monitoring area to the central control center; Multi-target tracking method of the traffic surveillance system, characterized in that. The method according to claim 8,
The IOT module
Equipped with its own TTS engine to perform a voice alarm according to the voice information corresponding to the surrounding environment information of the traffic monitoring area; Multi-target tracking method of the traffic surveillance system, characterized in that. The method according to claim 8 or claim 9,
The IOT module
Receiving an external voice related to traffic analysis or vehicle number recognition to perform noise canceling and audio output; Multi-target tracking method of the traffic surveillance system, characterized in that.

Images