JP7125843B2 - Fault detection system - Google Patents

Description

The present disclosure relates to a failure detection system, and in particular, to a failure detection system such as automatic detection of obstacles that occur in a monitoring area such as a road and abnormality detection of vehicles traveling on a road that is a monitoring area from surveillance camera images. .

In recent years, the occurrence of accidents and traffic jams suspected to be caused by objects such as falling objects that interfere with normal driving on highways and the like has become a problem. In addition, the above problem may also occur when a vehicle traveling on a road travels in an unusual manner. In order to improve the safety of road users, it is necessary to detect obstacles that hinder normal driving. Action is required.

One of the means for finding objects that interfere with normal driving is the use of surveillance cameras. Facilities such as highways are often equipped with PTZ (pan, tilt, zoom) type surveillance cameras equipped with an electric camera platform and an electric zoom function to check the traffic flow and the situation in case of an emergency. . The PTZ-type surveillance camera enables a wide range of visibility with a single unit thanks to its swivel/zoom function. If it is possible to automatically detect objects that interfere with normal driving by analyzing surveillance camera images, it will be possible to reduce the occurrence of accidents and traffic jams on highways.

Video analysis techniques for detecting fallen objects using surveillance cameras installed on roads include, for example, the following Patent Documents 1 and 2.

Patent document 1 recognizes a figure of a falling object candidate based on a discrimination condition from among a set of pixels extracted by a difference between images. In addition, it detects the occurrence of falling objects.

In Japanese Patent Laid-Open No. 2002-200000, a fallen object region is extracted by determining the size conditions and the like for other pixels, excluding pixels detected as a vehicle, from the pixels extracted by the difference between images.

JP-A-2002-99903 JP-A-2006-59184

It is considered that the techniques disclosed in Patent Documents 1 and 2 assume video analysis using a dedicated camera with a fixed angle of view. In this case, the surveillance camera can be installed to detect obstacles on the road and can continue to observe the road surface with the same angle of view. It is thought that the occurrence of a running obstacle can be detected while suppressing erroneous detection due to changes in conditions.

However, when trying to detect a wide range with a single PTZ type surveillance camera, zooming/rotating operations are required to rotate the angle of view of the surveillance camera at regular intervals. We cannot continue to observe. In order to detect running obstacles that exist in the screen when the angle of view is changed, it is necessary to extract the occurrence of running obstacles as a change based on information obtained from the video during the previous patrol. In this case, it is necessary to extract points of change based on video data with time differences. is detected as a change point.

Therefore, it is important to set a condition for determining a pixel detected as a change point as a running obstacle. For example, if a simple discrimination condition such as size is used, the background (an area where there are no objects such as obstacles or vehicles) may be detected as an obstacle due to changes in environmental conditions. It can occur frequently.

In order to detect only traveling obstacles while suppressing erroneous detection, it is necessary to set discrimination conditions for distinguishing only traveling obstacles from the characteristics of a set of pixels extracted as change points. Japanese Patent Laid-Open No. 2002-200000 uses a determination condition that combines a plurality of elements for specifying the shape of a traveling obstacle. However, obstacles that should actually be detected are assumed to have various shapes, such as rigid wood and iron scraps, and soft clothing and sheets. It is difficult to set conditions in advance.

The problem of this disclosure is to use a PTZ-type surveillance camera installed on an outdoor road as a means of photographing, and automatically detect traveling obstacles that occur on the road while suppressing false detections caused by changes in environmental conditions. The purpose of the present invention is to provide a video analysis system that automatically detects road surface anomalies.

Other problems and novel features will become apparent from the description of the specification and the accompanying drawings.

A brief outline of a representative one of the present disclosure is as follows.

That is, the obstacle detection system comprises a photographing means (101) for photographing a surveillance area such as on a road at a single or a plurality of angles of view; and object extraction means (201) for extracting values. The obstacle detection system further divides the object region and the pixel value obtained by the object extraction means (201) into blocks based on the determination criteria set for each angle of view and position in the image to obtain local features. An object recognition means (202) for identifying the type of an object from the quantity, and an obstacle detection means for detecting the presence/absence of an obstacle occurring in an image from the information on the type of the object acquired by the object recognition means (202). (204) and

Alternatively, the obstacle detection system includes a photographing means for photographing a monitoring area such as a road at a single or multiple angles of view, and extracting the area and pixel value of an object generated in the monitoring area from the image captured by the photographing means. an object extraction means for tracking an object in a monitoring area from the area and pixel values of the object acquired by the object extraction means; and only a vehicle is selected from the objects tracked by the object tracking means. movement learning means for recording the reference value calculated from the tracking result of the vehicle obtained by the vehicle selection means in association with the photographed angle of view and the position in the screen; and the vehicle Abnormal vehicle detection means for detecting an abnormality in the running direction of a vehicle present in the image based on the tracking result of the vehicle obtained by the selection means and the reference value recorded by the motion learning means.

According to the obstacle detection system described above, it is possible to efficiently and automatically detect an obstacle that has occurred within a surveillance area by means of a PTZ type surveillance camera (shooting means) installed outdoors.

1 is a block diagram of a running obstacle detection system according to an embodiment; FIG. 4 is a block diagram of a video analysis unit; FIG. 4 is a block diagram of an object extraction unit; FIG. 4 is a block diagram of an object recognition unit; FIG. FIG. 10 is a diagram showing an example of a flow chart for object identification; FIG. 4 is a diagram showing the relationship between an object region and local blocks in a surveillance camera image; FIG. 10 is a diagram showing an example of a flow chart of traveling obstacle determination; FIG. 3 is a block diagram of an escape detection unit; It is a figure explaining the flow of learning of motion information.

Examples will be described below with reference to the drawings. However, in the following description, the same components may be denoted by the same reference numerals, and repeated descriptions may be omitted. In addition, in order to clarify the description, the drawings may be represented schematically as compared with actual embodiments, but they are only examples and do not limit the interpretation of the present invention.

FIG. 1 is a block diagram of a running obstacle detection system for a monitoring area such as a road, which is a running obstacle detection system according to an embodiment. Hereinafter, as an embodiment of the obstacle detection system, a traveling obstacle detection system for detecting obstacles when traveling on a road, which is a monitoring area, will be described. The running obstacle detection system is composed of a monitoring camera 101 as a photographing means, a camera control section 102 , a video analysis section 103 and an information output section 104 .

The monitoring camera 101 is, for example, a PTZ (pan, tilt, zoom) camera equipped with an electric platform and an electric zoom. The pan, tilt and zoom states to be applied are given as camera control information.

The camera control unit 102 has a function of outputting camera control information for controlling the pan, tilt, and zoom of the surveillance camera 101 to the surveillance camera 101 . The camera control unit 102 outputs camera control information so that the surveillance camera 101 circulates and monitors one or more angles of view at regular intervals. By changing the angle of view of the surveillance camera 101 at regular intervals, one surveillance camera 101 can automatically detect traveling obstacles in a wider range.

The video analysis unit 103 analyzes the monitoring camera video of the monitoring camera 101 to detect the presence or absence of a traveling obstacle in the video.

FIG. 2 shows a block diagram of the video analysis unit 103. As shown in FIG. The video analysis unit 103 includes an object extraction unit 201 as object extraction means, an object recognition unit 202 as object recognition means, an escape detection unit 203 as escape detection means, and a traveling obstacle detection unit as traveling obstacle detection means. 204. Details of each unit constituting the video analysis unit 103 will be described below.

The object extraction unit 201 extracts object information (an image from which the object is extracted and an object area in the image) of an object present in the surveillance camera image based on the input surveillance camera image and camera control information.

FIG. 3 shows a block diagram of the object extraction unit 201. As shown in FIG. The object extraction unit 201 includes a background difference processing unit 301 , a background model generation unit 302 , a background model recording unit 303 , an inter-background model difference processing unit 304 , and an object determination processing unit 305 .

The background subtraction processing unit 301 uses, for example, a background subtraction method to extract object information of an object moving within the image and an object occurring within the image. The background subtraction processing unit 301 has a role of extracting an obstacle that has moved or occurred while the monitoring camera 101 is continuously capturing an image at a certain angle of view as an object.

Obstacles here are situations that can hinder vehicle travel due to an abnormality on the road, such as a load dropped from a vehicle onto the road, or an abnormality on the road such as bumps, depressions, or cracks in the road itself. As a result, it becomes a traveling obstacle on the road, which is the monitoring area.

The background model generation unit 302 is intended to generate a model representing information only on the background in the surveillance camera video. For example, the background model is a time-smoothed image obtained by applying the update formula shown in Equation 1 below to each frame of the input monitor camera video for a certain period of time.

ここで、Pavg(x,y)は時間平滑化画像の座標(x,y)における画素値、Pin(x,y)は監視カメラ映像のフレームの座標(x,y)における画素値、αは更新のための係数である。

where Pavg(x, y) is the pixel value at coordinates (x, y) of the time-smoothed image, Pin(x, y) is the pixel value at coordinates (x, y) of the surveillance camera video frame, and α is Coefficient for update.

The background model recording unit 303 records the background model generated by the background model generating unit 302 in association with the serial number of the angle of view that the monitoring camera 101 tours based on the camera control information. In addition, the background model recording unit 303 outputs one or more background models generated by the background model generation unit 302 during past tours associated with the angle of view of the current surveillance camera image based on the camera control information.

The inter-background model difference processing unit 304 uses the background model generated by the background model generation unit 302 and the past cyclic model generated from the image having the same angle of view as the input surveillance camera image recorded in the background model recording unit 303 . From the difference of the background model at the time, the object information of the object that occurred during the period from the past tour to the same angle of view again is extracted. The inter-background-model difference processing unit 304 plays a role of extracting a running obstacle existing in the image at that time as an object after the angle of view of the monitoring camera 101 is changed.

For example, when the background model is the temporally smoothed image obtained by Equation 1, the inter-background model difference processing unit 304 stores the current temporally smoothed image obtained from the background model generation unit 302 and the background model recording unit The area of the object is obtained by thresholding the difference of the edge information of the time-smoothed image recorded in 303 in the previous cycle. The inter-background model difference processing unit 304 outputs the object region thus obtained and the current time-smoothed image corresponding to the object region as object information.

The object determination processing unit 305 determines whether or not the object information obtained from the background difference processing unit 301 and the inter-background model difference processing unit 304 is subjected to subsequent image analysis processing. For example, only the object information of the object region whose area is equal to or larger than a certain amount and which is continuously extracted from frames above a certain amount is determined as the object of analysis.

The object recognition unit 202 in FIG. 2 identifies the type of object based on the features of the object information obtained by the object extraction unit 201 .

FIG. 4 shows a block diagram of the object recognition unit 202. As shown in FIG. The object recognition unit 202 includes a feature extraction unit 401 as feature extraction means, an object identification unit 402 as object identification means, a local block extraction unit 403, a local feature extraction unit 404 as local feature extraction means, and a local feature learning unit. It is composed of a local feature learning section 405 and a traveling obstacle determination section 406 as traveling obstacle determination means.

A feature extraction unit 401 calculates a feature amount from object information. Based on one or a plurality of numerical values of the feature amount calculated here, the subsequent object identification unit 402 identifies the type of the object. The feature amount is calculated based on, for example, the number of pixels in the object area, edge amount, color, movement amount within the screen, and the like.

Based on the feature quantity obtained from the feature extraction unit 401, the object identification unit 402 identifies the object as a traveling obstacle candidate for determining whether the object is of a specific type or a traveling obstacle in a subsequent process.

FIG. 5 shows an example flow chart of object identification for each object. The flowchart of FIG. 5 is an example of a flowchart in which objects are classified into "vehicles", "lighting poles", "running obstacles", and other "running obstacle candidates".

In step S101, based on the feature of the object obtained by the feature extraction unit 401, it is determined whether the object is a "vehicle". The determination may be performed by subjecting a plurality of feature amounts to threshold processing, or may be performed by using a discriminator created in advance by performing machine learning on a sample image or the like. Also, the determination condition may be changed according to the environment such as the angle of view of the camera based on the camera control information.

An object determined to satisfy the condition of "vehicle" in step S102 is identified as "vehicle" in step S103.

In step S104, it is determined whether the object determined as not being a "vehicle" in step S102 is a "lighting pole". Objects that are determined to satisfy the condition of "illumination column" in step S105 are identified as "illumination column" in step S106.

In step S107, it is determined whether or not the object determined as not being the "illumination column" in step S105 is the "moving obstacle". Objects that are determined to satisfy the condition of "traveling obstacle" in step S108 are identified as "traveling obstacles" in step S109.

In step S110, an object that is not determined as a "vehicle", "illumination pole", or "running obstacle" is identified as a "running obstacle candidate".

A local block extraction unit 403 extracts a local block 503 to be analyzed by subsequent processing from the object information. A local block 503 is a pixel block extracted from object information that can specify an area in the surveillance camera image by a serial number.

FIG. 6 shows the relationship between an object region 502 and local blocks 503 in a surveillance camera image 501. As shown in FIG. FIG. 6 shows an example in which one local block 503 is provided inside an object region 502 . One or more local blocks 503 can be extracted for one piece of object information. In this case, the local blocks 503 may have a plurality of different sizes and shapes, or may overlap each other, as long as the area can be identified by the serial number.

The local feature extraction unit 404 extracts the feature amount of the local block 503 corresponding to the object information identified by the object identification unit 402 as the “traveling obstacle candidate” among the local blocks 503 extracted by the local block extraction unit 403. Extract. The feature amount of the local block 503 is hereinafter referred to as a local feature amount. For the local feature amount, a numerical value calculated based on the direction of edges in the local block 503, the distribution of luminance, or the like, or the feature of the entire object calculated by the feature extraction unit 401 can be used.

A local feature learning unit 405 learns the features obtained by the local feature extraction unit 404 . Learning is performed for each serial number of the local block 503 . As a feature learning method, for example, mixed distribution parameter learning by an EM (Expectation Maximization) method is used. By using the EM method, it becomes possible to express the tendency of the features of each local block 503 by a mixed normal distribution. The learning result is recorded in association with the angle of view that the monitoring camera 101 tours when changing the angle of view based on the camera control information. The learning results recorded during the previous round can be used as the initial values for learning.

The traveling obstacle determination unit 406 uses the identification result of the object identification unit 402, the local feature amount of the “traveling obstacle candidate” obtained from the local feature extraction unit 404, and the past “ Whether the object is a traveling obstacle or not is determined based on the learning result of "candidate traveling obstacle".

FIG. 7 shows an example of a flow chart of the travel obstacle determination for each object, which is performed by the travel obstacle determination unit 406. In FIG.

An object identified as a "traveling obstacle" by the object identifying unit 402 in step S201 is determined as a "traveling obstacle" in step S202.

An object identified as not being a "running obstacle candidate" by the object identifying unit 402 in step S203 is determined to be "not a running obstacle" in step S204.

In step S<b>205 , the likelihood of each local feature amount of the object is calculated based on the learning result of the local feature learning unit 405 .

In step S206, it is determined whether the object is a traveling obstacle based on the likelihood of each local feature amount. For example, the running obstacle condition is that the ratio of the number of local blocks 503 whose likelihood is greater than or equal to a set value by threshold processing out of all the local blocks 503 obtained from the object information is equal to or greater than a certain value.

An object that satisfies the traveling obstacle condition in step S207 is determined as a "traveling obstacle" in step S202. An object determined not to satisfy the traveling obstacle condition in step S207 is determined to be "not a traveling obstacle" in step S204.

The effect of the traveling obstacle determination process of FIG. 7 described above will be described below.

In the object recognition unit 202 , the type of object is first identified by the feature extraction unit 401 and the object identification unit 402 . As described above, various types of objects are assumed to be traveling obstacles, so it is often difficult to identify traveling obstacles depending on their characteristics and conditions. Therefore, the object recognizing unit 202 identifies an object that cannot be determined as a "running obstacle" or "not a running obstacle" from its characteristics as a "running obstacle candidate", and performs the determination process at a later stage. I decided to do it with Objects identified as "traveling obstacle candidates" include, in addition to traveling obstacles, part of the background, shadows, light reflections, etc. extracted due to changes in environmental conditions, for example. As with the obstacles on the road, various cases are conceivable, so it is difficult to specify the type of the object only from its characteristics.

Therefore, in the object recognition unit 202, the local block extraction unit 403, the local feature extraction unit 404, and the local feature learning unit 405 learn the local feature amount of the object identified by the object identification unit 402 as a “traveling obstacle candidate”. Is going. Learning of the local feature amount is performed for each local block 503 of the angle of view that the traffic flow monitoring camera 101 tours. In other words, it learns what kind of features a "moving obstacle candidate" has in an arbitrary area of an arbitrary angle of view. In general, it is rare for a running obstacle to appear in the image of the surveillance camera 101, and most of what is identified as a "running obstacle candidate" is a background portion that causes erroneous detection. Therefore, in the object recognition processing by the object recognition unit 202, the position and characteristics of the object information that caused the false alarm are mainly learned for each angle of view. Based on the learning result of the local feature amount, the traveling obstacle determination unit 406 determines that an object having many local feature amounts close to the local feature amount learned in the past is not a traveling obstacle, and determines other objects as a traveling obstacle. do.

Through the object recognition processing by the object recognition unit 202, even if it is not possible to determine that the object is a running obstacle based on the features of the object, by learning the tendency of the object information that causes false alarms, it is possible to detect the obstacle while suppressing false alarms. can be detected.

Based on the object information obtained by the object extraction unit 201, the escape detection unit 203 in FIG. 2 determines whether or not the vehicle has escaped in the monitoring camera image when the object is a vehicle. conduct.

FIG. 8 shows a block diagram of the escape detection unit 203. As shown in FIG. The avoidance detection unit 203 is composed of an object tracking unit 601 as object tracking means, a vehicle selection unit 602 as vehicle selection means, a motion information learning unit 603 as motion learning means, and an escape determination unit 604 .

An object tracking unit 601 tracks a moving object existing in the surveillance camera image. The object tracking unit 601 performs object tracking by associating feature points between frames using, for example, a KLT (Kanade-Lucas-Tomasi) tracker. A new feature point is extracted from within the object region, and tracking is continued until the tracking end condition is met, such as the feature point disappearing or becoming stationary. By performing such object tracking processing, the trajectory of each feature point moving within the screen can be obtained. By clustering the trajectory of each feature point based on information such as the position, moving direction, and object area where the feature point was obtained, it is possible to obtain multiple clusters that represent the movement of the object moving in the surveillance camera image. can.

A vehicle selection unit 602 selects clusters corresponding to vehicles from the clusters of the trajectory of the feature points obtained by the object tracking unit 601 . For example, based on the moving speed, size, number of feature points, tracked period, etc. of the object on the screen, conditions for being a vehicle are set, and clusters of objects that satisfy the conditions are selected as vehicle motion information. do.

A motion information learning unit 603 divides the surveillance camera image into blocks and learns the motion of the vehicle in units of blocks. Examples of objects to be learned include the moving direction of the vehicle. FIG. 9 is a diagram showing the flow of motion information learning, and the flow of motion information learning will be described using (a), (b), (c), and (d) of FIG.

First, the motion information learning unit 603 obtains the trajectory of the feature points of the vehicle generated during a certain period based on the tracking result of the vehicle. In (a) of FIG. 9, an image 701 shows the result of the trajectory of feature points generated over a certain period of time. An image 702 represents the trajectory of feature points generated during a certain period of time. In (b) of FIG. 9, an image 703 shows an example in which the image 701 is divided into blocks. In (c) of FIG. 9, an image 704 shows the result of calculating the total sum of trajectories passing through each block. The total sum of trajectories in each block can be obtained, for example, by Equation 2 below.

ここでｄ_tmp（i,j）は一定期間内に座標のブロックを通過した軌跡の総和を示す方向ベクトルである。Ｎは特徴点の軌跡の総数、ｎは特徴点の軌跡の通し番号を表す。Ｍ_nはｎ番目の軌跡に対して追跡が行われたフレーム数であり、ｍは追跡が開始された時点からのフレーム数を表す。p(n,m)はｎ番目に対して追跡が開始されてからｍ番目のフレームでの特徴点の位置を示すベクトルである。B(i,j)は座標（i,j）のブロックの領域を示している。if(X,Y,Z)は条件Ｘを満たした際には値Ｙを、それ以外の場合は値Ｚを返す関数である。

Here, d _tmp (i, j) is a direction vector indicating the total sum of trajectories passing through the coordinate block within a certain period. N is the total number of feature point trajectories, and n is the serial number of the feature point trajectories. M _n is the number of frames tracked for the n-th trajectory, and m represents the number of frames from the start of tracking. p(n,m) is a vector indicating the position of the feature point in the m-th frame after tracking is started for the n-th. B(i,j) indicates the area of the block with coordinates (i,j). if(X, Y, Z) is a function that returns a value Y when the condition X is satisfied, and a value Z otherwise.

In (d) of FIG. 9, image 705 shows the learning result of the trajectory direction of each block, updated using the result of image 704 . For example, Equation 3 below can be used to update the learning result.

ここで、ｄ(i,j)は座標(i,j)のブロックにおける車両の移動方向の傾向を示す方向ベクトルである。αは更新のための係数である。

Here, d(i,j) is a directional vector indicating the tendency of the moving direction of the vehicle in the block of coordinates (i,j). α is the coefficient for updating.

Based on the camera control information, the learning result is recorded in association with the angle of view that the monitoring camera 101 tours.

Based on the learning result of the motion information learning unit 603, the avoidance determination unit 604 quantifies and extracts the avoidance of the vehicle from the vehicle motion information, and performs escape detection processing. The avoidance amount can be obtained by, for example, Equation 4 below.

ここで、avdは車両の避走量を表す数値である。Lは析対象の車両のクラスタが持つ特徴点の軌跡の総数であり、lは特徴点の軌跡の通し番号である。Ｍ_ｎはｎ番目の軌跡に対して追跡が行われたフレーム数であり、ｍは追跡が開始された時点からのフレーム数を表す。p(n、m)はｎ番目に対して追跡が開始されてからｍ番目のフレームでの特徴点の位置を示すベクトルである。d(i_ｌｍ、ｊ_ｌｍ)は動き情報学習部６０３によって得られた、座標(i,j)のブロックにおける車両の移動方向の傾向を示す方向ベクトルである。i_ｌｍ及びｊ_ｌｍは特徴点の位置p(l,m)が含まれるブロックの座標を示す。inner(a,b)はベクトルaとベクトルbの内積を返す関数である。Ｍは、以下の式５に示される回転行列である。

Here, avd is a numerical value representing the avoidance amount of the vehicle. L is the total number of feature point trajectories possessed by the vehicle cluster to be analyzed, and l is the serial number of the feature point trajectories. _Mn is the number of frames tracked for the nth trajectory, and m represents the number of frames from the time the tracking started. p(n, m) is a vector indicating the position of the feature point in the m-th frame after tracking is started for the n-th. d(i _lm , j _lm ) is a directional vector obtained by the motion information learning unit 603 and indicating the tendency of the moving direction of the vehicle in the block of coordinates (i, j). i _lm and j _lm indicate the coordinates of the block containing the feature point position p(l,m). inner(a,b) is a function that returns the inner product of vector a and vector b. M is the rotation matrix shown in Equation 5 below.

式４で得られた車両の避走量avdを閾値処理し、閾値を超えた避走量を持つ車両を避走した車両として検知する。

The avoidance amount avd of the vehicle obtained by Equation 4 is subjected to threshold processing, and a vehicle having an avoidance amount exceeding the threshold is detected as a vehicle that has escaped.

The effect of the escape detection process described above will be described below.

The avoidance detection process is used to estimate the presence of a traveling obstacle from the action of the vehicle avoiding the obstacle (abnormal action in the traveling direction of the vehicle). By the avoidance detection process, it is possible to detect a traveling obstacle even when the obstacle is not captured by the monitoring camera 101 because it is hidden by a structure or the like.

The avoidance detection process is characterized by dividing the surveillance camera image into blocks and learning the movement of the vehicle in block units for each angle of view of the surveillance camera 101 circulating. The avoidance detection process has the advantage of not requiring adjustments necessary for escape detection, such as designating the direction of the lane for each angle of view of the surveillance camera.

In the above explanation of the avoidance detection process, only the moving direction of the vehicle is learned, but the amount of escape of the vehicle may be learned in units of blocks. By learning the average value of the escape distance of the vehicle and detecting the escape distance based on the value normalized by the average value, the influence of variations in the escape distance due to differences in shooting conditions and shooting locations. can be reduced.

A traveling obstacle detection unit 204 detects the presence of a traveling obstacle from the object identification result of the object recognition unit 202 and the escape detection result of the escape detection unit 203 . A running obstacle detection unit 204 detects a running obstacle when an object determined as a running obstacle by the object recognition unit 202 is detected over a set number of frames at positions where object regions in the surveillance camera video overlap. do. The traveling obstacle detection unit 204 also detects when more than a set number of vehicles make an escape within a period set from the escape detection result of the escape detection unit 203, or when the set number of vehicles pass through the monitoring camera video. When more than a set number of vehicles in the vehicle run away, obstacles are detected.

The information output unit 104 outputs the detection result of the traveling obstacle by the video analysis unit 103 to other devices and systems such as an information display board. The information to be output includes not only the presence or absence of traveling obstacle detection, but also information on whether traveling obstacle detection is performed by object recognition processing or escape detection processing. Accuracy information may be included.

The configuration of the embodiment described above is summarized below.

The running obstacle detection system includes a PTZ type surveillance camera 101 with a turning/zoom function, a camera control unit 102 that controls the surveillance camera 101 to rotate and capture images from a plurality of angles of view, surveillance camera images and camera control information. and a video analysis unit 103 that detects the presence or absence of obstacles on the road based on (setting values of turning/zooming for controlling the monitoring camera and the serial number of the angle of view that the monitoring camera tours). The running obstacle detection system further includes an information output unit 104 that outputs the detection result of the running obstacle from the video analysis unit 103 to another display device.

The video analysis unit 103 extracts object information (image from which the object is extracted and the object area in the image) of an object existing in the monitoring camera video based on the input monitoring camera video and camera control information. and an object recognition unit 202 that identifies the type of object based on the camera control information and the features of the object information obtained by the object extraction unit 201 . The video analysis unit 103 further detects the avoidance run of the vehicle in the video from the camera control information and the object information obtained by the object extraction unit 201 (driving that the vehicle seems to do when avoiding obstacles such as changing lanes). and a traveling obstacle detection unit 204 that detects traveling obstacles based on the object identification result obtained by the object recognition unit 202 and the escape detection result obtained by the escape detection unit 203. , has

The object extraction unit 201 included in the image analysis unit 103 includes a background difference processing unit 301 that extracts the area and pixel value of an object that moves within the surveillance camera image and that occurs in the surveillance camera image from the surveillance camera image, and a background model generation unit 302 for generating a background model (a model representing the state of the background portion other than the object moving in the surveillance camera image) from the image. The object extraction unit 201 further includes a background model recording unit 303 that records the background model generated by the background model generation unit 302 and outputs one or more background models recorded based on the camera control information, and a background model generation unit. It has a background model difference processing unit 304 that extracts the area and pixel value of a stationary object existing in the surveillance camera image from the background model generated in 302 and the background model recorded in the background model recording unit 303. The object extraction unit 201 also performs object determination processing for determining an object to be analyzed from the object area and pixel values extracted by the background difference processing unit 301 and the inter-background model difference processing unit 304, and outputting the object as object information. and a part 305 .

The object recognition unit 202 included in the video analysis unit 103 includes a feature extraction unit 401 for extracting one or more feature amounts for identifying an object from the object information obtained from the object extraction unit 201, camera control information and feature extraction. and an object identification unit 402 for identifying an object as a traveling obstacle or one or more specific types or traveling obstacle candidates based on the feature amount obtained in the unit 401 . The object recognition unit 202 further includes a local feature extraction unit 404 for extracting local features of the object information of the object identified as a traveling obstacle candidate by the object identification unit 402, and a camera control information and the local feature extraction unit 404. A local feature learning unit 405 that learns (obtains the tendency of the input feature) the feature of the object of the traveling obstacle candidate extracted at each angle of view that the surveillance camera tours based on the local feature amount obtained by , has The object recognition unit 202 further determines whether the object is a running obstacle based on the object identification result obtained from the object identification unit 402, the local feature amount obtained from the local feature extraction unit 404, and the local feature learning result obtained from the local feature learning unit 405. and a traveling obstacle determination unit 406 that determines whether the object is an object.

The escape detection unit 203 included in the image analysis unit 103 includes an object tracking unit 601 that tracks an object in the surveillance camera image over a plurality of frames based on the object information obtained from the object extraction unit 201, and the object tracking unit 601 and a vehicle selection unit 602 for selecting a vehicle for motion analysis from the tracked object. The avoidance detection unit 203 further learns the movement of the vehicle at each angle of view that the surveillance camera 101 circulates based on the camera control information and the vehicle movement information obtained by the vehicle selection unit 602. A motion information learning unit 603 that acquires a tendency), and based on the vehicle motion information obtained by the vehicle selection unit 602 and the motion information learning result obtained by the motion information learning unit 603, it is determined whether the vehicle has run away. and a avoidance determination unit 604 .

Further, according to the embodiment of the present invention, there are provided a photographing means for photographing the road at a single or a plurality of angles of view, and extracting the area and pixel value of the object generated on the road from the image captured by the photographing means. an object extraction means for tracking an object on the road from the area and pixel values of the object acquired by the object extraction means; and only a vehicle is selected from the objects tracked by the object tracking means. movement learning means for recording the reference value calculated from the tracking result of the vehicle obtained by the vehicle selection means in association with the photographed angle of view and the position in the screen; and the vehicle Abnormal vehicle detection means for detecting an abnormality in the running direction of a vehicle present in the image based on the tracking result of the vehicle obtained by the selection means and the reference value recorded by the motion learning means.

Further, according to the embodiment of the present invention, a photographing means for photographing a road at a single or a plurality of angles of view, a road surface information storage means for storing predetermined road surface information on the road photographed in advance, and anomaly extracting means for comparing an image fetched from the means with predetermined road surface information stored in the road surface information storage means, and extracting an area of an abnormality occurring on the road and a pixel value from the comparison result; and abnormality recognition means for identifying the type of abnormality from the local feature amount by dividing the abnormal region and pixel value obtained by the abnormality extraction means into blocks based on the determination criteria set for each position in the image. and obstacle detection means for detecting the presence or absence of a road surface abnormality occurring in the image as a running obstacle based on the abnormality type information acquired by the abnormality recognition means.

According to the above travel obstacle detection system 1, it is possible to efficiently and automatically detect road surface abnormalities such as falling objects that have occurred in the surveillance area using a PTZ type surveillance camera installed outdoors. becomes.

Although the invention made by the present inventor has been specifically described above based on the examples, it goes without saying that the invention is not limited to the above-described embodiments and examples, and can be variously modified. .

1: Running obstacle detection system 101: Surveillance camera 102: Camera control unit 103: Video analysis unit 104: Information output unit 201: Object extraction unit 202: Object recognition unit 203: Escape detection unit 204: Travel obstacle detection unit 301: Background difference processing unit 302: background model generation unit 303: background model recording unit 304: inter-background model difference processing unit 305: object determination processing unit 401: feature extraction unit 402: object identification unit 403: local block extraction unit 404: local features Extraction unit 405: Local feature learning unit 406: Traveling obstacle determination unit 601: Object tracking unit 602: Vehicle selection unit 603: Motion information learning unit 604: Avoidance determination unit

Claims (3)

a photographing means for photographing a surveillance area with a single or multiple angles of view;
an object extracting means for extracting an area and pixel values of an object generated in a monitoring area from the image captured by the photographing means;
Based on the determination criteria set for each angle of view and position in the image, the object region and pixel values obtained by the object extraction means are divided into blocks, and the type of the object is identified from the local feature amount. a recognition means;
escape detection means for detecting the avoidance of the vehicle in the image captured by the image capture means from the image captured by the image capture means and the area and pixel value of the object obtained by the object extraction means;
An obstacle that detects the presence or absence of an obstacle occurring in the image captured by the photographing means based on the information on the type of the object acquired by the object recognition means and the escape detection result obtained by the escape detection means. and object detection means,
The object recognition means is
feature extracting means for extracting one or more feature quantities for identifying an object from the object area and pixel values obtained by the object extracting means;
object identification means for identifying an object as an obstacle or one or more specific types or obstacle candidates based on the feature amount obtained by the feature extraction means;
local feature extracting means for dividing the area and pixel values of the object identified as an obstacle candidate by the object identifying means into blocks and extracting one or more feature amounts for each block;
local feature learning means for calculating a criterion for determining an obstacle from the local feature amount obtained by the local feature extraction means;
an obstacle determination means for determining whether an object identified as an obstacle candidate by the object identification means is an obstacle based on the determination criteria obtained by the local feature learning means ;
A result of determination as to whether the object identified as the obstacle candidate is an obstacle is input to the obstacle detection means as information on the type of the object;
A failure detection system characterized by: In the failure detection system according to claim 1,
an object tracking means for tracking an object in a monitoring area from the area and pixel values of the object obtained by the object extraction means;
vehicle selection means for selecting only vehicles from among the objects tracked by the object tracking means;
a motion learning means for recording the reference value calculated from the tracking result of the vehicle obtained by the vehicle selection means in association with the photographed angle of view and the position in the screen;
escape detection means for detecting an abnormality in the running direction of a vehicle present in an image based on the tracking result of the vehicle obtained by the vehicle selection means and the reference value recorded by the motion learning means;
an obstacle detection means for detecting the occurrence of an obstacle from the frequency of escape detection by the escape detection means;
A failure detection system characterized by: a photographing means for photographing a surveillance area with a single or multiple angles of view;
an object extracting means for extracting an area and pixel values of an object generated in a monitoring area from the image captured by the photographing means;
an object tracking means for tracking an object in a monitoring area from the area and pixel values of the object obtained by the object extraction means;
vehicle selection means for selecting only vehicles from among the objects tracked by the object tracking means;
a motion learning means for recording the reference value calculated from the tracking result of the vehicle obtained by the vehicle selection means in association with the photographed angle of view and the position in the screen;
Abnormal vehicle detection means for detecting an abnormality in the running direction of a vehicle existing in an image based on the vehicle tracking result obtained by the vehicle selection means and the reference value recorded by the motion learning means;
A feature amount is calculated from the area and pixel values of the object acquired by the object extraction means, and based on the calculated feature amount, the type of the object or whether it is a traveling obstacle is determined in a subsequent process. an object recognition unit for identifying as a traveling obstacle candidate;
a traveling obstacle detection unit that detects the presence of a traveling obstacle from the object identification result of the object recognition unit and the abnormality in the traveling direction of the vehicle detected by the abnormal vehicle detection means ;
A failure detection system characterized by:

Images