JP7536125B1 - Surveillance system - Google Patents

Abstract

[Problem] The object of the present invention is to solve the problems of the prior art, that is, to provide a surveillance system capable of generating an image for evaluation based on a plurality of photographs, and further capable of monitoring an object by generating an image for evaluation using the remaining regions even if some regions of the image are unusable. [Solution] The surveillance system of the present invention comprises a fixed camera installed at one location, a mask processing means, a representative image generating means, a matching processing means, and a changed region extracting means. The representative image generating means is a means for generating a period representative image based on a period image that is a compilation of a plurality of images acquired by the fixed camera during a shooting period, and the changed region extracting means is a means for extracting a common region as a changed region when the position of a common region in the period representative image differs between two periods and the amount of movement between the common regions exceeds a predetermined movement amount threshold. [Selected Figure] Figure 2

Description

This invention relates to technology for observing the dynamic behavior of an object, and more specifically, to a surveillance system that extracts areas of change in an object using photographs taken with a fixed camera.

Two-thirds of Japan's land area is said to be mountainous, and as a result, many homes are built on land backed by slopes, and roads and railways almost always pass along slopes. Slopes are prone to disasters such as collapses and landslides, and in the past the country has often suffered severe damage from slope collapses and other such disasters.

On slopes at risk of collapse (including natural slopes and artificial slopes) or on slopes showing signs of landslides, measurements are sometimes taken to monitor their movement. For example, on slopes showing signs of landslides, measurements using extensometers and punching plates, measurements using borehole inclinometers, and measurements of ground surface displacement have been carried out. However, measurements using extensometers and punching plates are only effective if they are installed across the landslide boundary (especially the head), and borehole extensometers are only effective if they can accurately estimate the landslide depth, and installing them in multiple locations is costly.

Surface displacement measurement involves determining the coordinates of multiple observation points installed on a slope and monitoring the movement of the slope by detecting displacement over time, making it possible to directly identify abnormalities and has the advantage that its effectiveness does not depend on the location of the instrument, as does the case with extensometers and borehole inclinometers. However, installing the observation points involves risk as people must enter the slope, and furthermore, the observation points must be positioned by hand using a total station or similar device, which requires a lot of effort and cost.

Therefore, various monitoring techniques have been proposed to replace direct measurements using total stations and the like. For example, Patent Document 1 proposes a technique for monitoring the movement of a slope using images of the slope.

JP 2019-201275 A

The invention disclosed in Patent Document 1 uses images (single photographs) obtained from a fixed camera, and is capable of monitoring an object after removing predictable noise. This invention makes it possible to realize real-time and constant monitoring of an object, while reducing the costs of equipment and installation compared to conventional technology. On the other hand, since single photographs taken at two different times are compared, some noise is unavoidable. Also, if there is even a part of an image that cannot be used due to cloud cover or other reasons, the single photograph cannot be used.

The objective of the present invention is to solve the problems of the past, that is, to provide a monitoring system that can generate an image for evaluation based on multiple photographs, and even if some areas of the image are unusable, generate an image for evaluation using the remaining areas, and then monitor an object.

The present invention focuses on the point that evaluation images are generated based on images taken within a certain period of time, and then the conditions at two different times are compared, and is an invention based on an idea that has not been seen before.

The surveillance system of the present invention includes a fixed camera installed at one location, a mask processing means, a representative image generating means, a matching processing means, and a change area extraction means. The fixed camera periodically (or intermittently) captures images of the object. The mask processing means compares the brightness of the image captured by the fixed camera with a predetermined brightness threshold, sets the portion of the image whose brightness exceeds the brightness threshold as a mask range, and sets the portion of the image excluding the mask range as a use range. The representative image generating means generates a period representative image based on a period image that is a compilation of multiple images captured by the fixed camera during a predetermined shooting period, and the matching processing means performs pattern matching between period representative images relating to two periods and extracts a portion common to the period representative images as a common area. The change area extraction means extracts the common area as a change area when the position of the common area in the period representative image differs between the two periods and the amount of movement between the common areas exceeds a predetermined movement amount threshold. The representative image generating means uses the range of use in the period image to calculate an average luminance value, which is the average of the luminance values associated with the same pixel, and generates a period representative image by assigning the average luminance value to each pixel.

The monitoring system of the present invention can also set an area where multiple mask pixels are gathered as a mask range. In this case, the mask processing means sets pixels in the image whose luminance exceeds a luminance threshold as mask pixels, and sets a collective portion where the number of gathered mask pixels exceeds a predetermined cluster threshold as a mask range.

The surveillance system of the present invention may further include a directional noise extraction means. This directional noise extraction means is a means for extracting candidate changed areas in which the movement direction of the common area is within a range of a predetermined directional threshold as "directional noise." The directional threshold is a range set with a predetermined width from the vertical upward direction. In this case, the changed area extraction means extracts the changed area excluding the directional noise.

The monitoring system of the present invention has the following advantages.
(1) By generating a representative image for a period based on images for multiple periods and then extracting changed areas, it is possible to reduce noise compared to the case of comparing single photographs from two periods. As a result, it is possible to extract changed areas with higher accuracy than with conventional techniques.
(2) Even if some areas of an image cannot be used, the period representative image is generated by utilizing the remaining areas, so that changed areas can be extracted more efficiently than with conventional techniques.
(3) By removing "directional noise," it is possible to accurately extract areas where significant changes have actually occurred.

1 is a block diagram showing the main configuration of a monitoring system according to the present invention. FIG. 4A is a model diagram that shows a grayscale image from which mask pixels and a mask range have been extracted, and FIG. 4B is a model diagram that shows a processed image from which the mask range has been removed from the grayscale image. FIG. 2 is a model diagram showing a shooting period, images of the period, and a representative image of the period. A model diagram showing a "convex common area consisting of seven pixels" extracted by performing turn matching. FIG. 11 is a model diagram showing a schematic diagram of a movement vector relating to a common area. FIG. 13 is a model diagram showing a change direction threshold set at a predetermined width on both sides from a vertically upward direction. FIG. 2 is a flowchart showing the flow of main processes in the monitoring system of the present invention.

An example of an embodiment of the monitoring system of the present invention is described below with reference to the drawings.

First, an overview of the present invention will be described. The present invention monitors changes in a target object (hereinafter simply referred to as the "target object") using photographs taken by a camera installed at one location (hereinafter referred to as the "fixed-point camera"). More specifically, the present invention is a technology that generates a representative image (hereinafter referred to as the "representative image for the period") using multiple single photographs (hereinafter referred to as the "images for the period") taken by a fixed-point camera within a predetermined period (hereinafter referred to as the "photography period"), and monitors changes in the target object by comparing the representative images for the period from two different periods. Note that the present invention can be implemented using a variety of objects as the target object, but for convenience, the present invention will be described in the case where the target object is a slope (such as a natural slope or a slope).

Figure 1 is a block diagram showing the main components of a surveillance system 100 according to the present invention. As shown in this figure, the surveillance system 100 according to the present invention is configured to include a fixed camera 101, mask processing means 102, representative image generation means 103, matching processing means 104, and changed area extraction means 105, and can also be configured to include directional noise extraction means 106 and image storage means 107.

Of the various means constituting the monitoring system 100, the system is configured to include a mask processing means 102, a representative image generating means 103, a matching processing means 104, and a change area extracting means 105, and further, the directional noise extracting means 106 can be manufactured as a dedicated means, or a general-purpose computer device can be used. In other words, the computer device executes arithmetic processing according to a specified program, thereby performing processing specific to each means. This computer device can be configured with a personal computer (PC), a tablet PC such as an iPad (registered trademark), a mobile terminal including a smartphone, or a PDA (Personal Data Assistance), etc. The computer device is equipped with a processor such as a CPU, memories such as ROM and RAM, and some also include input means such as a mouse and keyboard, and a display.

The main elements that make up the monitoring system 100 of the present invention are explained in detail below.

(Fixed camera and image storage means)
The fixed camera 101 constituting the monitoring system 100 can be a conventional camera, such as a trail camera for photographing wild animals. The fixed camera 101 is preferably one that automatically photographs without human operation, and may be set to photograph periodically (or irregularly or intermittently) at intervals of, for example, one hour or 30 minutes. The fixed camera 101 is installed at a position (one point) where it can photograph the slope as an object from above, and photographs the same direction from the same position, so that images of the same range of the slope are basically obtained. Note that, when photographing automatically over a relatively long period of time (several months to about a year), a corresponding battery is prepared.

Since the fixed camera 101 takes pictures periodically over a long period of time, the number of images acquired is enormous. These images are stored in the image storage means 107. The image storage means 107 can be constructed, for example, in a database server, and can be placed in the vicinity of the fixed camera 101 and connected (for image data communication) to the fixed camera 101 via a local network (LAN: Local Area Network), or can be a cloud server that stores images via the Internet (in this case, wireless communication).

(Mask Processing Means)
The mask processing means 102 is a means for dividing the image acquired by the fixed camera 101 (hereinafter referred to as the "original image") into a "mask range" and a "usage range" based on the luminance of each pixel. Therefore, the original image is converted into an image in which a luminance is assigned to each pixel (hereinafter referred to as the "grayscale image"). That is, the original image is converted into a grayscale image by assigning an appropriate luminance to each pixel according to the pixel value (e.g., RGB, etc.) of the pixel of the original image. Of course, if the original image is acquired as a grayscale image, there is no need to convert the original image into a grayscale image. Note that various conventional techniques can be used to convert into a grayscale image.

The procedure by which the mask processing means 102 separates the "mask range" and the "usage range" will be described below. First, the mask processing means 102 compares the brightness of each pixel in the grayscale image with a predetermined threshold (hereinafter referred to as the "brightness threshold"), and extracts the pixel as a "mask pixel" when the brightness exceeds (or is equal to or greater than) the brightness threshold. Pixels with a fairly high brightness can be assumed to be hazy or cloudy areas, and as will be described later, are not useful when matching parts of an object, but rather act as noise. Therefore, mask pixels are extracted as candidates to be removed from the image. For example, FIG. 2(a) shows an original image consisting of 9 x 6 pixels, of which 9 shaded "mask pixels" have been extracted.

When a mask pixel is extracted, the mask pixel can be extracted as it is, that is, for each mask pixel, as the "mask range". Alternatively, when the number of mask pixels (hereinafter referred to as the "number of clustered pixels") constituting an area where adjacent mask pixels are clustered (hereinafter simply referred to as the "mask cluster area") exceeds a predetermined threshold (hereinafter referred to as the "cluster threshold"), the mask cluster area can be extracted as the "mask range". For example, in the case of Figure 2(a), a mask cluster area with a clustered pixel count of 5 (upper left of the image), a mask cluster area with a clustered pixel count of 2 (lower right of the image), and a mask cluster area with a clustered pixel count of 1 (2 locations) are formed, and the cluster threshold is set to 3. Therefore, in this case, only the mask cluster area with a clustered pixel count of 5 is extracted as the mask range.

The mask processing means 102 also sets the portion of the grayscale image excluding the mask range as the "used range" and generates an image in which the mask range has been removed from the grayscale image, in other words, an image consisting only of the used range (hereinafter referred to as the "processed image"). For example, in the case of Figure 2, a mask aggregate area with 5 aggregate pixels has been extracted as the mask range, and therefore a processed image is generated in which this mask range has been removed from the grayscale image, as shown in Figure 2(b).

(Representative image generation means)
The representative image generating means 103 is a means for generating a period representative image based on a plurality of single photographs (i.e., period images) taken by the fixed camera 101 during a shooting period. FIG. 4 is a model diagram showing a representative image. However, the period image used to generate the period representative image is a processed image from which the mask range has been removed as shown in FIG. The procedure by which the representative image generating means 103 generates a period representative image will now be described.

First, multiple period images (processed images) included in the same period image are read, and the luminance of the pixels that make up the period image (i.e., the range of use) is read. Then, the average value (hereinafter referred to as the "average luminance value") is calculated using the luminance of pixels at the same position in the multiple period images. For example, the pixel in the upper left corner is extracted as many times as the number of period images, the luminance of each pixel is summed up, and the average luminance value for the pixel in the upper left corner is calculated by dividing the sum by the number of period images. However, since all of the period images used here are processed images with the mask range removed, there may be no pixels corresponding to the period image depending on the pixel position. In other words, the shape of the processed image shown in FIG. 2(b) differs even for period images included in the same shooting period, and the number of pixels used to calculate the average luminance value also differs depending on the position. Therefore, the luminance of all period images is not used to calculate the average luminance value.

Once the average luminance value is obtained for all pixels that make up the original image (9 x 6 pixels in the case of Figure 2), a representative image for the period is generated by assigning the average luminance value to each pixel. Of course, a representative image for the period is generated for each shooting period. Note that it is also possible to set shooting periods without overlapping as shown in Figure 3 and generate a representative image for the period, or to set shooting periods with some overlapping and generate a representative image for the period.

(Matching Processing Means)
The matching processing means 104 is a means for performing pattern matching between period representative images relating to two time periods and extracting a portion common to the period representative images (hereinafter referred to as a "common area"). Specifically, two period representative images generated based on shooting periods of different time periods are compared and pattern matching is performed to extract a common area. For example, in FIG. 4, as a result of performing pattern matching using a period representative image relating to a first time period (upper figure) and a period representative image relating to a second time period (lower figure), a convex common area consisting of seven pixels is extracted.

(Changed area extraction means)
The changed area extraction means 105 is a means for obtaining a movement vector related to the common area and extracting a "changed area" from the common area. As shown in FIG. 4, the position of the common area in the period representative image related to the first period is different from the position of the common area in the period representative image related to the second period, that is, it is inferred that the common area has moved. If the common area has moved to a considerable extent, it is extracted as a "changed area" to be monitored. Therefore, as shown in FIG. 5, the changed area extraction means 105 obtains a movement vector consisting of the distance (hereinafter referred to as "movement amount") and the direction (hereinafter referred to as "movement direction") based on the position of each common area, that is, by connecting the same positions of the common areas. In this case, it is preferable to obtain a movement vector by setting the common area related to the earlier period (the first period in FIG. 5) as the starting point and the common area related to the later period (the second period in FIG. 5) as the end point.

Once the movement vector is obtained, the change area extraction means 105 extracts the change area based on the amount of movement. Specifically, the amount of movement is compared with a predetermined threshold (hereinafter referred to as the "movement amount threshold"), and when the amount of movement exceeds (or is equal to or greater than) the movement amount threshold, the common area related to that amount of movement is extracted as the change area.

(Directional noise extraction means)
The directional noise extraction means 106 is a means for extracting, as noise (hereinafter referred to as "directional noise"), common areas in which the movement direction of the common areas in two periods is within a range of a predetermined threshold (hereinafter referred to as "directional threshold"). Even in a common area in which the movement amount exceeds the movement amount threshold, a movement vector that is hardly considered as a slope movement may occur. For example, a part of an abnormal slope such as a shallow collapse, a deep collapse, a landslide, or a mudslide should move in the direction of gravity (i.e., downward). When monitoring a slope as a target, it is desirable to exclude at least common areas that have moved in a direction against gravity (vertically upward) (so to speak, candidates for changed areas).

Therefore, it was decided that common areas such as the above would be excluded from the change area as directional noise. Specifically, if the movement direction of the movement vector is generally upward, the common area is determined to be directional noise and is not extracted as a change area. Note that, when determining that the displacement direction of the displacement vector is generally upward, a reference directional threshold is determined in advance, and when the displacement direction of the displacement vector is within the range of this directional threshold, it can be determined to be generally upward. This directional threshold can be a range set with a specified width on both sides of the vertical upward direction, for example, as shown in Figure 6.

(Processing flow)
Below, the main processing of the monitoring system 100 of the present invention will be described in detail with reference to Fig. 7. Fig. 7 is a flow diagram showing an example of the flow of main processing of the monitoring system of the present invention, in which the central column shows the processing to be performed, the left column shows input information required for that processing, and the right column shows output information resulting from that processing.

When the fixed camera 101 acquires an original image periodically (or irregularly or intermittently), for example at hourly or 30-minute intervals (Step 210 in FIG. 7), the image is stored in the image storage means 107. The mask processing means 102 then creates a grayscale image based on the original image stored in the image storage means 107, and classifies the grayscale image into a "mask range" and a "usage range" (Step 220 in FIG. 7). However, as mentioned above, if the original image has been acquired as grayscale, there is no need to convert the original image into a grayscale image.

When the grayscale image is classified into a mask range and a use range, the mask processing means 102 extracts the "mask range." At this time, the mask pixels can be extracted as the mask range as is, or when the number of pixels constituting a mask set area exceeds the cluster threshold, the mask set area can be extracted as the mask range. Then, when the mask range is extracted, the mask processing means 102 removes the mask range from the grayscale image to generate a "processed image" (Step 230 in FIG. 7).

Once the processed image has been generated, the representative image generating means 103 generates a "period representative image" based on the multiple period images captured during the capture period (Step 240 in FIG. 7). Once the period representative image has been generated, the matching processing means 104 extracts a "common area" by performing pattern matching between the period representative images relating to the two periods (Step 250 in FIG. 7). Next, the changed area extraction means 105 determines the movement vector relating to the common area.

Once the movement vectors for the common regions are obtained, the directional noise extraction means 106 extracts the common regions whose movement direction is within the range of the directional threshold as "directional noise" (Step 260 in FIG. 7). The changed region extraction means 105 then excludes the common regions determined to be directional noise, and extracts the common regions whose movement amount exceeds the movement amount threshold as "changed regions" (Step 270 in FIG. 7).

The monitoring system of the present invention can be used to determine changes in natural slopes, cut slopes, and embankment slopes, as well as concrete structures such as concrete dams, reclaimed land, and soft ground.

REFERENCE SIGNS LIST 100 Surveillance system 101 Fixed camera (of surveillance system) 102 Mask processing means (of surveillance system) 103 Representative image generation means (of surveillance system) 104 Matching processing means (of surveillance system) 105 Changed area extraction means (of surveillance system) 106 Directional noise extraction means (of surveillance system) 107 Image storage means (of surveillance system)

Claims (2)

A fixed camera is installed at one location and periodically or intermittently captures images of an object;
a mask processing means for comparing the brightness of the image captured by the fixed camera with a predetermined brightness threshold, and setting a portion of the image whose brightness exceeds the brightness threshold as a mask range, and setting a portion of the image excluding the mask range as a usage range;
a representative image generating means for generating a period representative image based on a period image that is a compilation of a plurality of images captured by the fixed camera during a predetermined photographing period;
a matching processing means for performing pattern matching between the period representative images relating to two periods and extracting a common portion between the period representative images as a common region;
a changed area extraction means for extracting the common area as a changed area when the position of the common area in the period representative image differs between the two periods and the amount of movement between the common areas exceeds a predetermined movement amount threshold value,
the mask processing means sets pixels in the image whose luminance exceeds the luminance threshold as mask pixels, and sets a cluster portion in which the number of the mask pixels exceeds a predetermined cluster threshold as the mask range;
the representative image generating means calculates an average luminance value, which is an average of luminance values of the same pixel, by using the range of use of the period image, and generates the period representative image by assigning the average luminance value to each pixel;
A monitoring system comprising: A directional noise extraction means extracts, as a directional noise, a common area in which a moving direction of the common area in two time periods is within a range of a predetermined directional threshold value;
The direction threshold is a range set with a predetermined width from a vertical upward direction,
the change region extraction means extracts the change region by removing the directional noise from the common region;
2. The monitoring system according to claim 1 .

Images