JP5865709B2 - Image monitoring device - Google Patents

Description

  The present invention relates to an image monitoring apparatus that detects a change in arrangement that occurs when an arrangement placed in a monitoring space is moved by a person or the like.

  In an image sensor that performs action recognition by tracking a change area in a monitoring image, if the person being tracked moves an object (chair, trolley, etc.), a change area in which the person and the object are integrated is detected. The For this reason, image characteristics of people and arrangements are mixed, the size and shape of the change area are changed, causing tracking accuracy to be reduced, and image processing such as mask processing for privacy protection is performed only on the person area It was difficult.

  In the conventional technology, a person changes his / her chair area by waiting for the person to leave the chair and the areas of change are separated (Patent Document 1). The change area of the chair is distinguished from the change area of the person by utilizing the difference in motion generated between the person and the chair so as to move as it is (Patent Document 2).

JP 2011-70332 A JP 2010-176206 A

  However, in the prior art, it was impossible to distinguish between the person and the arrangement object until the person or the arrangement object moved away or until a difference in movement between the person and the arrangement object occurred. For this reason, until the person moves his hand after the placement object starts moving, the tracking accuracy is reduced, and different processing cannot be performed on the area of the person and the placement object.

  The present invention has been made in view of the above problems, and an object of the present invention is to detect that an arrangement change has occurred from the time when an arrangement object starts to move. Another object of the present invention is to detect a region in which an image change has occurred due to movement of an arrangement object from the time when the arrangement object starts to move.

  An image monitoring apparatus according to the present invention detects an arrangement change of an arrangement placed in the monitoring space using input images that are sequentially captured and input in the monitoring space. A storage unit that stores a background image of the monitoring space in which the background is captured, and a background corresponding point on the background image side corresponding to the background image and the input image as having similar image characteristics to each other And a background flow detection unit that detects an input corresponding point on the input image side and obtains a background flow from a vector from the background corresponding point to the input corresponding point, and the background flow indicates a movement amount equal to or greater than a predetermined value. An arrangement change detection unit that detects occurrence of the arrangement change.

  In another image monitoring apparatus according to the present invention, the background flow detection unit is configured such that an image feature between the input image and a preceding input image captured at a preceding time close to a capturing time of the input image. Is detected as a preceding input corresponding point on the preceding input image side, and the vector using the background corresponding point located at substantially the same position as the preceding input corresponding point is defined as the background flow.

  In the configuration, the background flow detection unit calculates the vector using the background corresponding point where the image feature of the preceding input image and the image feature of the background image substantially coincide with each other at the position of the preceding input corresponding point. The background flow can be used.

  In still another image monitoring apparatus according to the present invention, the background flow detection unit detects, as the background flow, a vector having a predetermined number or more of the vectors whose directions substantially coincide with each other.

  In another image monitoring apparatus according to the present invention, each time the input image is input, the background flow detection unit reduces the background corresponding point candidates or the input corresponding point candidates to a predetermined image range at a low density. Sparse background flow detection means for setting and obtaining the background flow; and when the occurrence of the arrangement change is detected, the background corresponding point candidate or the input corresponding point candidate is displayed in the image more than the sparse background flow detection means. Dense background flow detection means for obtaining the background flow by setting a high density in a range, and the arrangement change detection unit detects the occurrence of the arrangement change using the background flow detected by the sparse background flow detection means. And a set of input corresponding points constituting the background flow detected by the dense background flow detection means is detected as a movement destination area of the arrangement.

  According to the present invention, it is possible to detect an image change due to the movement of an arrangement from the movement start time. Further, according to the present invention, it is possible to detect a region where an image change has occurred due to the movement of the arrangement object from the time when the movement starts.

1 is a block diagram illustrating a schematic configuration of an image monitoring apparatus according to an embodiment of the present invention. It is a schematic diagram of an example of the image which extracted the blob and set the attention point. It is a schematic diagram explaining the process of a background flow detection part. It is a flowchart which shows the operation | movement of the outline of the image monitoring apparatus which concerns on embodiment of this invention. It is a general | schematic flowchart of a background flow detection process. It is a schematic flowchart of an arrangement | positioning change detection process. It is a typical image which shows the example when the change area of a movement origin and the change area of a movement destination are the same, and different. It is a schematic diagram of the example of the change area extracted corresponding to the input image in three continuous times which the change area of the arrangement under tracking appears.

  Hereinafter, an image monitoring apparatus 1 according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

  In the monitoring space monitored by the image monitoring apparatus 1, there are articles arranged in advance (hereinafter referred to as arrangement objects), and people enter and exit. For example, the monitoring space is a room, and the arrangement is equipment or facilities in the monitoring space, or temporarily placed items. As an example, a chair or a carriage is the arrangement.

  An arrangement is normally stationary and forms part of the background of the surveillance space, but it may be temporarily detected as a moving object when it is moved by a person. The arrangement changes. The image monitoring apparatus 1 detects this change in arrangement. Note that the arrangement change includes a change that falls within the field of view of the image monitoring apparatus 1 and a change in which the arrangement object is taken away and moved out of the field of view.

[Configuration of Image Monitoring Apparatus 1]
FIG. 1 is a block diagram illustrating a schematic configuration of the image monitoring apparatus 1. The image monitoring apparatus 1 includes an imaging unit 2, a storage unit 3, a control unit 4, and an output unit 5, and the imaging unit 2, the storage unit 3, and the output unit 5 are connected to the control unit 4.

  The imaging unit 2 is a surveillance camera and photographs the surveillance space at predetermined time intervals. The captured surveillance images (input images) of the surveillance space are sequentially input to the control unit 4. In order to grasp the position and movement of an arrangement object or a person using a background image, the imaging unit 2 is installed with a field of view fixed at a height at which a bird's-eye view can be taken. For example, in this embodiment, the imaging unit 2 is fixedly installed on the ceiling of the room. The time interval at which the monitoring image is captured is 1/5 second, for example. Hereinafter, the unit of time recorded at the imaging time interval is referred to as time, the latest imaging time is referred to as the current time, and the input image at the current time is referred to as the current image.

  The storage unit 3 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 3 stores various programs and various data, and inputs / outputs such information to / from the control unit 4. Various data includes a background image 30, a previous image 31, arrangement information 32, and area information 33.

  The background image 30 is an image that includes only the background image of the monitoring space and does not include the human image. The background image 30 is generated and stored in the storage unit 3 prior to the arrangement change detection process and the arrangement tracking process. Specifically, an input image captured in a state where a moving object such as a person does not exist in the monitoring space can be used as the background image 30, and the background image 30 includes an object that is arranged in advance in the monitoring space. An image can be included. The background image 30 is appropriately updated during the process of detecting the change of arrangement and the process of tracking the arrangement.

  The immediately preceding image 31 (preceding input image) is an input image one hour before the current time (preceding time close to the current image capturing time), and the current image input at each time is stored as the immediately preceding image 31 at the next time. Part 3 is written.

  In this embodiment, the time difference between the current image and the preceding input image is one time. However, the time difference of about several hours included in the preceding input image by the person who moves the placement object can also be used.

  The arrangement information 32 includes the identification number of each arrangement, the state of the arrangement (arrangement state), the position and area of the arrangement in the monitoring image, and the image characteristics of the arrangement. When the arrangement is moved and the arrangement is detected, arrangement information 32 of the arrangement is created. While the arrangement is moving, the image monitoring apparatus 1 tracks the position of the arrangement, and the position history is stored as the arrangement information 32. Tracking and completion of movement are defined as attributes indicating the arrangement state. The image feature includes a feature point representing a characteristic part of the image of the arrangement, in addition to a color histogram representing the characteristics of the entire image of the arrangement.

  The area information 33 includes an identification number of each change area and information indicating whether or not the change area includes an arrangement.

  The control unit 4 is configured by using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit), and reads and executes a program from the storage unit 3. , Function as a background flow detection unit 41, an arrangement change detection unit 42, a background image generation unit 43, a mask image generation unit 44, and the like.

  The change area extraction unit 40 compares the input image and the background image, extracts a change area between them, and outputs information on the extracted change area to the background flow detection unit 41 and the arrangement change detection unit 42. The change area can be extracted by a known background difference process or background correlation process. That is, the degree of difference (difference or correlation value) of the pixel values at the same coordinates between the input image and the background image is calculated, and a pixel whose degree of difference exceeds the threshold value is detected as a change pixel. Then, change pixels adjacent to each other are collected, and the combined area is extracted as a change area. The detection threshold is set in advance based on a prior experiment or the like. Note that an area of about 1 to several pixels is excluded from the extraction target because it is noise.

  The extracted change area includes a change area caused by an arrangement change of the arrangement object and a change area caused by the presence and movement of a person. While the arrangement change of the arrangement by the person is in progress, a change area in which the arrangement and the person are integrated is extracted. Further, as a change area derived from the movement of the arrangement object, an area where the arrangement object exists in the background image 30 (movement source area) is extracted in addition to an area where the arrangement object exists in the current image (movement destination area). The Hereinafter, the movement source region that is the trace of the image of the arrangement object among the change areas derived from the arrangement object is referred to as a ghost. Here, at the initial stage of the arrangement change, a change area in which the arrangement object, the person, and the ghost are integrated can be extracted.

  The background flow detection unit 41 detects corresponding points having similar image characteristics between the background image 30 and the current image, and detects a background flow that is a motion vector connecting the corresponding points. Both the background image 30 and the current image include an image of an arrangement, and when the arrangement moves, a movement amount (motion vector) corresponding to the movement of the arrangement is calculated between corresponding points of the background image 30 and the current image. Is done. At this time, the current image includes the image of the person who is moving the arrangement, but the background image 30 does not include the image of the person, and thus the background flow of the person is basically not detected. Further, since the field of view of the imaging unit 2 is fixed, only a movement amount of an error that can be regarded as 0 or 0 is calculated from the background other than the arrangement object. Therefore, if the movement amount obtained by the background flow detection unit 41 is used, the presence / absence of an arrangement change can be determined with high accuracy. As an image feature, a color distribution, a luminance gradient, or a light / dark distribution in a block of a predetermined size (for example, 5 × 5 pixels) centered on a candidate for the attention point and the most similar point can be used.

  Specifically, the background flow detection unit 41 sets a plurality of attention points (background corresponding point candidates or input corresponding point candidates) on one of the background image 30 and the current image (reference image). At the same time, in the other image (comparison image), the most similar point having the most similar image point and image feature is detected, and a vector connecting the attention point and the most similar point is calculated. Incidentally, the corresponding point on the background image 30 side (background corresponding point) is the start point of the background flow, and the corresponding point on the current image side (input corresponding point) is the end point.

  The background flow detection unit 41 calculates a background flow and outputs the background flow to the arrangement change detection unit 42 every time a current image is input from the imaging unit 2.

  Here, the motion vector calculated between the immediately preceding image 31 and the current image is generally called an optical flow, but the background flow is different from the general optical flow in that it is calculated between the background image 30 and the current image. . Hereinafter, a general optical flow calculated between the immediately preceding image 31 and the current image is referred to as an input flow in distinction from the background flow.

  A Hessian-Laplace detector is used to set the point of interest. A blob is extracted by applying the detector to the reference image, and the extracted blob is set as a point of interest. A blob is one of feature points (key points) of an image. Specifically, the blob can be extracted corresponding to the maximum point and the minimum point of luminance, and the feature point coordinates and the feature point corresponding to the blob can be extracted. The luminance distribution of the local area centered on the coordinates is obtained.

  Since many blobs are extracted in the closed region, it is possible to set an effective number of attention points in order to detect a change in the arrangement in the arrangement object or the human area. FIG. 2 is a schematic diagram of an example of an image in which attention points are set by extracting a blob, and attention points set for a person and a chair existing in the monitoring space are indicated by black dots.

  Setting attention points at locations with few features makes it easier to detect erroneous corresponding points, but setting feature points as attention points as in this embodiment suppresses erroneous detection of corresponding points, and The amount of processing can also be reduced.

  In addition, since the purpose of this embodiment is to detect a change in arrangement, the background flow detection unit 41 is not in the whole image but in a portion where an image change has occurred, that is, inside the change region obtained by the change region extraction unit 40. It is possible to adopt a configuration in which the attention point is set and the attention point is not set outside the change region. Thereby, errors that erroneously detect background flows other than the arrangement object are reduced, and the processing amount can be reduced.

  The background flow detection unit 41 can search for the most similar point by a search process on the comparison image. In this configuration in which blob is set as the point of interest, the candidate point is moved on the comparison image, the similarity of the luminance distribution with the point of interest is calculated, and the pair of the candidate point and the point of interest having the highest degree of similarity is calculated. Decide on corresponding points. For this search, various steepest descent methods such as the Lucas-Kanade method can be applied. That is, the search is performed by sequentially moving the candidate points in the direction in which the difference in luminance is reduced, with the same coordinates as each point of interest as the initial value of the candidate points.

  In another embodiment, the search is performed by setting each coordinate of a predetermined search range including the coordinates of the point of interest as a candidate point. In this case, the search range can be the entire comparison image, but since the purpose is to detect a change in arrangement, the distance on the image that the arrangement can move in one time is estimated in advance. The search range is preferably within a circle centered on the coordinates of the distance and the radius of the distance, and further inside the change region obtained by the change region extraction unit 40. That is, an overlapping area between the circle and the change area is set as a search range. By doing so, erroneous detection of corresponding points can be prevented, and the processing amount of corresponding point detection can be reduced.

  Here, as described above, the input image at the time of moving the arrangement includes a human image, and the human image on the input image has characteristics similar to a part of the background image 30, The background flow may be erroneously detected for the person. If the background flow that has been erroneously detected in this way is used for detection of an arrangement change, an erroneous detection that there is an arrangement change may occur when there is no arrangement change.

  On the other hand, both the current image at the start of movement of the arrangement and the immediately preceding image 31 immediately before that include images of the same person, and it can be expected that the images of the same person are more similar to the person and the background. Therefore, compared to the background flow, the motion vector is less likely to be erroneously detected between the person and the background in the input flow.

  Due to this property, if only the background flow that matches the input flow is used for detecting the change in arrangement (start of arrangement movement), the erroneous detection of the change in arrangement due to the erroneously detected background flow can be prevented. The image monitoring apparatus 1 according to the present embodiment has the above configuration, and the background flow detection unit 41 includes an input flow detection unit 410 that detects an input flow and a background flow selection unit 411 that selects a background flow that matches the input flow. Is provided. Hereinafter, these will be described.

  The input flow detection unit 410 detects corresponding points having similar image characteristics between the immediately preceding image 31 (preceding input image) and the current image (subsequent input image), which are successive input images. Specifically, among the current image and the immediately preceding image 31, a plurality of attention points are set on one image, and the most similar point whose feature is most similar to each attention point in the other image is detected. A motion vector between the most similar points is calculated and used as an input flow. Incidentally, the corresponding point on the immediately preceding image 31 side (preceding input corresponding point) is the start point of the input flow, and the corresponding point on the current image side (second input corresponding point) is the end point. The calculated input flow is output to the background flow selection unit 411.

  Note that the attention point setting and the most similar point detection can be performed in the same manner as in the background flow detection described above. That is, a blob is extracted by applying a Hessian Laplace detector to one of the images, and the extracted blob is set as a point of interest. Further, the most similar point having the most similar feature to each attention point is searched on the other image using the Lucas-Kanade method or the like. When a search method in which a search range is determined is adopted, the search range can be set within a circle whose radius is the distance on the image at which the object can move during one time.

  The background flow selection unit 411 selects a background flow that matches the input flow detected by the input flow detection unit 410.

  Here, it is possible to perform background flow detection separately from the input flow detection, and to determine whether the background flow obtained thereby matches the input flow. In that case, in order to detect the background flow In addition, a feature point extraction process and a most similar point search process with a large amount of processing must be performed.

  Therefore, paying attention to the fact that the background flow to be detected for the detection of the arrangement change is included in the input flow, and that the current image is common among the background flow and the original data of the input flow, the background of this embodiment In the flow selection unit 411, the selection method is devised to reduce the processing load. Specifically, the background flow selection unit 411 determines whether or not the image feature of the immediately preceding image 31 at the start position of each input flow matches the image feature of the background image 30 at the position, A motion vector corresponding to an input flow determined to match is defined as a background flow. That is, the start and end of the background flow are defined based on the start and end of the input flow. Considering that a minute change due to illumination fluctuation or noise may occur between the immediately preceding image 31 and the background image 30, a slight deviation (for example, for one pixel) at the start end is allowed and the coincidence determination is performed.

  FIG. 3 is a schematic diagram for explaining the processing of the background flow detection unit 41, and shows the background flow detection processing based on specific examples of the background image 200, the immediately preceding image 210, and the current image 220.

  The background image 200 includes a chair 201 (image 201a) as an arrangement, and the background image 200 includes a background object 202 (image 202a) that can cause erroneous detection.

  In addition to the chair 201 and the background object 202, the person 203 is shown in the immediately preceding image 210 and the current image 220. The person 203 has a portion having a color similar to that of the background object 202. In the immediately preceding image 210, the arrangement has not changed yet, the position of the chair 201 (image 201b) is the same as that of the background image 200, and the person 203 (image 203b) is not yet in contact with the chair 201. On the other hand, the current image 220 is a state in which a person 203 (image 203c) is in contact with the chair 201, and the chair 201 (image 201c) has started to change its arrangement. The background object 202 (image 202c) remains in its original position.

  In the schematic images 230 and 240, the background flow and the input flow are indicated by arrows, respectively. Further, the images 230 and 240 also show the change area. Specifically, the change area 231 in which the chair 201 and the person 203 are integrated is extracted from the current image 220 as the change area, and the ghost 232 of the chair 201 is extracted from the background image 200. In addition, a change area 233 of the person 203 is extracted from the immediately preceding image 210.

  In the background flow shown in the image 230, in addition to the background flow 234 caused by the movement of the chair 201, an erroneous background flow 235 is detected between the chair 201 and the person 203. An incorrect background flow 236 has been detected.

  In the input flow shown in the image 240, an input flow 241 due to the movement of the chair 201 and an input flow 242 due to the movement of the person 203 are detected. Here, since the degree of similarity between the people 203 is higher than the degree of similarity between the chair 201 and the person 203, an input flow corresponding to the erroneous background flow 235 is not detected. Further, since the degree of similarity between the people 203 is higher than the degree of similarity between the background object 202 and the person 203, an input flow corresponding to the erroneous background flow 236 is not detected.

  Incidentally, the input flow 241 of the chair 201 and the input flow 242 of the person 203 are detected in substantially the same direction and size at positions close to each other. Therefore, it is difficult to distinguish the input flow of the arrangement from the human input flow. Therefore, it is difficult to detect a change in arrangement only from the input flow.

  A schematic image 250 shows the background flow shown in the image 230 that matches the input flow shown in the image 240, and the background flow is detected by the background flow detection unit 41. That is, the background flow 234 caused by the movement of the chair 201 in the background flow shown in the image 230 is in common with the input flow 241 caused by the movement of the chair 201 shown in the image 240, and is selected as the background flow 251. On the other hand, erroneous background flows 235 and 236 shown in the image 230 are excluded because they are not in common with the input flow of the image 240.

  Note that, as described above, the background flow and the input flow can be generated separately, and the common flow of them can be selected as the background flow. However, the background flow selection unit 411 of the present embodiment has the configuration of the input flow. Of these, a configuration in which the starting edge has the same image characteristics as the same position of the background image 200 is selected as the background flow, and the processing load associated with the background flow detection is reduced. With this configuration, the processing load for obtaining the background flow 234 is reduced, and waste of detecting erroneous background flows 235 and 236 that are motion vectors that do not occur in the input flow is eliminated.

  Here, since the background object 202 does not change the position and brightness and does not generate a change region, the background flow and the input flow are separately generated and the common flow is used as the background flow. If the above-described process is performed to limit the search range of the most similar point to the change region, erroneous detection of the background flow 236 can be prevented.

  The arrangement change detection unit 42 determines that the arrangement has started to move, that is, an arrangement change has occurred when a background flow that exceeds a preset reference is detected by the background flow detection unit 41. This determination is performed at each time using a sparse background flow. On the other hand, when the background flow above the reference is not detected, it is determined that the arrangement change has not occurred.

For example, any one of the following (1) to (4) can be used as the determination criterion for the arrangement change detection. These criteria are more reliable the later.
(1) A background flow in which the movement amount, that is, the magnitude of the motion vector is equal to or greater than the motion detection threshold Ts is detected. Here, Ts is a value for excluding a background flow of an error level detected by a minute illumination variation, and can be set to a size corresponding to a few pixels, for example.
(2) A plurality of background flows in the same direction are detected when the movement amount is Ts or more.
(3) A background flow having a movement amount equal to or greater than Ts has been detected a predetermined number Tn or more. Here, Tn is ideally the total number of points of interest set in the arrangement, but it can be set to a constant ratio so that arrangement changes can be detected even when occlusion occurs. Moreover, the estimated value estimated beforehand based on an experimental result etc. can be used as the said total. Further, the fixed ratio can be about 30%. In the experiment, Tn was set to 10 and it was possible to detect the change in arrangement with high accuracy.
(4) A background flow in the same direction is detected by a predetermined number Tn or more when the movement amount is Ts or more from the same change region.

  The arrangement change detection unit 42 has a function other than the function of detecting the occurrence of the arrangement change. Specifically, the function of detecting the area of the arrangement from the change area in which the person and the arrangement are integrated. An arrangement object detection unit 420 having a function, a ghost detection part 421 having a function of detecting a ghost from the change area, and an arrangement object tracking part 422 having a function of tracking the progress of the detected arrangement change are provided.

  The arrangement object area detection unit 420 identifies the area of the arrangement object based on the background flow or the input flow for each pixel. The background flow or the input flow for each pixel is detected by the background flow detection unit 41 and used for processing by the arrangement change detection unit 42. That is, the background flow detection unit 41 performs the input flow and background flow detection processing using the feature points of the image described above as the attention point at each time to detect the arrangement change, and also by the arrangement change detection unit 42. When the start of movement of the arrangement is detected, the input flow and background flow detection processing in which each pixel is set as a point of interest is performed. The motion vector detected for each pixel has a higher spatial density than that detected for each feature point. Therefore, the background flow and the input flow used for the processing in the arrangement change detection unit 42 are referred to as a dense background flow and a dense input flow. On the other hand, the background flow and the input flow used for detecting the occurrence of the arrangement change have a relatively low density, and can be called a sparse background flow and a sparse input flow. As described above, the motion vector is densely obtained only when the arrangement change is detected, and the motion vector is sparsely obtained at the time when the arrangement change is not detected, thereby reducing the load of the motion vector detection processing. .

  When the movement start of the arrangement is detected, the arrangement area detection unit 420 detects the arrangement area that is the movement destination area of the arrangement based on the corresponding point on the current image side of the background flow, that is, the position of the end. To do. In the present embodiment, since the above-described dense background flow is used for this processing, a set of pixels at the end corresponds to an arrangement object region. That is, the arrangement object area detection unit 420 detects a collection of end points of the dense background flow detected in the change area between the current image and the background image as the arrangement area at the current time. Then, a new identification number and an attribute “tracking” are assigned to the detected arrangement area and registered in the arrangement information 32.

  Further, when the arrangement object is being tracked, the arrangement area detection unit 420 extracts a dense input flow obtained from the background flow detection unit 41 that has a start in the arrangement area detected in the immediately preceding image. Then, a set of terminations detected in the change area in the current image for the extracted input flow is detected as an arrangement area at the current time. At that time, the zero vector input flow is also used for detection in order to detect the area of the stopped arrangement.

  The ghost detection unit 421 determines a portion including a corresponding point on the background image side of the background flow as a ghost region in the change region between the current image and the background image. That is, the change area in which the starting edge of the background flow is detected is detected as a ghost. At this time, the ghost detection unit 421 determines whether the background flow detected by the background flow detection unit 41 is within a single change area and determines whether the background area of the ghost is divided. Detect efficiently.

  When the start and end of the background flow are in the same change area, the arrangement object area or the person area and the ghost are integrated. In this case, the ghost detection unit 421 obtains a dense background flow in which each pixel in the change region is set as the start point from the background flow detection unit 41, and the end point of the start point of the background flow detected in the change region. Detect a gathering as a ghost.

  On the other hand, when the start end and the end of the background flow are separated in different change areas, the arrangement object area or the person area and the ghost are separated. In this case, the ghost detection unit 421 detects, as a ghost, a change area in which the starting edge of the background flow detected by the background flow detection unit 41 is detected. In this case, the background flow is used to specify which of the plurality of change areas is the ghost area, and does not need to be a dense background flow that can specify the shape of the ghost area. It ’s enough.

  The arrangement tracking unit 422 associates the arrangement area detected from the previous image and the arrangement area detected from the current image as the same arrangement if they are connected by the input flow. When a plurality of arrangement areas are detected in one or both of the two images, the association is determined for each combination of arrangement areas between the two images. Note that an input flow of a zero vector is also used for association in order to associate the stopped arrangement.

  Also, the arrangement tracking unit 422 compares the positions and shapes of the change areas including the same arrangement, and determines that the arrangement has been moved within the field of view when the positions and shapes are substantially the same. In this case, an attribute of “movement completed within the field of view” is given as the arrangement state of the arrangement information 32, and the arrangement is excluded from the tracking target. Whether or not they are the same position / shape can be determined based on, for example, the degree of overlap between the two change regions, and the degree of overlap can be thresholded, and the case where the degree of overlap is high can be made the same position / shape.

  On the other hand, when the arrangement area corresponding to the arrangement area detected from the immediately preceding image is not detected from the current image, the arrangement tracking section 422 determines that the arrangement has been moved out of the field of view. In this case, an attribute of “moving out of the field of view” is given as the arrangement state, and the object is not tracked. In this case, an abnormal signal indicating the take-out may be output to the output unit 5.

  The background image generation unit 43 registers the unmanned input image as the background image 30 as an initialization process when the image monitoring apparatus 1 is activated. The background image generation unit 43 functions as a background image update unit that appropriately updates the background image 30 using the input image during the operation of the image monitoring apparatus 1. Specifically, for areas other than the change area between the current image and the background image 30, the background image 30 is replaced with the current image, or the background image 30 and the current image are updated by weighted averaging. Can do.

  Further, the background image generation unit 43 as the background image update unit updates the ghost region in the background image 30 where the arrangement object is absent in the current image with the current image. Specifically, when completion of movement of the arrangement is detected, the arrangement area in the background image 30 is replaced with the current image and updated. Thus, after the movement is completed, the arrangement object area can be prevented from being extracted as a change area. Further, when it is detected that the ghost area is separated from the arrangement object area and the person area in the current image, the ghost area in the background image 30 is replaced with the current image and updated, and thereafter, the area becomes a change area. It can be prevented from being extracted.

  In addition, in the update process of the background image 30 related to the arrangement object region described above, the arrangement image may not be included in the background image 30, but once the arrangement change is detected, the arrangement image is distinguished from the background. Since the position and shape are grasped, it is possible to continue processing such as detection of a change in arrangement and detection of an arrangement object area using the information. On the other hand, the update may be performed so that the image of the arrangement object at the movement destination is included in the background image 30. In this case, the process such as the detection of the arrangement change and the detection of the arrangement area is continued in the same manner as after the initialization process. The

  For the purpose of protecting the privacy of the person shown in the input image, the mask image generation unit 44 performs a process of masking (including filter processing) a change area (process target area) excluding the placement object area and the ghost in the current image. The generated image (mask image) is generated and output to the output unit 5.

  Since the mask image generation unit 44 uses the changed region excluding the placement object region and the ghost as the processing target region, the monitor can visually recognize the shape of the person in the processed image and can also visually recognize the state of the placement object. Therefore, the situation can be grasped only with the processed image, and both privacy protection and monitoring can be achieved.

  The output unit 5 includes a sound output unit such as a speaker or a buzzer that outputs a warning sound in response to an abnormal signal, a display unit such as a liquid crystal display or a CRT that displays a privacy-processed composite image, and the like.

[Operation of the image monitoring apparatus 1]
FIG. 4 is a flowchart showing a schematic operation of the image monitoring apparatus 1.

  The image monitoring apparatus 1 is activated in a state where there is no person in the monitoring space, and the imaging unit 2 starts an operation of imaging the monitoring space every predetermined time. The background image generation unit 43 acquires or generates an image in which no person is captured from an input image captured during a preset initialization period, and initializes the background image 30 of the storage unit 3 with the image (S1). For example, the background image generation unit 43 generates the background image 30 by averaging the input images at a plurality of times in the initialization period.

  After the initialization period, the control unit 4 starts a monitoring operation using the input image acquired from the imaging unit 2. The control unit 4 sequentially receives input images (hereinafter, the current image) (S2), and repeats the processes of steps S2 to S16 each time the image is input.

  The change area extraction unit 40 compares the current image with the background image 30 and extracts a change area (S3). The control unit 4 determines whether or not the change region is extracted (S4). If the change region is not extracted (in the case of “NO” in S4), the control unit 4 omits steps S5 to S14. The processes of S15 and S16 are performed. In this case, the background image generation unit 43 updates the entire background image 30 by weighted averaging with the current image, the control unit 4 outputs the current image as it is to the output unit 5, and the output unit 5 displays the current image as it is. .

  On the other hand, when the change area is extracted (in the case of “YES” in S4), the control unit 4 advances the process to step S5, and the background flow detection unit 41 performs the process of detecting the background flow (S5).

  The background flow detection process will be described below based on the flowchart of FIG. The input flow detection unit 410 of the background flow detection unit 41 extracts blobs from the immediately preceding image 31 (S100). Then, the coordinates of the extracted blob are compared with the change area of the previous time stored as the area information 33 and the change area of the current time extracted in step S3, and the coordinates of the blob in any of the change areas are compared. The point of interest is set (S101).

  The input flow detection unit 410 searches the current image for the most similar point of interest (S103), and detects a vector starting from the attention point and ending with the most similar point as an input flow (S104). A zero vector is also detected as an input flow. Further, if only the most similar point with a similarity equal to or less than the threshold value is found in step S103, it can be assumed that the input flow corresponding to the point of interest is not detected.

  The background flow selection unit 411 selects a background flow that matches the input flow. Specifically, a predetermined deviation allowable range is set around the starting end coordinates of each input flow detected in step S104 (S105), and the attention point of the immediately preceding image 31 in the background image 30 in the deviation allowable range. (S106). The allowable shift range can be set to 3 × 3 pixels, for example, and a shift within one pixel is allowed.

  When a matching point is detected in the background image in step S106, the background flow selecting unit 411 detects a vector starting from the matching point and ending at the corresponding most similar point as a background flow (S107). On the other hand, if the attention point does not match the background image within the allowable deviation range, the background flow corresponding to the input flow starting from the attention point is not selected.

  When the background flow is detected by the background flow detection process S5 described above, the control unit 4 performs the arrangement change detection process S6 of FIG. FIG. 6 is a schematic flowchart of the arrangement change detection process S6. Hereinafter, the arrangement change detection process will be described with reference to FIG. Here, a process using the above-described arrangement change detection criterion (4) will be described.

  The arrangement change detection unit 42 compares the size of each background flow detected in step S5 with the motion detection threshold value Ts, and selects a background flow whose size is equal to or greater than Ts (S200). Then, the background flow is grouped for each change area by comparing the selected start edge coordinates of each background flow with the change area extracted in step S3 and specifying the change area including the start edge coordinates (S201). Further, the direction (inclination) of each background flow is calculated in each group, and the number of background flows that can be regarded as having the same inclination is counted (S202). Whether the directions are the same can be determined based on, for example, that the difference in the gradient of the background flow is less than a predetermined threshold value Tk. In addition, the background flow group for each change region may be statistically processed for the distribution of the slopes, and those having a statistically significant variance value or less may be used as the background flow in the same direction.

  The arrangement change detection unit 42 compares the count value obtained in step S202 with the threshold value Tn (S203). If there is a group whose count value is Tn or more (in the case of “YES” in S203), the change area extracted on the start end side of the background flow that constitutes the group is set as the movement source of the arrangement that has started moving ( In step S204, the change area extracted on the end side is set as the movement destination of the placement object that has started moving (S205). If there is no corresponding group (in the case of “NO” in S203), it is determined that there is no change area caused by the start of movement of the arrangement.

  After the arrangement change detection process S6 described above, the control unit 4 continues the process shown in FIG. 4 from step S7. The arrangement change detection unit 42 is arranged when a movement of a new arrangement is detected in step S6 or when there is an arrangement being tracked (an arrangement for which movement has been detected and tracking has not been completed by the previous time). It is determined that it is changing (in the case of “YES” in S7).

  Here, if the arrangement is changing (in the case of “NO” in S7), the control unit 4 omits steps S8 to S13 and performs the processes of steps S14 to S16. In this case, all the change areas are processed as person areas. That is, the mask image generation unit 44 performs mask processing on all the changed regions, the background image generation unit 43 updates the background image 30 other than the change regions by weighted averaging with the current image, and the control unit 4 performs the mask processing. The image is output to the output unit 5, and the output unit 5 displays the masked image.

  If it is determined in step S7 that the arrangement is being changed, the arrangement tracking unit 422 checks whether there is an arrangement being tracked (S8). If there is an arrangement being tracked (in S8, “ In the case of “YES”), the arrangement object is tracked to detect a change area corresponding to the current position (S9). On the other hand, when there is no tracked arrangement (“NO” in S8), the process of step S9 is omitted. In addition, the arrangement | positioning object by which the movement start was detected in step S6 will be tracking after the next time, and is not made into the object of tracking process S9 at the present time.

  The background flow detection unit 41 sets each pixel of the background image 30 in the change area set as the movement source of the arrangement in step S204 as an attention point for the arrangement whose movement start is detected at the current time. . Then, the most similar point of the attention point is detected from the current image in the change area set as the movement destination in step S205 (detection of dense background flow), and the arrangement object area detection unit 420 detects the detected most similar point. The gathering is detected as an arrangement object area at the current time (S10). On the other hand, for the arrangement being tracked, the background flow detection unit 41 reads the arrangement area at the immediately preceding time from the arrangement information 32 of the arrangement, and sets each pixel of the immediately preceding image 31 in the area as a point of interest. Then, the most similar point of the attention point is detected from the current image in the change area associated as the arrangement in step S9 (detection of dense input flow), and the arrangement area detection unit 420 detects the most similar. A group of points is detected as an arrangement area at the current time (S10).

  The arrangement change detection unit 42 determines separation / mixing of the change area of the arrangement in the current image with a person change area or a ghost (S11). For the arrangement in which the movement start is detected at the current time, the arrangement change detection unit 42 confirms whether the movement source change area set in step S204 is the same as or different from the movement destination change area set in step S205. Based on the result, separation / mixing determination is performed.

  FIG. 7 is a schematic image showing an example in which the movement source change area and the movement destination change area are the same or different. FIG. 7A shows a case where the movement source area 500 and the movement destination area 501a are the same change area, and the movement amount of the arrangement object whose movement start is detected is small, and the person and the arrangement object (movement destination in the current image). A change area in which the area 501a) and the ghost (movement source area 500) are integrated is extracted. The arrangement change detection unit 42 determines the change area of the arrangement in this case as “a mixed area with a person” and “a mixed area with a ghost”.

  On the other hand, FIG. 7B shows a case where the movement source area 500 and the movement destination area 501b are different change areas, and a person and an arrangement (movement destination area 501b) and a ghost (movement source area 500) in the current image. ) Are extracted as separated change areas. The arrangement change detection unit 42 determines the change area of the arrangement in this case as a “mixed area with people” and a “separated area with ghost”.

  For the arrangement being tracked, the arrangement change detection unit 42 confirms whether or not the shape of the change area at the immediately preceding time and the change area at the current time matched in step S9 match. Based on this, separation / mixing judgment is performed.

  FIG. 8 is a schematic diagram of an example of an input image at three consecutive times captured during tracking of an arrangement and an example of a change area extracted from the image, and the time advances in the order of FIGS. In each of FIGS. 4A to 4C, the left side is an input image, and the right side is an image showing a change area. When the shape of the change area associated with the arrangement does not match between the previous time and the current time, for example, the person and the arrangement shown in FIG. Immediately after the person and the arrangement are separated. In this case, the arrangement change detection unit 42 does not immediately determine the separation, but maintains the determination of the change area of the arrangement as the “mixed area with people”. On the other hand, when the shape of the change area associated with the arrangement matches the previous time and the current time, for example, when the state transitions from the state shown in FIG. 8B to the state shown in FIG. In this way, the person and the arrangement are separated at both the previous time and the current time, and can be said to be a stable separation state. In this case, the arrangement change detection unit 42 determines that the change area associated with the arrangement object is a “separated area from a person”.

  The ghost detection unit 421 performs the process of detecting the ghost region described above for an arrangement whose movement start has been detected at the current time and an arrangement that is being tracked and for which ghost separation has not been determined. S12). If the movement start of the arrangement is detected at the current time, the ghost detection unit 421 refers to the dense background flow detected in step S10 and detects the set of attention points as a ghost.

  If it is determined in step S7 that the arrangement is changing, the arrangement change detection unit 42 performs the processes in steps S8 to S12 described above, and updates the arrangement information 32 and the area information 33 based on the result (S13). . Specifically, in step S10, the arrangement area newly detected at the current time is registered in the arrangement information 32 together with the new identification number and the attribute being tracked. Also, the information on the arrangement area in the arrangement information 32 of the arrangement being tracked is updated with the arrangement area of the arrangement detected at the current time. For an arrangement for which completion of movement out of the field of view is detected in S9, the arrangement state of the arrangement information 32 is changed to an attribute "completion of movement out of field of view". Further, the separation / mixing information of the arrangement information 32 of the corresponding arrangement is updated with the separation / mixing determination result of step S12, and the attribute of the arrangement for which “separation area with human” is determined in step S12 is displayed. Change to “Completed within view”.

  When the arrangement area and the ghost in the change area are detected by the above processing, the mask image generation unit 44 generates the processing target area by excluding the arrangement area and the ghost from the change area. Subsequently, a single color image having the same shape as the processing target area and the background image 30 of the processing target area are transparently synthesized to generate a composite image, and the processing target area in the current image is replaced with the composite image to replace the mask image. Generate (S14). In addition, when the arrangement object area | region is not detected, the process which excludes an arrangement object area | region is abbreviate | omitted. If no ghost is detected, the process of excluding the ghost is omitted.

  When the generation of the mask image is completed for the current image from which the change area is extracted, the background image generation unit 43 excludes the ghost whose separation is determined and the arrangement area of the arrangement where the movement completion is detected from the change area. The update target area is calculated, and the part of the update target area of the background image 30 is replaced with the current image in the update target area (S15). If there is no arrangement for which movement completion is detected, the process of excluding the arrangement area is omitted. If there is no ghost determined to be separated, the process of excluding the ghost is omitted.

  Further, the background image generation unit 43 updates the background image 30 other than the change area by performing a weighted average with the current image.

  The control unit 4 outputs the mask image to the output unit 5 when the mask image is generated by the mask image generation unit 44, and outputs the current image to the output unit 5 when the mask image is not generated. The output unit 5 displays the image input from the control unit 4 on the monitor (S16).

  When the above processes S2 to S16 are completed for the current image, the control unit 4 replaces the immediately preceding image 31 with the current image and also replaces the area information 33 with the information of the changed area extracted in step S3. Return to step S2.

[Modification]
As mentioned above, although embodiment of this invention was described, the said embodiment can be changed and it can also be set as other embodiment of this invention. Examples thereof will be described below.

<Setting attention point>
The attention points for detecting the sparse background flow may be set at regular intervals on the reference image. For example, the attention points can be set in a lattice shape. In this case, the set interval is set to be sufficiently shorter than the assumed width / height of the arranged object so that a plurality of attention points are surely set in the area of the arranged object. This setting method is excellent in that the amount of processing is small, although erroneous detection of the most similar points increases compared to the case of using feature points. In this case, the input flow detection is also performed by setting a common attention point.

<Background flow selection>
As described above, the background flow detection unit 41 may detect the background flow and the input flow separately. In this case, the background flow selection unit 411 calculates the degree of coincidence for the combination of each background flow and each input flow, and selects a background flow whose degree of coincidence is equal to or less than the coincidence determination threshold value. The degree of coincidence can be the distance between the start ends and the distance between the end points. If both of these distances can be regarded as minute changes due to illumination fluctuations or noise (for example, one pixel), it is determined that they match.

  When the background flow is detected separately from the input flow, the reference image can be the background image 30 and the comparison image can be the current image. That is, a background flow is detected by setting a point of interest on the background image 30 and searching for the most similar point on the current image. In this case, since the person is not included in the background image 30, it is possible to set the attention point on the arrangement object without being affected by the occlusion by the person. Therefore, it is effective in a monitoring space where many people come and go.

  In addition, the background flow detection unit 41 changes to a configuration in which the background flow is selected based on the match with the input flow in the above embodiment, and a group of motion vectors detected by associating feature points between the background image 30 and the current image. There are a predetermined number or more of those whose directions match each other (that is, a group consisting of the motion vectors in the same direction as each other and the number of vectors belonging to the group is equal to or more than a predetermined number). It can also be set as the structure detected as a background flow corresponding to an arrangement | positioning thing. This also eliminates erroneous detection such as the background flows 235 and 236 of FIG. Further, the input flow detection unit 410 can be omitted. Also in this method, the reference image may be the current image and the comparison image may be the background image 30, or the reference image may be the background image 30 and the comparison image may be the current image.

<Feature point>
As feature points, corners or subpixel corners of the contour of the image can be extracted instead of the blob.

<Tracking features>
The arrangement change detection unit 42 records the arrangement object area (movement destination area) of the current image or the ghost part (movement source area) of the background image in the arrangement object information 32 as the arrangement object template, and the arrangement object tracking unit 422 It is possible to detect the movement destination area of the arrangement object in the input image by matching processing using the template, and to track the arrangement object.

  Furthermore, the arrangement information 32 that has been moved can be stored in the template without being unregistered and used for tracking at the time of detecting the next arrangement change.

<Tracking target>
It is possible to use the detection result of arrangement change for person tracking. For example, a change area other than the arrangement object area can be tracked as a person area. In addition, until a change area determined to contain an arrangement is detected until a separation between the person and the arrangement is detected, the change area is characterized by a feature quantity (a feature quantity in which the person and the arrangement are mixed), During this time, there is a method in which the feature amount of the person alone is saved and switching to tracking by the feature amount of the person alone is detected if separation is detected.

  DESCRIPTION OF SYMBOLS 1 Image monitoring apparatus, 2 Image pick-up part, 3 Storage part, 4 Control part, 5 Output part, 30 Background image, 31 Immediately preceding image, 32 Arrangement information, 33 Area information, 40 Change area extraction part, 41 Background flow detection part, 42 arrangement change detection unit, 43 background image generation unit, 44 mask image generation unit, 410 input flow detection unit, 411 background flow selection unit, 420 arrangement object region detection unit, 421 ghost detection unit, 422 arrangement object tracking unit.

Claims (5)

An image monitoring device that detects an arrangement change of an object placed in the monitoring space using an input image that is sequentially captured and input in the monitoring space,
A storage unit that stores a background image of the monitoring space in which a background including the arrangement is captured;
A background corresponding point on the background image side and an input corresponding point on the input image side that correspond to each other as having similar image characteristics between the background image and the input image are detected, and the input corresponding point on the input image side is detected from the background corresponding point. A background flow detection unit for obtaining a background flow from a vector to an input corresponding point;
An arrangement change detector that detects the occurrence of the arrangement change by indicating the movement amount of the background flow greater than or equal to a predetermined value;
Equipped with a,
The background flow detection unit is configured so that the preceding input image side corresponding to the input image and the preceding input image captured at the preceding time close to the capturing time of the input image as having similar image characteristics to each other. The preceding input corresponding point is detected, and the vector using the background corresponding point located at substantially the same position as the preceding input corresponding point is used as the background flow . The image monitoring apparatus according to claim 1 ,
The background flow detection unit determines the vector using the background corresponding point where the image feature of the preceding input image and the image feature of the background image substantially match at the position of the preceding input corresponding point as the background flow. An image monitoring apparatus characterized by that. The image monitoring apparatus according to claim 1 , wherein:
The background flow detection unit detects, as the background flow, a vector having a predetermined number or more of the vectors whose directions substantially coincide with each other. The image monitoring apparatus according to any one of claims 1 to 3 ,
The background flow detection unit detects the background flow by setting the background corresponding point candidate or the input corresponding point candidate at a low density within a predetermined image range each time the input image is input. And setting the background corresponding point candidate or the input corresponding point candidate at a higher density in the image range than the sparse background flow detecting means when occurrence of the arrangement change is detected. A dense background flow detection means to be obtained,
The arrangement change detection unit detects the occurrence of the arrangement change using the background flow detected by the sparse background flow detection means, and collects a set of input corresponding points constituting the background flow detected by the dense background flow detection means. Detecting as a movement destination area of the arrangement,
An image monitoring apparatus characterized by the above. An image monitoring device that detects an arrangement change of an object placed in the monitoring space using an input image that is sequentially captured and input in the monitoring space,
A storage unit that stores a background image of the monitoring space in which a background including the arrangement is captured;
A background corresponding point on the background image side and an input corresponding point on the input image side that correspond to each other as having similar image characteristics between the background image and the input image are detected, and the input corresponding point on the input image side is detected from the background corresponding point. A background flow detection unit for obtaining a background flow from a vector to an input corresponding point;
An arrangement change detector that detects the occurrence of the arrangement change by indicating the movement amount of the background flow greater than or equal to a predetermined value;
With
The background flow detection unit detects the background flow by setting the background corresponding point candidate or the input corresponding point candidate at a low density within a predetermined image range each time the input image is input. And setting the background corresponding point candidate or the input corresponding point candidate at a higher density in the image range than the sparse background flow detecting means when occurrence of the arrangement change is detected. A dense background flow detection means to be obtained,
The arrangement change detection unit detects the occurrence of the arrangement change using the background flow detected by the sparse background flow detection means, and collects a set of input corresponding points constituting the background flow detected by the dense background flow detection means. Detecting as a movement destination area of the arrangement,
An image monitoring apparatus characterized by the above.

Images