JP2002191044A - Face image monitoring system - Google Patents

Abstract

(57) [Problem] To provide a face image monitoring system that realizes more efficient face image monitoring processing by extracting only images necessary for monitoring from a captured image. SOLUTION: A moving object detecting function S5 for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, determining that the moving object is a human image, An image in which the face area is evaluated by a predetermined method to output an evaluation value, and based on the evaluation value, a predetermined number of the image information is selected and output from the plurality of pieces of image information including the moving object. A face image monitoring system having information selection functions S6 and S7. This makes it possible to selectively output only images that are effective for monitoring.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, and more particularly to a face image monitoring system using image information from a camera or the like.

[0002]

2. Description of the Related Art Recently, for example, in a store such as a convenience store, a monitoring system based on a monitoring video from an ITV camera or the like installed in the store has become very popular. This monitoring system has various forms, for example, the following three types are representative. One is a monitoring system that transmits a monitoring image captured by an ITV camera installed in a store to a monitoring center so that a monitoring person can monitor.

There is also a surveillance system that similarly records surveillance video shot by an ITV camera on a time-lapse VTR at regular intervals. In this system, the observer does not need to constantly monitor, but can refer to the recorded image later as needed. Further, there is also a monitoring system for detecting a person based on a change in an image from a monitoring video imaged by an ITV camera and recording the image on a VTR or saving the image on a disk. In this system, since storing only image when a target image change should be monitored,
More efficient monitoring image collection can be performed with a small storage capacity.

[0004]

However, each of the above monitoring systems has the following problems. That is, in the first monitoring system, there is a problem that a monitoring person must monitor the monitoring image at all times, and the labor cost is extremely burdened as the cost of the system. In the second monitoring system, since recording is performed only at regular intervals, many meaningless images without a person are recorded. Therefore, there are problems such as a shortage of storage capacity and difficulty in searching for necessary images. Further, in the third monitoring system, only an image including a person can be obtained, but there is a problem that the image is not necessarily an image suitable for checking a monitoring image later. An object of the present invention is to provide a face image monitoring system that realizes more efficient monitoring processing by extracting and storing only images necessary for monitoring from images captured from an ITV camera or the like.

[0005]

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information; When the moving object is detected, if a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image, and the person image determining unit determines that the human image is a human image. The face area of the moving object is evaluated by a predetermined method, and a face area evaluation unit that outputs an evaluation value, based on the evaluation value output by the face area evaluation unit, the moving object is included. A face image monitoring system comprising: image information selecting means for selecting and outputting a predetermined number of the image information from a plurality of pieces of image information. According to the present invention, instead of monitoring all moving objects as in the related art, the present invention detects a face region from the moving object, evaluates the face region by various methods, and performs monitoring based on the evaluation result. It is selected and output as a target. As a result, it is possible to monitor a human face important for a person determination very efficiently.

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. A face direction evaluation unit that detects a direction of a person's face in the face area and outputs an evaluation value based on the face direction, and the moving object is included based on an evaluation value output by the face direction evaluation unit. Image information selecting means for selecting and outputting a predetermined number of pieces of the image information from the plurality of pieces of image information. According to the present invention, instead of selectively outputting all face images,
By detecting the direction of the face and selecting and outputting a face image in front that is effective by, for example, a person determination, a face image that is very effective for face image monitoring can be automatically detected.

Further, according to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. A face area evaluation unit that detects an area of a face with respect to the face area and outputs an evaluation value based on the face area; and the plurality of areas including the moving object based on the evaluation value output by the face area evaluation unit. Image information selecting means for selecting and outputting a predetermined number of the image information items from the image information items. The present invention realizes more efficient face monitoring processing by selecting and outputting only an image effective for face image monitoring, for example, having a relatively large face area, instead of extracting all face images by the above configuration. can do.

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. A face image contrast evaluation unit that detects a contrast of the face image for the face region and outputs an evaluation value based on the face image contrast; and Image information selecting means for selecting and outputting a predetermined number of the image information items from the plurality of image information items included therein. It is. According to the present invention, not all face images are selectively output by the above-described configuration, but more efficient face monitoring processing is realized by preferentially selecting and outputting a high-contrast face image effective for face monitoring. be able to.

Further, according to the present invention, a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and the moving object detecting means detecting the moving object When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. A face image skin color clarity evaluation unit that detects the skin image clarity of the face image for the face area and outputs an evaluation value based on the face image skin clarity, and an evaluation output by the face image skin color clarity evaluation unit. Image information selecting means for selecting and outputting a predetermined number of pieces of the image information from the plurality of pieces of image information including the moving object based on the value. .

Further, according to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. A face image expression evaluation unit that detects an expression of a face image for the face region and outputs an evaluation value based on the face image expression; and the moving object, based on the evaluation value output by the face image expression evaluation unit, A face image monitoring system, comprising: image information selecting means for selecting and outputting a predetermined number of the image information from the plurality of pieces of image information included.

Further, according to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and when the person image determining unit determines that the person image is a person image, Based on the number of moving objects determined as in the image, based on the number of people evaluation means that outputs an evaluation value of the image information including the moving object, based on the evaluation value output by the number of people evaluation means, A face information monitoring system, comprising: image information selecting means for selecting and outputting a predetermined number of the image information from the plurality of pieces of image information including the moving object. According to the present invention, by detecting the number of moving objects by the above-described configuration, for example, an image of one person can be preferentially selected and output, or an image of a plurality of persons can be preferentially selected and output. Becomes possible.

Further, according to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image judging unit judges the moving object to be a person image, and the moving object judged by the person image judging unit to be a person image. Based on the ratio of the whole body of the human image being shown, a systemic evaluation unit that outputs an evaluation value, and based on the evaluation value output by the systemic evaluation unit, the plurality of objects including the moving object A face image monitoring system comprising: image information selecting means for selecting and outputting a predetermined number of the image information from the image information. According to the present invention, it is possible to preferentially output an image including not only the face of a person but also the whole body of the person by considering the ratio between the face image and the whole body image.

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face region is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines is a person image, Image information selecting means for selecting and outputting a predetermined number of pieces of the image information from the plurality of pieces of image information including the moving object so as to exist at as many positions as possible in the monitoring area in the selected image. This is a face image monitoring system characterized by the following. According to the present invention, with the above-described configuration, the diversity of the position of a person is evaluated so that a person image comes to various places of a selected image, and an image is selectively output in accordance with the evaluation value. By selectively outputting an image having such a variety of positions, it is possible to collect additional information which is convenient for grasping the action range of the person to be monitored.

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. The direction of the face of a person is detected for the face area, and a predetermined number of pieces of image information are extracted from the plurality of pieces of image information including the moving object so that a predetermined number of pieces of image information to be selected have face directions as diverse as possible. And an image information selecting means for selecting and outputting a face image. According to the present invention, with the above-described configuration, a variety of face directions is provided as an evaluation value so that the face direction of a selected image is not monotonous, and thereby face images in various directions are selectively output. This makes it possible to collect more detailed information on the human face of the person to be monitored.

According to the present invention, there is provided a moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, and wherein the moving object detecting means detects the moving object. When a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a person image, and a moving object that the person image determining unit determines that the moving object is a person image. The face area is evaluated by a predetermined method, and a face area evaluation unit that outputs an evaluation value, based on the evaluation value output by the face area evaluation unit, based on the plurality of pieces of image information including the moving object, Image information selecting means for selecting and outputting a predetermined number of the image information, and image information storing means for storing the predetermined number of image information selected by the image information selecting means in a storage area, Is that face image monitoring system. The present invention provides a monitoring system that automatically stores image information selectively detected based on a face area in a storage area by the above-described configuration, and can facilitate subsequent search processing and the like.

[0016]

Embodiments of the present invention will be described below in detail with reference to the drawings. (First Embodiment) First, a first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of the first to fourth embodiments.

In FIG. 1, the entire face image monitoring system includes an image input unit 1, an A / D conversion unit 2, an image buffer (image memory) 3, and an image processing unit 4 (processor). The image captured by the image input unit 1 is A / D
The signal is converted into a digital signal by the converter 2 and stored in the image memory 3. The processor 4 reads the image data and performs digital image processing. The image input unit 1 may be, for example, a normal CCD camera.

Hereinafter, the processing contents of the processor 4 will be described in detail. FIG. 2 is a flowchart illustrating a processing flow of the processing processor unit according to the first embodiment.
FIG. 2 shows a flow of processing performed by the processor 4. The processing procedure includes a process of cutting out a sequence of N consecutive images in which the same person exists (S5), a process of determining the number P of selected images (S6), and a process of selecting P images from the sequence of N images (S7). ).

FIG. 3 is an explanatory view showing a situation of scene extraction. The above-described clipping processing (S
5) comparing the image read from the image memory 3 with the reference background image, detecting the intruding person candidate area from the difference between them, and using the continuous image sequence in which the person is detected as a scene. It is to extract.

In the detection of a person region, for example, a difference process is performed on continuous images, a smoothing process is performed to suppress the influence of noise, and then binarization is performed with a certain threshold value.
It is only necessary to perform integration and labeling as a spatially connected region in the sense of four or eight connections, and select a region having a high likelihood as a person from the size and shape of the region. At this time, the threshold value for binarization may be a value given in advance, or may be determined by a method [1] in which the variance ratio of the frequency distribution is maximized.

As another method, a method [2] based on the posterior probability may be used. Assuming that the event θ0 is a background and the event θ1 is a person, θ before observing the luminance of a pixel in an image
The prior probabilities of 0 and θ1 are defined as w0 and w1 (where w
0 + w1 = 1), and if the conditional probability of the image luminance I under the event θk is P (I | θk), the posterior probability w1 ′ after I is observed is w1 ′ by Bayes' theorem. =
w1 * P (I | θ1) / ｛w0 * P (I | θ0) + w1
* P (I | θ0)}. A pixel having w1 'larger than a certain threshold value may be a pixel belonging to a person, and a certain pixel in this area may be detected as a person.

Next, an example of a procedure for cutting out a continuous image sequence in which the same person is detected as one scene will be described. The first image (first frame) is determined when, for example, a person candidate area is first detected at one end of the monitoring screen (when a person enters). The last image (final frame) is determined when, for example, the person candidate area is not detected after the person candidate area is detected at one end of the monitoring screen (when the person leaves). As a result, the number of cut out images is set to N.

The method of cutting out a continuous image including the same person is not limited to the above. For example, the image may be cut out for each zone determined in advance according to the detection position in the screen. In addition, by performing a tracking process while associating a person between images based on the similarity of the position, size, moving speed, color information, and the like, if the movement is not regarded as a person, the person candidate area is determined as a candidate. It is also possible to increase the reliability of the person determination by performing processing such as removal.

Further, the number of persons is not limited to one. Even when a plurality of persons are included, it is possible to cut out a continuous image in which each person is present by tracking them (in this case, the image is Are duplicated). Alternatively, a continuous image including a plurality of persons may be cut out as a unit.

Finally, a method of determining the number P of selected images will be described. Here, the number of selected images P is set to P = [the maximum integer not exceeding α * N] based on the selectivity α. The selection rate α may be given by the user in advance, or may be dynamically changed according to the degree of congestion of the image transmission unit 5. Then, it is assumed that P sheets are selected by evenly thinning out of the N sheets.

According to the procedure described as the first embodiment, all the images (N
Instead of selecting and outputting P), the representative P is selected by selecting this by a certain method. By doing so, it is possible to greatly save the storage capacity even when storing images, and it is possible to perform efficient processing even when collation search or the like is performed later.

(Second Embodiment) Next, a second embodiment of the present invention will be described with reference to the drawings. The overall configuration of the embodiment of the present invention is the same as that of the first embodiment shown in FIG.
The following describes in detail the processing content of the processing processor 4.

FIG. 4 is a flowchart showing the flow of the processing of the processor unit of the second embodiment, and shows the processing performed by the processor 4. In FIG. 4, the processing procedure is a processing of buffering N continuous image sequences (S
8), a process of performing face recognition on each image of the continuous image sequence to calculate a selection priority (S9), and a process of selecting an image based on the selection priority (S10).

In the process S8, N images may be simply buffered at regular time intervals, or motion detection may be performed as in the first embodiment, and N continuous images in which a person exists may be cut out. . The value of the parameter P may be given in advance as a fixed value (when P is N or more, no selection is made), or P = [α based on the selectivity α as in the first embodiment. * Maximum integer not exceeding N]. Next, a method of performing face recognition on each image of a given continuous image sequence and calculating a selection priority will be specifically described with reference to FIG. FIG. 5 is an explanatory diagram illustrating a specific example of the processing in the image selection unit.

2-1. Face direction evaluation value of a person For example, an image in which a person's face appears in a frontal direction is considered to be an image more suitable for person verification. Therefore, a high evaluation value is given to an image close to the face direction (for example, the front face) required by the user, and the image is preferentially selected.

The face direction of a person can be obtained by, for example, canonical discriminant analysis [3]. This can be realized by preparing learning face images classified into classes such as a frontal face, a left-facing face, and a right-facing face, and performing identification by the following canonical discriminant analysis. (A) The pixel value of the face area of A × B dots is considered as an A × B dimensional vector, and the intra-class variance Sb and the inter-class variance Sw are calculated. (B) inv (Sw) solving the Sb v = λv become eigenvalue problem (inv (Sw) is the inverse matrix of Sw). (C) Find a projection matrix having V = eigenvector vi as a column vector. (D) The average vector of each class is projected onto V, and the average projected vector of each class is calculated. (E) The input image area is projected onto V at the time of recognition, and the result is compared with the average projected vector in (d). By performing the process on the person candidate area detected in the image selection processing, face detection and face direction identification can be performed.

FIG. 6 is an explanatory diagram showing the concept of the face direction identification processing, and complements the above description. In addition to this method, a standard pattern such as an eye, a nose, and a mouth may be prepared, the pattern may be matched with a person area, and the orientation of the face may be recognized from the positional relationship of the pattern.

With respect to the face direction thus obtained,
Users were given evaluation values {frontal face = 100, left oblique face = 70, right slanting face = 70,
By applying the right face = 40, the left face = 40, the back side = 10,..., The face direction evaluation value E1 of the person can be determined.

2-2. Face area evaluation value of a person An image in which a person's face is large is considered to be an image more suitable for person verification. Therefore, a high evaluation value is given to an image having a large face area of a person, and the image is preferentially selected. The area S of the face area can be obtained, for example, by calculating the area of the connected flesh color area. Therefore, this value may be used as the face area evaluation value of the person. E2 = S 2-3. Human face contrast evaluation value An image with high face contrast is considered to be an image more suitable for human verification. Therefore, a high evaluation value is given to the image with high contrast, and the image is selected with priority. The face contrast can be calculated based on the variance μ of the luminance of the face area. The larger the variance, the greater the difference in luminance and the higher the contrast, so this may be used as the evaluation value. Therefore, E3 = μ.

2-4. Evaluation value of skin color clarity of a person An image with a high skin color clarity is considered to be an image more suitable for person verification. So a score value to the sharpness is high image preferentially selected. Skin color vividness is RGB
Images are converted to HSV (Hue, Saturation, Val)
ue) based on the formula [3] to be converted in the space can be evaluated by looking at the average saturation (Saturation) of the face area.

For example, the saturation S is as follows: S = (I−i) / I where I = max (R, G, B), i = min (R,
G, B) or S = {(BR) ² + (RG) ² + (GB) ² } /
3 can be obtained. Therefore, the evaluation value of skin color sharpness can be set to E4 = S.

2-5. Face expression evaluation value of the person Face expression is neutral (no blinking)
Such an image is considered to be an image more suitable for person verification. Therefore, an evaluation value is calculated by giving a priority to this expression. For facial expression evaluation values, for example, a learning facial image classified into classes such as a normal facial expression, a blinking face, and a mouth-opening face is prepared, and identification by the canonical discrimination analysis similar to the face direction identification described above is performed. Evaluation value given by the user to each class 各 Normal expression = 100, blinking face = 70, mouth open face =
E5 may be determined based on 60,. Alternatively,
A standard pattern such as an open eye, a closed eye, an open mouth, and a closed mouth may be prepared, the pattern may be matched with a face region, and the facial expression may be identified by a combination of the patterns.

The embodiment is not limited to the above. When a plurality of persons exist in a scene, for example, an evaluation value for each person may be added. Also,
In the above, separate and independent evaluation values are used, but these may be combined into one evaluation value. For example, weighting coefficients w1, w2,..., Wx may be defined in advance for a plurality of evaluation values E1, E2,.

E6 = w1 * E1 + w2 * E2 +... + Wx * Ex Also, a priority order is defined for the evaluation scale itself.
However, in the case of a tie, a process of assigning priority in E3 may be performed.

Next, a process of selecting an image based on the selection priority will be described. Here, P images are selected from the N image sequences in the order of the selection priority (P is an integer equal to or less than N and is appropriately given by the system). The selected image can be used for network image transmission or for the purpose of storing it in an appropriate image storage device such as a magnetic disk or a VTR. Since only the selected appropriate image needs to be transmitted and stored, it is transmitted. Requires less space and storage.

As for the processing mode, here, from N sheets to P
Although it is described that an image is to be selected, the sequential processing may be such that the priority Ei is calculated each time one image is input, and the selected image is selected when the priority is equal to or more than a certain threshold.
As described above, by selecting and outputting the moving image by the method described in the second embodiment, it is possible to realize more efficient face image monitoring processing according to various conditions of the face image.

(Third Embodiment) Next, a third embodiment of the present invention will be described with reference to the drawings. The overall configuration of the embodiment of the present invention is the same as that of the first embodiment shown in FIG.
Hereinafter, the processing contents of the processor 4 will be described in detail. FIG. 7 is a flowchart illustrating a processing flow of the processing processor unit according to the third embodiment, and illustrates a processing flow performed by the processing processor 4. The processing procedure is a process of buffering N continuous image sequences (S11).
And a process of performing the whole body recognition of each person of the continuous image sequence to calculate the selection priority (S12), and a process of selecting an image based on the selection priority (S13).
Consists of The process (S11) may be the same as in the second embodiment.

Next, a method (process S12) of calculating the selection priority by performing overall recognition of a person for each image of a given continuous image sequence will be specifically described. FIG.
FIG. 9 is an explanatory diagram showing the counting, moving, and whole-body recognition processing of a person, and illustrates the third embodiment.

3-1. Evaluation value of the number of persons It is conceivable that an image having a large number of persons can be selected as an appropriate image on the assumption that the state of the monitoring area is better captured. Conversely, the number of persons is small scene, and that there is a high possibility that the abuse in the monitoring area is carried out,
It is conceivable to select these as appropriate images.
Therefore, the evaluation value of the priority can be obtained based on the number of persons in the image. First, a person area is extracted in the same manner as in the first embodiment, and the number Nh of persons included in the screen is obtained from the number of these areas. Therefore, the evaluation value of the number of people can be set to E7 = Nh (when a large number of people has priority) or E7 = 1 / Nh (when a small number of people has priority).

3-2. Evaluation value of the whole body of the person The image that captures the whole image of the person is useful for later person verification because the height of the person, the color and pattern of the clothes, etc. can be understood,
A high evaluation value is given to such an image, and the image is preferentially selected. First, the area of the person region for each image is calculated by the same method as in the first embodiment. Next, the area of the face region is calculated by the same method as in the second embodiment. That is, assuming that the area of the person area is S0 and the area of the face area is S1, E8 = S0 / S1 is an evaluation value of how much the entire image is captured. For example, when the face area S1 is smaller than the person area S0, it means that the whole image up to the body part is well captured, and at that time, S0 / S1 has a large value.

The embodiment is not limited to the above. When a plurality of persons exist in a scene, for example, an evaluation value for each person may be added. Also,
In the above, separate and independent evaluation values are used, but these may be combined into one evaluation value. For example, weighting factors w7 and w8 may be defined in advance for a plurality of evaluation values E7 and E8, and weighted addition may be performed.

E9 = w7 * E7 + w8 * E8 It is also possible to define a priority order in the evaluation scale itself, and perform a hierarchical process of assigning a priority order in E7 and assigning a priority order in E8 in the case of a tie. Good.

Next, the process of selecting an image based on the selection priority (S13) will be described. Here, as in the second embodiment, P images are selected from the N image sequences in the order of the selection priority (P is an integer equal to or less than N and is appropriately given by the system). ). The selected image can be used for network image transmission or for the purpose of storing it in an appropriate image storage device such as a magnetic disk or a VTR. Since only the selected appropriate image needs to be transmitted and stored, it is transmitted. Requires less space and storage.

As for the processing mode, here, from N sheets to P
Although it is described that an image is to be selected, the sequential processing may be such that the priority Ei is calculated each time one image is input, and the selected image is selected when the priority is equal to or more than a certain threshold.
Also in the third embodiment, by setting conditions such as the number of persons and the generality of persons, more efficient person monitoring processing can be realized.

(Fourth Embodiment) The overall configuration of the embodiment of the present invention is the same as that of the first embodiment shown in FIG. Hereinafter, the processing contents of the processor 4 will be described in detail. FIG. 9 is a flowchart showing a processing flow of the processing processor unit according to the fourth embodiment.
1 shows the flow of the processing performed in step (a). Processing procedure is a processing for buffering the N sheets of the continuous image sequence (S14), the image I _i with N sheets of continuous image sequence given, I
A function E (f _i ) = f (I _i−k , I _{i−k + 1} ) for obtaining the selection priority based on the information of _i itself and the past k images
.., I _i ) (I _{i- k} represents the “ik-th image I”), a process of calculating the selection priority based on this function (S15), and (S16). Processing (S14)
Can be the same as in the second embodiment. Next, the process (S15) of calculating the selection priority based on the function will be described. FIG. 8 is an explanatory diagram showing the counting, moving, and systemic recognition processing of a person.

4-1. Evaluation value of moving speed of a person In a scene with a low moving speed, a person is stably stationary and is considered to be an appropriate image that can be easily verified by a person. Therefore, a high evaluation value is given to a scene with a lower moving speed, and the scene is selected with priority.

With respect to the information on the person areas included in the images I _i-1 and I _i , the correspondence between the person areas is obtained based on the similarity of the position, size, color, etc. (person tracking). The movement vector (movement speed) V is obtained from the change in the position. Therefore, the evaluation value of the moving speed of the person can be E10 = 1 / (c + V) (c is a constant for avoiding division by zero when V = 0).

4-2. Non-stationarity evaluation value of a person A scene in which a person performs an unusual operation is likely to have some special event, and is important in verification. Therefore, a high evaluation value is given to these non-stationarities, and a priority is selected.

The standard pattern of the change sequence of the characteristic amount (for example, movement vector, position, size, shape) of the person region for the stationary operation and the non-stationary operation is learned in advance. The input image sequence I _i-k , I _{i-k + 1} ,.
Similarly, for _i , a change sequence of the feature amount of the person area is obtained, and these are subjected to DP matching, matching using an HMM model, and the like to perform matching with the standard pattern.

As a steady operation at this time, for example, in the case of a face image monitoring system in a convenience store, intrusion into a cashier counter where a general customer cannot enter is considered. By setting a cashier counter area in an image, a case where a moving object enters this image area is determined to be a special event.

Here, with respect to the input image sequence, the matching score Score (Am) is highest in Am in the feature amount change pattern group in the steady operation, and Bn is the highest matching score Score in the feature amount change pattern group in the non-stationary operation.
If (Bn) is high, the evaluation value indicating the non-stationarity is EE = (similarity with non-stationary) − (similarity with stationary) = Sc
ore (Bn) -Score (Am). When using the evaluation values EE1, EE2,..., EEx for a plurality of feature amount sequences, weighting addition may be performed using weighting coefficients ww1, ww2,.

E11 = ww1 * EE1 + ww2 * EE2
+ ... + wwx * EEx Although the above are separate and independent evaluation values, they may be combined into one evaluation value. For example, weighting coefficients w10 and w1 are assigned to a plurality of evaluation values E10 and E11.
1 may be defined in advance and weighted and added.

E12 = w10 * E10 + w11 * E11 It is also possible to define a priority order in the evaluation scale itself, and perform a hierarchical process of assigning a priority order at E10 and assigning a priority order at E11 when the evaluation scales are the same. Good. Next, the process of selecting an image based on the selection priority (S11) will be described. Here, as in the second embodiment, P images are selected from the N image sequences in the order of the selection priority (P is an integer equal to or less than N and is appropriately given by the system). ). The selected image can be used for network image transmission or for the purpose of storing it in an appropriate image storage device such as a magnetic disk or a VTR. Since only the selected appropriate image needs to be transmitted and stored, it is transmitted. Requires less space and storage.

As for the processing mode, here, N to P
Although it is described that an image is to be selected, the sequential processing may be such that the priority Ei is calculated each time one image is input, and the selected image is selected when the priority is equal to or more than a certain threshold.
In the fourth embodiment, and the moving speed of the person, with reference to detection of intrusion into the predetermined area by selecting an image, it is possible to realize more efficient image monitoring processing.

(Fifth Embodiment) The overall configuration of the fifth embodiment of the present invention is the same as that of the first embodiment shown in FIG. Hereinafter, the processing contents of the processor 4 will be described in detail. FIG. 10 is a flowchart showing the flow of processing of the processing processor unit of the fifth embodiment, and shows the flow of processing performed by the processing processor 4. The processing procedure is a processing of buffering N continuous image sequences (S17).
And an evaluation function g for “combination” nCp ways of selecting P from all N images (I ₀ to I _N ) included in the scene
() Is defined, a process of calculating the selection priority based on this function (S18), and a process of selecting an image based on the selection priority (S19). Processing (S1
7) may be the same as in the second embodiment. Next, the processing 14 for calculating the selection priority based on the function will be described with reference to FIG. FIG. 11 is an explanatory diagram illustrating a specific example of processing in the image selection unit based on a combination of P sheets.

5-1. Diversity of Person Positions If an image is selected that samples the trajectory of the movement of a person from intrusion to departure at the same interval as possible, it is considered that the entire action of the person is summarized more appropriately. Therefore, P such images are selected. (In a simple equidistant sampling based on the number of images standing pat state only has in some cases been selected) of a person in the image I _i coordinates (X (I _i), Y
(I _i )) is obtained as a person candidate area in the selecting section 8 for N image sequences. The movement vector from _{I i} to _{I j} is calculated as the difference between the coordinates in the coordinate and _{I j} in _{I i.} Therefore, a set (nCp) of {I ′ _i ,..., I ′ _P } images obtained by selecting P images from N images
Against street) obtains the variance _{V d isp} average _{V ave} and the movement vector of the moving _{vector, V ave} up, V
_What is necessary is just to select a set of P sheets that minimizes _disp . In other words, a large V _ave indicates that the movement between the images is appropriately selected to be large, and a small V _disp indicates that sampling at a more equal interval can be performed. Is shown. Figure 12 is a diagram illustrating an example of a selection based on diversity person position.

5-2. Diversity of human faces If you select an image in which a person faces more various directions,
It captures more of the characteristics of the person and is considered useful for verification. Therefore, P such images are selected. First, the face direction and its likelihood are obtained by face direction identification based on the canonical discriminant analysis described above. At this time, the number of types of the face direction classes included in the scene is set to C. In order to obtain images in more various face directions, a set having a large variance in the face direction may be selected from combinations of selecting N to P images.
For example, class numbers of 1, 2,..., C are assigned to the face direction classes, and a selection is made so that the variance of the class number values of the P images is maximized. (If all classes are evenly selected, the variance of class number values will increase.)
The height of the face likelihood of each image may be weighted. Through the above-described processing, it is possible to select a face image that faces more various directions.

In the first to fifth embodiments described above,
The processing may be performed in real time, or images may be stored once and processed in a batch type later. No. 5
Also in the embodiment, an efficient monitoring process can be realized by selectively outputting an image in consideration of the variety of person positions and the variety of person faces.

(Sixth Embodiment) Finally, FIG. 13 shows the entire configuration of an embodiment of the present invention. In FIG. 13, the face image monitoring system includes an image input unit 20 and an A / D conversion unit 2.
1, an image buffer (image memory) 22, an image processing unit 23 (processing processor), and an image storage unit 24.

The image picked up by the image input unit 20 is converted into a digital signal by the A / D converter 21.
It is stored in the image memory 22. The processor 23 reads the image data and performs digital image processing. The image input unit 20 may be, for example, a normal CCD camera, and the image storage unit 24 may be a normal magnetic disk.

Hereinafter, the processing contents of the processor 23 will be described in detail. FIG. 14 shows the processor 2
3 shows the flow of the processing performed in Step 3. The processing procedure is a process of cutting out a series of N continuous images in which the same person exists (S2
5) and a process of performing person and face image recognition on each image and selecting P images from the N image sequence (S26);
A process of registering P images in the image storage unit (S27);
The process (S28) of searching and displaying an image from the image storage unit.

The process (S25) for cutting out a series of N continuous images in which the same person exists is the same as in the first embodiment. Also, a process of performing person and face image recognition on each image and selecting P images from the N image sequence (S26)
Are the same as in the second to fifth embodiments. As for the process of registering the P images in the image storage unit, database registration is performed together with auxiliary information on the image such as the camera ID and the time at which the image was taken. It is assumed that the definition of the data structure of the database is made in advance in, for example, a relational table format. Here, a method for searching and displaying an image will be specifically described. FIG. 15 is an explanatory diagram showing the concept of image search and collation according to the present invention.

By displaying the selected image in the thumbnail display (representative image display), the user can easily verify the monitoring image. Further, it is possible to automatically collate a face photograph (for example, a wanted criminal) presented by the user with these images, and to display a face image or an image series having a high degree of similarity. Also, the user may specify an image sequence instead of an image.

The person matching is performed by searching for a pattern similar to the designated person by principal component analysis or canonical discriminant analysis based on the image / image sequence of the designated person and a group of face images of other persons. [4] [5]. For the search, the similarity of the color information may be used, or the similarity of the feature amount series as described in the fourth embodiment may be used. As a result, for example, it is possible to search for a certain person or extract only a certain behavior (for example, entry into a cash register) from the monitoring image sequence.

According to the sixth embodiment, the image storage unit is provided in the face image monitoring system, so that the images selectively output according to the predetermined conditions can be automatically and sequentially stored. Furthermore, for these stored images, representative images according to certain conditions are displayed as thumbnails,
This enables a more intuitive grasp of the monitoring status.

References are listed below. [1] Otsu: "Automatic threshold selection method based on discrimination and least square criterion", IEICE, Vol.J63-D, No.4, pp.349-35
6, 1980. [2] Nakai: "Moving object detection method using posterior probability",
[3] Takagi, Shimoda: "Image Analysis Handbook", The University of Tokyo Press, 1991. [4] W Zhao, R. Chellappa et al. “Discriminant Analy
sis of Principal Components for Face Recognitio
n ”, International Conference on Automatic Face &
Gesture Recognition, Nara, Japan, April 1998. [5] B. Moghaddam, A. Pentland et al .: “Beyond Eigenfac
es: Probabilistic Matching for Face Recognitio
n ”, International Conference on Automatic Face &
Gesture Recognition, Nara, Japan, April 1998.

[0072]

As described in detail above, according to the present invention,
By recognizing the movement and face direction of the person in the monitoring image,
Only appropriate images can be selected and transmitted / stored as monitoring images, so that transmission capacity / storage capacity can be reduced and efficiency can be improved. Furthermore, by selecting only an appropriate image and presenting a series of monitoring images to the user, and performing a search using the example image series, it is possible to easily search and verify the monitoring image later.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of first to fourth embodiments.

FIG. 2 is an exemplary flowchart illustrating the flow of a process performed by a processor unit according to the first embodiment;

FIG. 3 is an explanatory diagram showing a state of scene extraction.

FIG. 4 is a flowchart illustrating a flow of processing of a processing processor unit according to the second embodiment;

FIG. 5 is an explanatory diagram illustrating a specific example of a process performed by an image selection unit;

FIG. 6 is an explanatory diagram illustrating the concept of face direction identification processing.

FIG. 7 is a flowchart illustrating a processing flow of a processing processor unit according to the third embodiment;

FIG. 8 is an explanatory diagram showing the counting, moving, and whole-body recognition processing of a person;

FIG. 9 is a flowchart illustrating a processing flow of a processing processor unit according to the fourth embodiment;

FIG. 10 is a flowchart illustrating the flow of a process performed by a processor unit according to a fifth embodiment;

FIG. 11 is an explanatory diagram illustrating a specific example of processing in an image selection unit based on a combination of P sheets.

FIG. 12 is a diagram illustrating an example of selection based on the diversity of person positions.

FIG. 13 is a block diagram showing an overall configuration of a sixth embodiment.

FIG. 14 is a flowchart illustrating a processing flow of a processing processor unit according to the sixth embodiment;

FIG. 15 is an explanatory diagram showing the concept of image search and collation according to the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Image input part 2 ... A / D conversion part 3 ... Image memory 4 ... Processing processor 24 ... Image storage part

Continued on the front page F term (reference) 5B057 AA19 BA02 CA12 CA16 DC01 DC25 DC32 5C054 AA01 CC05 CG06 CH01 EA01 EA05 EE00 EF06 FB03 FC01 FC04 FC08 FC13 FE09 FF02 GB02 HA18 5L096 AA02 BA02 CA02 FA01 FA15 HA03 JA11

Claims (11)

[Claims] 1. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, when the moving object detecting means detects the moving object,
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image A face area evaluation unit that evaluates a predetermined method and outputs an evaluation value, based on the evaluation value output by the face area evaluation unit, a predetermined number of the plurality of pieces of image information including the moving object, A face information monitoring system comprising: an image information selection unit that selects and outputs the image information. 2. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object,
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image A face direction evaluation unit that detects a direction of a person's face with respect to the face direction, and outputs an evaluation value based on the face direction; based on the evaluation value output by the face direction evaluation unit, the plurality of moving objects including the moving object A face information monitoring system comprising: image information selecting means for selecting and outputting a predetermined number of the image information from the image information. 3. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image detecting the area of the face for a face area evaluation means for outputting an evaluation value based on the face area, based on the evaluation value output by the face area evaluation unit, wherein the plurality of image information including the moving object And an image information selecting means for selecting and outputting a predetermined number of the image information items from the image data. 4. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image A face image contrast evaluation unit that detects a contrast of the face image and outputs an evaluation value based on the face image contrast, and the moving object is included based on the evaluation value output by the face image contrast evaluation unit. A face image monitoring system, comprising: image information selecting means for selecting and outputting a predetermined number of the image information from a plurality of pieces of image information. 5. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object,
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image. A face image skin color sharpness evaluation unit that detects the skin color sharpness of the face image and outputs an evaluation value based on the face image skin color sharpness; and A face information monitoring system, comprising: image information selecting means for selecting and outputting a predetermined number of pieces of image information from the plurality of pieces of image information including the moving object. 6. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image. Detecting the facial expression of the face image,
A face image expression evaluation unit that outputs an evaluation value based on the face image expression; and a predetermined number of the plurality of pieces of image information including the moving object, based on the evaluation value output by the face image expression evaluation unit. A face image monitoring system comprising: image information selecting means for selecting and outputting image information. 7. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining means for determining that the moving object is a person image; and Based on the number of moving objects in the image, based on the number of people evaluation means that outputs an evaluation value of the image information including the moving object, based on the evaluation value output by the number of people evaluation means,
Image information selecting means for selecting and outputting a predetermined number of the image information from the plurality of image information including the moving object,
A face image monitoring system comprising: 8. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
A person image determining unit that determines that the moving object is a person image if a face area is detected from the moving object image information; and a person image in the moving object that the person image determining unit determines is a person image. Based on the ratio of the whole body being shown, a systemic evaluation unit that outputs an evaluation value, based on the evaluation value output by the systemic evaluation unit, from the plurality of pieces of image information including the moving object. A face image monitoring system comprising: image information selection means for selecting and outputting a predetermined number of the image information. 9. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image, and a moving object that the human image determining unit determines is a human image is selected. Image information selecting means for selecting and outputting a predetermined number of pieces of image information from the plurality of pieces of image information including the moving object so as to exist at as many positions as possible in a monitoring area in the image. Face image monitoring system. 10. A moving object detecting means for continuously receiving a plurality of pieces of image information to be monitored from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object.
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image The direction of the face of the person is detected, and a predetermined number of the image information is selected from the plurality of pieces of image information including the moving object so that the selected predetermined number of pieces of image information have as many face directions as possible. A face image monitoring system, comprising: image information selecting means for outputting. 11. A moving object detecting means for continuously receiving a plurality of pieces of image information of a monitoring target from the outside and detecting a moving object in the image information, wherein the moving object detecting means detects the moving object,
If a face area is detected from the moving object image information, a person image determining unit that determines that the moving object is a human image; and the face region of the moving object that the human image determining unit determines is a human image. A face area evaluation unit that evaluates a predetermined method and outputs an evaluation value, based on the evaluation value output by the face area evaluation unit, a predetermined number of the plurality of pieces of image information including the moving object, Face image monitoring, comprising: image information selecting means for selecting and outputting the image information; and image information storing means for storing the predetermined number of image information selected by the image information selecting means in a storage area. system.