JP7379213B2 - Image matching device - Google Patents

Description

Embodiments of the present invention relate to an image matching device, an image matching system, and an image matching method.

2. Description of the Related Art An image matching device is known that identifies a person included in an input image by comparing a face image (its characteristic information) included in an input image with a pre-registered face image (its characteristic information). For example, the input image is an image from a surveillance camera that photographs pedestrians and the like. In order to improve matching accuracy, the image matching device narrows down a plurality of frames included in an input image to frames that satisfy a criterion and performs matching processing.

JP2019-57815A

In cases where a frame that meets the criteria is not obtained or a frame that meets the criteria is insufficient, it is possible to postpone the matching process. However, in an environment where a sufficient number of frames that meet the criteria cannot be obtained, the matching process is often postponed. Furthermore, in such a case, it may be possible to select any one frame and execute the matching process, but there is a concern that the matching accuracy will be significantly reduced.

The problem to be solved by the present invention is to provide an image matching device, an image matching system, and an image matching device that can prevent a decrease in matching accuracy when a frame meeting a standard is not obtained or when there is a shortage of frames meeting a standard. An object of the present invention is to provide an image matching method.

An image matching device in which software is installed on a computer according to an embodiment includes an acquisition device, a detection device, a tracking device, a frame selection device, a matching device, and an output device. The acquisition means continuously acquires matching target images from the imaging device. The detection means detects one or more matching targets from a plurality of input frames included in the matching target image. The tracking means tracks a predetermined matching target based on the detection result of the matching target, or extracts a set of frames including the matching target. The frame selection means dynamically selects one or more matching frames from the plurality of predetermined frames, depending on the number of matching frames that meet the criteria among the plurality of predetermined frames including the predetermined matching target. The matching means matches the registered image registered in the database and the predetermined matching target image included in the matching frame on a one-to-one or one-on-multiple basis. The output means outputs the verification result.

FIG. 1 is a configuration diagram showing an example of an image matching system according to the first embodiment. FIG. 2 is a diagram illustrating an example of frame selection according to the first embodiment. FIG. 3 is a flowchart illustrating an example of matching processing by the image matching system according to the first embodiment. FIG. 4 is a configuration diagram showing an example of an image matching system according to the second embodiment. FIG. 5 is a diagram showing an example of a parameter setting screen according to the second embodiment. FIG. 6 is a configuration diagram showing an example of an image matching system according to the third embodiment. FIG. 7 is a configuration diagram showing an example of an image matching system according to the fourth embodiment.

A first embodiment will be described below with reference to the drawings.
FIG. 1 is a configuration diagram showing an example of an image matching system according to the first embodiment.
In this embodiment, one or more matching frames are dynamically selected from a plurality of predetermined frames according to the number of matching frames that meet the matching criteria among a plurality of (N) predetermined frames that include a predetermined matching target. select. For example, when the number of compatible frames is equal to or greater than a specified number (n, n≦N), a matching frame is selected from a plurality of predetermined frames using the first selection method (selection method S1), and the number of compatible frames is determined by the specified number. If the number is less than the number, a frame for verification is selected from a plurality of predetermined frames by a second selection method (selection method S2). For example, using the selection method S1, a first predetermined number (m1, m1≦n) of compatible frames that is equal to or less than a specified number are selected as matching frames. By selection method S2, a second predetermined number (m2, m2≦N) of frames are selected as frames for verification.

In this embodiment, as an example, a system will be described in which the detection target is a person's face and a registered person (=principal) is matched using a face image. Note that the detection target may be part or all of a moving object such as a person or an animal, or part or all of an artificial moving object such as a vehicle, an aircraft, or a drone.

As shown in FIG. 1, the image matching system according to the first embodiment includes a registration device 100, an input device 200, and an image matching device 300. The registration device 100 can be configured by one or more computers, and includes a registration section 111 and a registered image database 112. The input device 200 includes an image input section (imaging device) 201. The image matching device 300 can be configured by one or more computers, and includes a processor 301, a main memory 302, an auxiliary storage device 303, and interfaces 304 and 305. In other words, the image matching device 300 is a device in which software is installed in a computer.

The processor 301 performs processing such as computation and control necessary for image matching processing. The processor 301 executes control to realize various functions based on programs such as system software, application software, or firmware stored in at least one of the main memory 302 and the auxiliary storage device 303. The processor 301 is, for example, a CPU (central processing unit), an MPU (micro processing unit), or a DSP (digital signal processor). Alternatively, processor 301 is a combination of more than one of these.

Main memory 302 is a computer-readable storage medium, and includes non-volatile memory (non-temporary storage medium) used exclusively for reading data and volatile memory used for reading and writing data. The nonvolatile memory plays a role as a program memory and stores part or all of programs, data, various setting values, etc. used by the processor 301 to perform various processes. The volatile memory plays a role as a work memory and stores data temporarily used by the processor 301 to perform various processes.

The auxiliary storage device 303 is a computer-readable storage medium, and is a non-temporary storage medium such as EEPROM (electric erasable programmable read-only memory) (registered trademark), HDD (hard disk drive), or SSD (solid state drive). be. The auxiliary storage device 303 stores part or all of programs, data, various setting values, etc. used by the processor 301 to perform various processes.

The interface 304 functions as an acquisition unit (acquisition means), continuously acquires input images from the image input unit 201, and outputs them to the processor 301. The interface 305 functions as an output unit and outputs the verification results and the like to an external device such as a display.

Furthermore, the processor 301 will be explained. The processor 301 includes a detection section (detection means) 3011, a tracking section (tracking means) 3012, a frame selection section (frame selection means) 3013, a collation section (verification means) 3014, and the like. The detection unit 3011 uses the input image as a matching target image, and detects one or more matching targets from a plurality of input frames included in the matching target image. The tracking unit 3012 tracks a predetermined matching target based on the detection result of the matching target, or extracts a set of frames including the matching target. The frame selection unit 3013 selects one or more matching frames from the tracking frame set according to the number of matching frames that meet the matching criteria among the plurality (N) of predetermined frames (i.e., the tracking frame set) including the predetermined matching target. Dynamically select frames for use. For example, the frame selection unit 3013 selects a matching frame that meets the face matching criteria from the tracked frame set. The frame selection unit 3013 selects a matching frame from the tracked frame set using selection method S1 when the number of matching frames is equal to or greater than a specified number (n), and selects a matching frame from the set of tracking frames when the number of matching frames is less than the specified number. A matching frame is selected from the set using selection method S2. The matching unit 3014 matches the registered image (or its characteristic information) registered in the database with the predetermined matching target image (or its characteristic information) included in the matching frame on a one-to-one or one-to-multiple basis.

A more detailed explanation is provided below.
The image input unit 201 can be installed at one location or multiple locations, inputs a human image of a person to be photographed, and is composed of, for example, an ITV camera. An ITV camera digitizes optical information obtained through a camera lens using an A/D converter and outputs it as image data. Note that the search target may end up being digital human image data. Therefore, the image input unit 201 may be an interface for importing an image file or a video file shot with a digital still camera via a medium. Note that a device for inputting ID (Identification) information used for identity verification may be provided together with the image input unit 201. For example, any device may be used as long as it can read information recorded on various media, such as an IC card reader, a QR code (registered trademark) reader, or a magnetic information reading device. By reading additional information about passersby from these devices, it is possible to use it for verification processing.

The detection unit 3011 detects the face image of the object from the input image from the image input unit 101. In face detection, for example, an average image of rectangular areas of many faces is created as a face template, and the face template is scanned on the input image from the image input unit 201 while being translated, rotated, and enlarged/reduced. This can be achieved by setting the face detection area to the position where the difference in brightness value is the smallest.

The tracking unit 3012 tracks the detected face image of the object between frames. Since it is assumed that the detected face images of the same person appear continuously over a plurality of frames, tracking processing of the face images of the person is performed so that they can be associated as the same person. As a means of achieving this, there are a method that uses optical flow to correlate the position of a face image detected from one frame with the next frame, and a method that uses the technology disclosed in Japanese Patent Application Laid-Open No. 2011-170711. . The facial feature extraction means described later includes a method of selecting at least one image suitable for a search from among multiple frames of facial area images associated with the same person, and a maximum number of detected frames. There is a method to use any number of images.

The frame selection unit 3013 selects n frames (1≦n≦N) to be used by the matching unit 3014 from a set of tracking frames (N frames) that are associated with the same person in the detection unit 102 and the tracking unit 3012. Select and output the frame.

The frame selection unit 3013 first selects n frames based on an index for determining whether or not the N frames meet the matching criteria. An example of the index is shown below.
・Size of object (resolution)
・Score of the certainty of the object ・Brightness distribution within the image area of the object ・Position of the object within the image area ・Orientation of the object In this embodiment, the size (resolution) of the object is determined by the detection unit 3011. Since the area where the target face is detected is determined as a rectangular area, the number of pixels in the horizontal direction may be used as an index. Furthermore, the number of pixels in the vertical direction may be used as an index, or the area obtained by multiplying the number of pixels in the vertical direction by the number of pixels in the horizontal direction may be used as an index.

As the score of the certainty of the target object, it is possible to use the score indicating the match with the template when detected by the detection unit 3011 as is.

It is generally said that it is desirable for the brightness distribution within the image area of the object to have a large contrast in the face image and no bias, so the variance and standard deviation of the brightness values within the face area are used. . It is also possible to use as an index the number or percentage of pixels that are too bright or too dark, such as the number of pixels with a luminance value above a predetermined value (overexposure) or the luminance value below a predetermined value (overexposure). The lower the percentage of brightness values above or below a predetermined value, the higher the index becomes. Therefore, convert the ratio into a reciprocal number, or convert it into a value obtained by subtracting the ratio from 1.

Regarding the position of the target object within the image area, as it approaches the top, bottom, left, and right edges of the input image, the image quality may deteriorate due to lens distortion, and part of the face image or the image around the face may be outside the area and information may be lost. Therefore, it is possible to use the distance from the center coordinates as an index so that the closer you get to the center of the image, the higher the score (in this case, the closer the distance from the center coordinates, the smaller the value, the better the index. It is possible to adopt the reciprocal number so that

The orientation of the object can be detected by using the technique disclosed in Japanese Unexamined Patent Publication No. 2007-4767, and the closer it is to the front, the higher the index. In addition to this method of calculating using a calculation formula, it is also possible to use Japanese Patent Application Laid-Open No. 2019-57815 to learn the correlation between multiple images of the subject and the differences between them and other people to find an appropriate image for matching. It is.

The frame selection unit 3013 evaluates whether each frame conforms to the matching criteria based on at least one index of each frame. When using a plurality of indicators, the frame selection unit 3013 uses a weighted average value obtained by multiplying each indicator by a predetermined coefficient and adding them as an indicator to evaluate whether each frame conforms to the matching criteria. It's okay. Further, the frame selection unit 3013 focuses on each index of each frame, extracts frames with high respective indexes at a predetermined ratio, and focuses on multiple indexes of the extracted frames to meet the matching criteria. Narrow down compatible frames. Alternatively, the frame selection unit 3013 sets a threshold value for each index of each frame, and narrows down frames that are equal to or higher than the threshold value as suitable frames. Further, the judgment criteria may be switched for each index, and after narrowing down the criteria to frames whose first index is equal to or higher than a predetermined threshold, the frame selection unit 3013 further selects frames in descending order of the second index. The frames may be narrowed down and suitable frames may be selected.

The frame selection unit 3013 selects n frames from among the frames that meet the matching criteria. For example, the frame selection unit 3013 selects n frames in descending order of index. Alternatively, the frame selection unit 3013 may select n frames depending on whether each index is equal to or greater than a preset reference value. Note that the number of selected frames may be less than n.

FIG. 2 is a diagram illustrating an example of frame selection according to the first embodiment.
For example, the tracking unit 3012 tracks a predetermined matching target and identifies a plurality of (N) predetermined frames including the predetermined matching target. As shown in FIG. 2, it is assumed that six frames F1 to F6 including a predetermined matching target have been identified. Furthermore, it is assumed that three indicators are used: "face certainty score (face score)", "face size", and "face direction".

The frame selection unit 3013 selects a frame based on the threshold value of each index. Frame F1, where the face size is small and the face size index is less than the threshold value, does not meet the matching criteria, and the face orientation is sideways, so the frame F1 whose face size index is less than the threshold value does not meet the matching criteria. Frames F3 and F4 also do not meet the matching criteria, and three frames F2, F5, and F6, in which all indicators are above the threshold, meet the matching criteria.

If the number of compatible frames is equal to or greater than the specified number, for example, if the number of compatible frames is 3 and the specified number is 2, the frame selection unit 3013 selects one of the three frames F2, F5, and F6 that meet the matching criteria. Two frames are selected from the following in descending order of face score (selection method S1-1). Alternatively, the frame selection unit 3013 may multiply each of the three indicators by a coefficient and select two frames from the three frames that meet the criteria based on the total value of the indicators (selection method S1-2). ).

On the other hand, if the number of compatible frames is less than the specified number, for example, if the number of compatible frames is 3 and the specified number is 4, the frame selection unit 3013 selects a plurality of (N) predetermined frames including the face to be matched. is directly selected as a frame for verification (selection method S2-1). Thereby, when there is a shortage of frames suitable for matching, it is possible to increase the amount of information and improve matching accuracy.

Alternatively, the frame selection unit 3013 preferentially selects frames excluded by low priority indicators based on the priorities specified for the above three indicators (selection method S2-2), and reduces the number of frames. make up for the shortage. The missing number of frames may be compensated for, or a number of frames greater than the insufficient number of frames may be compensated for. If priority is given to matching time, compensate for the insufficient number of frames. When prioritizing matching accuracy, supplement the number of frames that are greater than the number of frames that are insufficient. For example, in addition to selecting compatible frames, the frame selection unit 3013 preferentially selects frames that are excluded based on a low priority index from among frames that do not meet the matching criteria. This makes it possible to add information according to the weight of the index, and it is possible to more effectively improve matching accuracy.

Alternatively, the frame selection unit 3013 selects extraction frames extracted at regular intervals from a plurality of (N) predetermined frames including faces to be matched as frames for matching (selection method S2-3). If priority is given to matching time, a fixed interval is set as the first interval, and if priority is given to matching accuracy, the fixed interval is set to the second interval (first interval < second interval). Furthermore, in addition to selecting compatible frames, the frame selection unit 3013 may select frames extracted at regular intervals. In the case of FIG. 2, in addition to selecting the compatible frames F2, F5, and F6, some of the frames F1, F3, and F5 extracted at regular intervals may be selected. For example, if priority is given to matching time, four frames (=prescribed number n) are selected, which is the matching frames F2, F5, and F6 and the extracted frame F1 or F3. If priority is given to matching accuracy, five frames (>specified number n) are selected, including the compatible frames F2, F5, and F6 and the extracted frames F1 and F3. This makes it possible to add information and improve matching accuracy more effectively.

Alternatively, the frame selection unit 3013 may select frames extracted based on time from a plurality of (N) predetermined frames including the face to be matched (selection method S2-4). Furthermore, in addition to selecting suitable frames, the frame selection unit 3013 may select frames to be extracted based on time. This makes it possible to add information and improve matching accuracy more effectively.

For example, a new higher predetermined number (=prescribed number n) of frames are extracted based on the time, and the extracted frames are selected (selection method S2-41). In the case of FIG. 2, it is assumed that frame F1 has the oldest time, and frames F2, F3, F4, F5, and F6 have newer times in this order. In this case, the frame selection unit 3013 selects frames F3, F4, F5, and F6. Furthermore, in addition to selecting the compatible frames F2, F5, and F6, frames that are insufficient to the specified number may be supplemented from new frames F3, F4, F5, and F6 based on time. For example, the frame selection unit 3013 selects compatible frames F2, F5, F6 and the newest frame F4 based on time.

Alternatively, a predetermined number (=predetermined number n) of older frames are extracted based on the time, and the extracted frames are selected (selection method S2-42). In the case of FIG. 2, frames F1, F2, F3 and F4 are selected. Furthermore, in addition to selecting the compatible frames F2, F5, and F6, the oldest frame F1 among the oldest frames F1, F2, F3, and F4 based on time may be selected.

Alternatively, if the number of compatible frames is less than a specified number, the frame selection unit 3013 groups the plurality of predetermined frames into frame groups based on the amount of change between frames, and selects a representative face image from each frame group. A frame is extracted and the extracted frame is selected as a frame for comparison (selection method S2-5). For example, the frame selection unit 3013 clusters facial images into groups that are similar, and selects a representative image from each group. For example, the degree of facial similarity (processing equivalent to the matching process of the matching unit 3014) is determined for all combinations of face images, and k-mean clustering is performed to group them. The index used for clustering may be a correlation value between face images, or the index used for frame selection may be used. After performing clustering, the frame selection unit 3013 selects a representative image of each group. As a method for selecting representative images, the most average image of each group may be selected, or images may be simply selected in chronological order of oldest or newest.

The registration unit 111 is a scanner or a camera. The scanner scans a facial image for registration (for example, a still image such as a photograph), extracts facial feature information from the acquired registered image, and registers the registered image and its feature information in the registered image database 112. Alternatively, the camera captures a facial image (video) for registration, extracts facial feature information from the acquired registered image, and registers the registered image and its feature information in the registered image database 112. Extraction of facial feature information is similar to the method used by the matching unit 3014 to extract facial feature information from an image. Note that the registration unit 111 may be an interface for importing a face image file via a medium.

The registered image database 112 is a database used when the matching unit 3014 calculates a similarity index, and stores facial feature information for each individual to be searched. In addition, the registered image database 112 also includes ID information, facial image, name, gender, age, height, etc. of the relevant person, in association with facial feature information, in order to make the matching results easier to understand for users etc. Store attribute information regarding. The facial feature information may be the data itself extracted in the facial feature extraction process, an m×n feature vector, a subspace, or a correlation matrix immediately before performing KL expansion.

The matching unit 3014 matches the feature information of the predetermined matching target image included in the matching frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the registered image database 112. When one or more IC cards or the like are used in the image input unit 201, the matching unit 3014 selects a predetermined image included in the matching frame selected from the input image obtained by photographing the image input unit 201. The image to be compared is compared with the registered image of the person associated with the information obtained from the IC card. If an IC card or the like is not used, the matching unit 3014 registers the predetermined matching target image included in the matching frame selected from the input image obtained by the image capturing by the image input unit 201 in the registered image database 112. All registered images are compared.

A matching unit 3014 generates a matching result including a degree of similarity indicating the authenticity of the person. For example, the matching unit 3014 generates matching results sorted in descending order of similarity. The matching unit 3014 may generate a matching result in association with ID information and attribute information such as a face image stored in the registered image database 112. The technique disclosed in Japanese Patent Laid-Open No. 2007-4767 may be used as the process for determining the facial similarity. This can be achieved by creating an image in which the direction and state of the face are intentionally varied using a model for one piece of person image information, and then searching and comparing facial feature information. In addition, there are documents that calculate multiple frames of the same person using moving images (Kazuhiro Fukui, Osamu Yamaguchi, Kenichi Maeda: "Face recognition system using moving images" IEICE research report PRMU, vol. 97, No. 113, pp. 17-24 (1997)) may be used. This can be achieved by using feature information such as a subspace outputted by such a method as individual feature information for the face detected in the input image. In this case, the similarity index is the degree of similarity between subspaces managed as facial feature information. Further, as a similarity index other than similarity, distance on a feature space or Mahalanobis distance can also be used. When distance is used, the larger the numerical value, the lower the similarity; therefore, as a comparison method with a threshold value, the smaller the value than the threshold value, the higher the similarity with the registered person.

Note that although this embodiment uses a matching method that uses facial features, this embodiment can be applied to any matching that uses features obtained from images other than faces; By changing the camera to a near-infrared camera, it is also possible to perform matching using information from the iris and retina of the eye, as well as using veins on the hands and face. In this embodiment, a matching technique based on a full-body image using characteristics, possessions, etc.) or a matching technique based on behavior using a gait or a predetermined motion may be applied. These methods can be realized based on the following literature information.

<Literature information>
・2007, Patent Application Technology Trend Survey Report, Biometric Verification Input/Recognition (Summary Version), April 2008, Japan Patent Office, Internet
<URL:https://www.jpo.go.jp/resources/report/gidou-houkoku/tokkyo/document/index/vaio_youyaku.pdf>
・Patent No. 3307936, Iris: JP 11-213164, Retina: JP 2006-099493, Vein: JP 2006-011988, Vein: JP 2018-82408, Whole body: Strategy Creative Research Promotion Project, CREST, Research Area: ``Construction of human-harmonious information technology for a symbiotic society'', Research Topic: ``Analysis of human behavior based on the gait intention behavior model and construction of an information environment that captures the mind'' , Research completion report, Research period: October 2010 - March 2017, Research representative: Yasushi Yagi, (Osaka University, Director/Vice President), Internet
<URL:https://www.jst.go.jp/kisoken/crest/evaluation/s-houkoku/sh_h28/JST_1111071_10101452_2016_PER.pdf>

The interface 305 outputs the verification results and the like to an external device. The external devices include a display device such as a liquid crystal display, a speaker that outputs notifications or alarms, and a traffic control device that controls traffic by opening and closing a gate. For example, the interface 305 outputs display information including the matching results to a display device. The display device displays the success or failure of matching, the image input by the image input unit 201, and the matching target (face area) detected by the detection unit 3011, based on the display information. In addition, the display device (monitoring display device) uses the display information to determine how similar the input image (face image) is to which person among the people registered in the registered image database 112. , an input image, an image of a person similar to the input image, and a degree of similarity to the input image may be displayed side by side.

Further, the interface 305 outputs an audio signal indicating a warning sound to the speaker according to the degree of similarity. The interface 305 outputs an audio signal indicating a warning sound to the speaker when a matching result with a degree of similarity equal to or higher than the threshold value is not obtained. The speaker outputs a warning sound based on the audio signal.

Further, the interface 305 outputs an unlocking signal for unlocking the gate to the traffic control device according to the degree of similarity. The interface 305 outputs an unlock signal when a matching result with a degree of similarity equal to or higher than a threshold value is obtained, and does not output an unlock signal when a matching result with a degree of similarity equal to or higher than a threshold value is not obtained. . When the traffic control device receives the unlock signal, it unlocks the gate based on the unlock signal.

As described above, even in an environment where images unsuitable for matching are input, it is possible to suppress a decrease in matching accuracy, and it is possible to improve the reliability of control such as entry/exit based on the matching results.

FIG. 3 is a flowchart illustrating an example of matching processing by the image matching system according to the first embodiment.
The image input unit 201 photographs the monitoring area for entrance or exit, and outputs the input image obtained by the photographing. The interface 304 of the image matching device 300 acquires the input image as a matching target image (ST101). The processor 301 of the image matching device 300 executes matching processing on a predetermined matching target image included in the acquired matching target images (ST102 to ST109).

The processor 301 implements various functions based on programs stored in at least one of the main memory 302 and the auxiliary storage device 303, and executes collation processing. For example, the detection unit 3011 detects one or more matching targets from a plurality of input frames included in the matching target image (ST102). The tracking unit 3012 tracks a predetermined matching target based on the detection result of the matching target (ST103). When tracking of the predetermined matching target by the tracking unit 3012 is completed (ST104, YES), the frame selection unit 3013 calculates the index of a plurality of (N) predetermined frames including the predetermined matching target (ST105). The frame selection unit 3013 dynamically selects one or more matching frames from a plurality of predetermined frames, depending on the number of frames whose index is equal to or greater than the reference value, that is, the number of frames that meet the matching criteria.

If the number of frames that meet the matching criteria is equal to or greater than the threshold (=prescribed number) (ST106, YES), the frame selection unit 3013 selects the matching frame based on the first frame selection process (frame selection method S1). (ST107), and the matching unit 3014 matches the feature information of the registered image registered in the registered image database 112 with the feature information of the predetermined matching target image included in the matching frame (ST109). The interface 305 outputs the matching result from the matching section 3014 to an external device (ST110).

If the number of frames that meet the matching criteria is less than the threshold (=prescribed number) (ST106, NO), the frame selection unit 3013 selects the matching frame based on the second frame selection process (frame selection method S2). (ST108), and the matching unit 3014 matches the feature information of the registered image registered in the registered image database 112 with the feature information of the predetermined matching target image included in the matching frame (ST109). The interface 305 outputs the matching result from the matching section 3014 to an external device (ST110).

A second embodiment will be described below with reference to FIG. 4.
FIG. 4 is a configuration diagram showing an example of an image matching system according to the second embodiment. The main difference between the image matching system according to the first embodiment and the image matching system according to the second embodiment is that the image matching system according to the second embodiment includes an interface 306. In the image matching system according to the first embodiment and the image matching system according to the second embodiment, common components are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

The interface 306 accepts parameter settings for processing by the frame selection unit 3013 and the matching unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the matching unit 3014. For example, the interface 306 is connected to a display device and an input device, outputs a parameter setting screen to the display device, and receives parameters input through the input device on the parameter setting screen. This allows parameters related to matching processing time and matching accuracy to be adjusted from the outside. For example, administrator and maintenance personnel parameters can be adjusted via interface 306.

FIG. 5 is a diagram showing an example of a parameter setting screen according to the second embodiment.
As shown in FIG. 5, the parameter setting screen includes a setting section for setting face matching parameters (denoted as "authentication threshold") for each gate. In this setting section, values from 0 to 1 can be set. The value may be adjusted by using a slider bar or by directly inputting the value.

The parameter setting screen also includes a setting section for setting parameters related to frame selection. This setting section accepts settings for the number of predetermined frames (N pieces) including the predetermined matching target. For example, as shown in FIG. 5, five (5 frames) are set. Note that the number of predetermined frames is not limited, and the ratio of the predetermined frames among all frames may be set.

The setting unit also accepts parameters for each of the selection method S1 (frame selection mode 1) and the selection method S2 (frame selection mode 2). If the number of compatible frames is greater than or equal to the specified number, frame selection mode 1 is executed, and if the number of compatible frames is less than the specified number, frame selection mode 2 is executed. Parameters regarding frame selection mode 1 and frame selection mode 2 are set. Frame selection methods for frame selection mode 1 and frame selection mode 2 are set. Frame selection methods can be selected from "best order (highest index)", "thinning", "newest in time", "oldest in time", "clustering", and "automatic learning selection". . Furthermore, the number of frames output in frame selection mode 1 and the number of frames output in frame selection mode 2 can also be set.

For example, if the total input frames are 10 frames and the matching frames are 7 frames, frame selection mode 1 is selected because it is larger than the set threshold of 5, and the frame selection mode 1 is selected among the 7 frames with the highest index suitable for matching. Three frames are output in sequence. On the other hand, if there are only two compatible frames, mode 2 is selected, and 10 frames are evenly thinned out and three frames are selectively output.

On the setting screen, you may also increase the number of setting parameters necessary for the system. It is clear that it is also possible to enable settings to be made from a mobile terminal via a network, or to have the settings reflected in the system by writing them in a settings file.

A third embodiment will be described below with reference to FIG. 6.
FIG. 6 is a configuration diagram showing an example of an image matching system according to the third embodiment. In the image matching system according to the first embodiment and the image matching system according to the second embodiment, common components are given the same reference numerals, and descriptions thereof will be omitted as appropriate. In the first and second embodiments, it is assumed that registered images are registered in advance in the passerby registration image database 1121, but in the third embodiment, images are taken while passing through an entrance/exit gate, etc. Registered images are registered from the captured images taken at multiple locations, and the images taken at multiple locations are used to determine whether they are the same person or not. For example, by registering facial images upon entering a predetermined area and associating facial images upon entrance with facial images upon exiting, it becomes possible to manage which people have entered and participated. . Furthermore, it is also possible to check whether a person other than the person who entered has participated.

Therefore, in the third embodiment, a plurality of passage gates are assumed, for example, passage gate G1 (entrance) and passage gate G2 (exit). As shown in FIG. 6, the image matching system according to the third embodiment includes a registration device 100, an input device 210 provided corresponding to the passage gate G1, an input device 220 provided corresponding to the passage gate G2, and An image matching device 300 is provided.

The registration device 100 can be configured by one or more computers, and includes a processor 101, a main memory 102, an auxiliary storage device 103, an interface 104, an interface 105, and a passerby registration image database 1121. The processor 101 includes a detection unit 1011, a tracking unit 1012, a frame selection unit 1013, a registration unit 1014, and the like. The input device 210 includes an image input section (imaging device) 211. The input device 220 includes an image input section (imaging device) 221. The image matching device 300 can be configured by one or more computers, and includes a processor 301, a main memory 302, an auxiliary storage device 303, and interfaces 304, 305, and 306.

The registration device 100 registers the registration target image included in the captured image from the input device 210. In this embodiment, it is assumed that the passage gate G1 and the passage gate G2 are not gates that allow passage in both directions, but are dedicated gates for entering and exiting the area. Note that by installing two image input units 211 and 221 in opposite directions, the present invention can also be applied to a gate that allows passage in both directions.

The image input unit 211 of the input device 210 photographs the passage area of the passage gate G1 for visitors and outputs the photographed image. The interface 104 of the registration device 100 acquires a photographed image, and the detection unit 1011 detects a registration target from the photographed image. The tracking unit 1012 tracks the detected registration targets by associating them between frames. The frame selection unit 1013 selects a suitable frame suitable for face matching from a plurality of predetermined frames (i.e., a set of tracked frames) including the registration target. The registration unit 1014 registers the selected frame in the passerby registration image database 1121. The interface 105 outputs display information indicating the registration result etc. as information at the time of passing to the display device, and also outputs an unlock signal etc. to the traffic control device according to the registration result.

The image input unit 221 of the input device 220 photographs the passage area of the passage gate G2 targeted at the contestant, and outputs the photographed image. The interface 304 of the image matching device 200 acquires a captured image, and the detection unit 3011 detects a predetermined matching target from the captured image. The tracking unit 3012 tracks the detected predetermined matching target by associating it between frames. The frame selection unit 3013 selects a suitable frame suitable for face matching from a plurality of predetermined frames (i.e., a set of tracked frames) including a predetermined matching target. The frame selection unit 3013 dynamically selects one or more matching frames from the tracked frame set according to the number of matching frames among the plurality of predetermined frames including the predetermined matching target. The matching unit 3014 matches the feature information of the predetermined matching target image included in the matching frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the passerby registered image database 1121. The interface 305 outputs the verification results and the like to an external device. Further, the interface 306 accepts settings of parameters for processing by the frame selection unit 3013 and the matching unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the matching unit 3014.

Note that the detection unit 1011 performs the same processing as the detection unit 3011, and the tracking unit 1012 performs the same processing as the tracking unit 3012. The frame selection unit 1013 may select all of a plurality of predetermined frames (i.e., a set of tracked frames) including the registration target, or may select all of the compatible frames suitable for face matching. Alternatively, the frame selection unit 1013 may thin out a certain number of frames from a plurality of predetermined frames including the registration target in order to shorten processing time or reduce the amount of information.

Similar to the matching unit 3014, the registration unit 1014 extracts facial feature information from the face image and outputs the result to the passerby registration database 112.

According to the third embodiment, it becomes possible to manage people who go in and out of a specific area in association with each other.

A fourth embodiment will be described below with reference to FIG. 7.
FIG. 7 is a configuration diagram showing an example of an image matching system according to the fourth embodiment. The main difference between the image matching system according to the third embodiment and the image matching system according to the fourth embodiment is that the image matching system according to the fourth embodiment has an ID registration unit (registration means) 113, It includes an ID information database 114, an ID confirmation section 1015, and an ID confirmation section 3015. In the image matching system according to the third embodiment and the image matching system according to the fourth embodiment, common components are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

In the fourth embodiment, a face image at the time of entrance is associated with a face image at the time of appearance to manage which person has entered and participated, and ID information is managed using the ID information database 114. It becomes possible to manage the entry and exit of persons by associating ID information with the persons entering or exiting.

For example, the image input unit 211 or 221 reads ID information from an IC card or the like in parallel with photographing a face image, the passerby registration image database 1121 stores facial feature information and attribute information, and the ID information database 114 stores ID information. The feature information and attribute information stored in the passerby registration image database 1121 and the ID information stored in the ID information database 114 are associated with each other using management identification information or the like.

Alternatively, the ID registration unit 113 reads the ID information and face image information (feature information) from the registration medium, and registers the ID information and face image information in association with each other. That is, the passerby registration image database 1121 stores facial image information (feature information), and the ID information database 114 stores ID information. The face image information (feature information) stored in the passerby registration image database 1121 and the ID information stored in the ID information database 114 are associated by management identification information or the like.

The image input unit 211 of the input device 210 photographs the passage area of the passage gate G1 for visitors and outputs the photographed image. The interface 104 of the registration device 100 acquires a photographed image, and the detection unit 1011 detects a registration target from the photographed image. The tracking unit 1012 tracks the detected registration targets by associating them between frames. The frame selection unit 1013 selects a suitable frame suitable for face matching from a plurality of predetermined frames (i.e., a set of tracked frames) including the registration target. The registration unit 1014 registers the selected frame in the passerby registration image database 1121. The interface 105 outputs display information indicating the registration result etc. as information at the time of passing to the display device, and also outputs an unlock signal etc. to the traffic control device according to the registration result.

Further, when it is necessary to read ID information at the time of entrance, for example, the image input unit 211 reads the ID information from an ID card or the like, and the ID information database 114 stores the read ID information. Further, the ID confirmation unit 1015 confirms whether the read ID information is registered ID information stored in the ID information database 114, and the interface 105 outputs the confirmation result of the ID information.

The image input unit 221 of the input device 220 photographs the passage area of the passage gate G2 targeted at the contestant, and outputs the photographed image. The interface 304 of the image matching device 200 acquires a captured image, and the detection unit 3011 detects a predetermined matching target from the captured image. The tracking unit 3012 tracks the detected predetermined matching target by associating it between frames. The frame selection unit 3013 selects a suitable frame suitable for face matching from a plurality of predetermined frames (i.e., a set of tracked frames) including a predetermined matching target. The frame selection unit 3013 dynamically selects one or more matching frames from the tracked frame set according to the number of matching frames among the plurality of predetermined frames including the predetermined matching target. The matching unit 3014 matches the feature information of the predetermined matching target image included in the matching frame selected by the frame selection unit 3013 with the feature information of the registered image registered in the passerby registered image database 1121. The interface 305 outputs the verification results and the like to an external device. Further, the interface 306 accepts settings of parameters for processing by the frame selection unit 3013 and the matching unit 3014, and inputs the accepted settings to the frame selection unit 3013 and the matching unit 3014.

Furthermore, when it is necessary to read ID information when participating, for example, the image input unit 221 reads the ID information from an ID card or the like, and the ID information database 114 stores the read ID information. Further, the ID confirmation unit 3015 confirms whether the read ID information is registered ID information stored in the ID information database 114, and the interface 305 outputs the confirmation result of the ID information.

Here, we will supplement about person identification based on ID information. The ID registration unit 113 acquires ID information that identifies a person passing through the area. For example, the ID registration unit 113 acquires ID information via a medium such as an IC card, a QR code, or a magnetic card. Alternatively, the ID registration unit 113 acquires a facial image taken with a camera, acquires a facial image read with a scanner, acquires a facial image file from outside via media or a network, and registers at least one piece of information that can identify an individual. . Note that the ID registration unit 113 acquires feature information from the face image through processing similar to that of the matching unit 3014. Further, ID information may be managed in association with information such as name, age, gender, etc., and credit card information for billing.

The ID confirmation unit 1015 uses the ID information input from the image input unit 211 to match it with the ID information database 407 to identify the person who has currently passed by. Furthermore, the ID confirmation unit 1015 may identify the person who has currently passed by by comparing the facial feature information obtained by the registration unit 1014 with the passerby registration image database 1121. By identifying a passerby, information on which person is in which area can be output from the interface 105, the traffic history of the identified passerby can be managed, and charges can be charged to the identified passerby. It is possible.

Similarly, the ID confirmation unit 3015 uses the ID information input from the image input unit 221 and matches it with the ID information database 407 to identify the person who is currently passing through. Furthermore, the ID confirmation unit 3015 may identify the person who has currently passed by by comparing the facial feature information obtained by the matching unit 3014 with the passerby registration image database 1121.

The matching process according to each of the embodiments described above is as follows, and according to each embodiment, a decrease in matching accuracy is prevented when a frame that meets the criteria is not obtained or when there is a shortage of frames that meet the criteria. It is possible to provide an image matching device, an image matching system, and an image matching method that can perform the following steps.

In the image matching system according to each embodiment, if there are at least n (n≦N) frames that meet a predetermined condition among N input frames, m1 frames among them (m1≦n) is selected as the frame to be used for face matching.

In addition, if the number of compatible frames that satisfy a predetermined condition among the input N frames is less than n (n≦N) frames, the image matching system will automatically select all N frames or m2 frames (m2≦N). is selected as the frame to be used for face matching.
The predetermined conditions include face size (resolution), face certainty score, face brightness distribution, face position in the image, face orientation, and one or more of the matching optimality index learned from the matching score. This is a condition that is determined by combining the following. Note that the matching optimality index is an automatic learning of a best shot evaluation formula in which an image with a high degree of similarity between images of the person is selected as the best shot.

That is, the image matching system according to each embodiment performs face matching using multiple frames of face images obtained by tracking faces in a video, and determines whether or not each frame is suitable for face matching. A score is given to indicate the degree of matching (the higher the score, the closer the match is), and the frames used for matching are selected as follows.

1) If there is a frame with a score higher than a predetermined standard, remove the frame with a low score based on the predetermined standard and use the remaining frames for matching.

2) If there are no frames with scores higher than a predetermined standard, frames are selected using a method different from 1). For example, most or all frames are used for matching, even if they are lower than the standard.

This makes it possible to ensure matching accuracy by increasing the number of images that can be used for matching even in environments or subjects where it is difficult to capture good images. It also has the effect of making it easier to ensure accuracy even when the images at the time of registration and the time of verification are both in poor environments.

Applications of this embodiment include entry/exit control devices for security areas or buildings, immigration control devices, entry/exit gates (ticket gates) for public transportation, and access control for devices that can only be used by specified persons.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.
Hereinafter, the invention described in the original claims of this application will be additionally described.
[C1]
An image matching device with software installed on a computer, this image matching device:
an acquisition unit that continuously acquires matching target images from the imaging device;
detection means for detecting one or more matching targets from a plurality of input frames included in the matching target image;
a tracking means for tracking a predetermined matching target or extracting a set of frames including the matching target based on the detection result of the matching target;
Frame selection means for dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of compatible frames that meet a criterion among the plurality of predetermined frames including the predetermined matching target;
a matching means for matching a registered image registered in a database and a predetermined matching target image included in the matching frame on a one-to-one or one-on-multiple basis;
an output means for outputting the matching results;
An image matching device comprising:
[C2]
The frame selection means selects the verification frame from the plurality of predetermined frames using a first selection method when the number of compatible frames is equal to or more than a specified number, and when the number of compatible frames is less than a specified number. The image matching device according to [C1], wherein the matching frame is selected from the plurality of predetermined frames by a second selection method.
[C3]
The image matching device according to [C1] or [C2], wherein the frame selection means selects, when the number of matching frames is a specified number or more, the matching frames that are less than or equal to the specified number as the matching frames.
[C4]
The image matching device according to any one of [C1] to [C3], wherein the frame selection means selects the plurality of predetermined frames as the matching frames when the number of compatible frames is less than a predetermined number.
[C5]
Any one of [C1] to [C3], wherein the frame selection means selects extracted frames extracted at regular intervals from the plurality of predetermined frames as the matching frame when the number of compatible frames is less than a specified number. or one image matching device.
[C6]
Any one of [C1] to [C3], wherein the frame selection means selects an extracted frame extracted based on time from the plurality of predetermined frames as the matching frame when the number of compatible frames is less than a specified number. or one image matching device.
[C7]
The image matching device according to [C6], wherein the frame selection means extracts a predetermined number of new or old upper frames based on time.
[C8]
The frame selection means groups the plurality of predetermined frames into frame groups based on the amount of change in the plurality of predetermined frames when the number of the compatible frames is less than a specified number, and selects an extracted frame extracted from each frame group as the matching frame. Any one of the image matching devices [C1] to [C3] to be selected.
[C9]
The frame selection means selects the number of matching frames that are determined to meet the criteria based on at least one of the size of the matching target, the probability score, the brightness distribution, the position, the orientation, and the matching optimality index. The image matching device according to any one of [C1] to [C8], which dynamically selects the matching frame accordingly.
[C10]
The image matching device according to any one of [C1] to [C9], wherein the matching target is a partial or entire image of a person, animal, vehicle, aircraft, or drone.
[C11]
The matching means is an image matching device according to any one of [C1] to [C10], which matches the whole body, face, eyes, or gait of a person identified based on the registered image and the predetermined matching target image. .
[C12]
An image matching device of any one of [C1] to [C11],
a registration device that registers the registration image from a still image or a video;
an input device that inputs the matching target image from a video;
An image matching system equipped with
[C13]
An image matching device of any one of [C1] to [C11],
a registration device that registers the registration image from a captured image of a passing area corresponding to an entry gate; an input device that inputs the matching target image from a captured image of a passing area corresponding to an exit gate;
An image matching system equipped with
[C14]
The image matching system according to [C12] or [C13], wherein the registration device includes a registration unit that registers the registered image and ID information in association with each other.
[C15]
Obtain the matching target image,
detecting one or more matching targets from a plurality of input frames included in the matching target image;
Tracking a predetermined verification target based on the detection result of the verification target;
dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of conforming frames that meet a criterion among the plurality of predetermined frames including the predetermined matching target;
comparing the registered image registered in the database with a predetermined matching target image included in the matching frame;
An image matching method that outputs matching results.

100... Registration device 101... Image input unit 101... Processor 102... Detection unit 102... Main memory 103... Auxiliary storage device 104... Interface 105... Interface 111... Registration unit 112... Registered image database 113... ID registration unit 114... Information database 200 ...Image matching device 200...Input device 201...Image input section 210...Input device 210 provided correspondingly...Input device 211...Image input section 220...Input device 220 provided correspondingly...Input device 221...Image input section 300 ...Image matching device 300...and image matching device 301...processor 302...main memory 303...auxiliary storage device 303...and auxiliary storage device 304...interface 305...interface 306...interface 407...information database 1011...detection section 1012...tracking section 1013 ...Frame selection section 1014...Registration section 1015...ID confirmation section 1121...Passerby registration image database 3011...Detection section 3012...Tracking section 3013...Frame selection section 3014...Verification section 3015...ID confirmation section

Claims (4)

An image matching device with software installed on a computer, this image matching device:
an acquisition unit that continuously acquires matching target images from the imaging device;
detection means for detecting one or more matching targets from a plurality of input frames included in the matching target image;
a tracking means for tracking a predetermined verification target based on the detection result of the verification target;
Frame selection means for dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of compatible frames that meet a criterion among the plurality of predetermined frames including the predetermined matching target;
a matching means for matching a registered image registered in a database and a predetermined matching target image included in the matching frame on a one-to-one or one-on-multiple basis;
an output means for outputting the matching results;
Equipped with
The frame selection device may be configured to select extraction frames extracted at regular intervals from the plurality of predetermined frames as the matching frames when the number of matching frames is less than a predetermined number. An image matching device with software installed on a computer, this image matching device:
an acquisition unit that continuously acquires matching target images from the imaging device;
detection means for detecting one or more matching targets from a plurality of input frames included in the matching target image;
a tracking means for tracking a predetermined verification target based on the detection result of the verification target;
Frame selection means for dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of compatible frames that meet a criterion among the plurality of predetermined frames including the predetermined matching target;
a matching means for matching a registered image registered in a database and a predetermined matching target image included in the matching frame on a one-to-one or one-on-multiple basis;
an output means for outputting the matching results;
Equipped with
The frame selection device may be configured to select an extraction frame extracted based on time from the plurality of predetermined frames as the matching frame when the number of compatible frames is less than a specified number. 3. The image matching device according to claim 2, wherein said frame selection means extracts a predetermined number of newer or older frames based on time. An image matching device with software installed on a computer, this image matching device:
an acquisition unit that continuously acquires matching target images from the imaging device;
Detection means for detecting one or more matching targets from a plurality of input frames included in the matching target image;
a tracking means for tracking a predetermined verification target based on the detection result of the verification target;
Frame selection means for dynamically selecting one or more matching frames from the plurality of predetermined frames according to the number of compatible frames that meet a criterion among the plurality of predetermined frames including the predetermined matching target;
a matching means for matching a registered image registered in a database and a predetermined matching target image included in the matching frame on a one-to-one or one-on-multiple basis;
an output means for outputting the matching results;
Equipped with
The frame selection means groups the plurality of predetermined frames into frame groups based on the amount of change in the plurality of predetermined frames when the number of the compatible frames is less than a specified number, and selects an extracted frame extracted from each frame group as the matching frame. Image matching device to select.