JP2021086573A - Image search apparatus, control method thereof, and program - Google Patents

Abstract

To provide a search apparatus which performs high-accuracy search even when there are a plurality of objects similar in feature quantity.SOLUTION: An image search apparatus 120 includes: a feature quantity extraction unit which extracts features of an object included in an input image; a normal search unit which searches a storage unit storing an image and features represented by multidimensional vector of the object in association with each other, to obtain a candidate image group including objects having features similar to the features extracted by the extraction unit; a different object group generation unit which determines whether a non-identical entity exists in the objects included in the candidate image group; a re-search unit which obtains, when it is determined that a non-identical entity exists, a candidate image group including objects having features similar to the features extracted by the extraction unit in a feature space where a difference in feature between entities is enhanced, by searching the storage unit; and a display unit which outputs, when a non-identical entity does not exist, a search result of the normal search unit, or displays a search result of the re-search unit when a non-identical entity exists.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image retrieval device, a control method and a program thereof.

In recent years, a system that uses a person image as a query and searches for a scene including the person from an image taken by a surveillance camera has been put into practical use. As a typical example, in order to follow the footsteps of a criminal suspect, it is possible to search the scene in which the suspect was shown in the surveillance video. In addition, it can be used to smoothly manage the event by grasping the places visited by VIPs at the event venue.

When performing a search using a person image, it is common to extract a feature amount consisting of a multidimensional vector from the image and perform the search based on the similarity of the feature amount. Naturally, it is desirable that similar features are extracted from the same person. However, due to differences in the way the images are projected due to differences in lighting conditions, shooting directions, postures, etc., there may be differences in the amount of features even for images of the same person. Then, the degree of similarity with the feature amount of another person may be higher than the feature amount of the same person, which leads to deterioration of the search accuracy. Especially when comparing the features of people wearing similar clothes such as suits, this problem is large because only slight differences such as ties and shoes are observed.

Patent Document 1 discloses a method for improving the search accuracy by emphasizing the features peculiar to the search query image in the original search of the printed image. Specifically, among the search results obtained using the query image, the difference area between the search result images having a high degree of similarity is determined, and the degree of similarity between the search result and the search query image is focused on the difference area. Is to update.

In Patent Document 2, attribute information having high discriminating ability is specified based on the attribute information of the search query image and the statistical information of the attribute information of the image taken by a person different from the person in the search query image. A method of improving the discriminating ability by increasing the weight of the attribute information is disclosed.

In addition, the feature amount used for person search and the feature amount matching method are also used for person detection for finding a person registered in a list in real time from a video shot in real time, and for person tracking for tracking the found person. Will be done.

Further, the technique for determining whether or not the images are the same based on the similarity of the images is not limited to the person image. For example, it is also used to identify objects that are similar but have different small parts, such as pets such as dogs and cats, artifacts excavated from archaeological sites, and handmade crafts.

JP-A-2017-138744 Japanese Patent No. 6254836

In an environment where there are many other people wearing clothes similar to the person in the query image, there is a high possibility that another person wearing clothes similar to the person in the query image will be ranked higher in the search results. In other words, the true person the user wants to find is ranked lower and is more likely to be buried in the results of another person.

Patent Document 1 aims at the effect of emphasizing the difference between the correct answer and the incorrect answer by paying attention to the difference region between the images obtained by the search. However, it is assumed that there are many images in the search target that differ only in a small part of the area such as the store name, such as leaflets, and it is assumed that correct and incorrect answers are mixed at the top of the search results. It has become. In a person image search, even if the image is the same person, the feature amount differs depending on the shooting conditions, so that there may be a case where the correct answer does not exist at the top of the search results. If only the incorrect answer exists at the top of the search results, and the incorrect answer image group shows the same person (another person with similar clothes), the difference between the top results cannot be obtained and the different person. It is not possible to emphasize the unique features in between. In this case, in the method of Patent Document 1, the similarity obtained in the initial search cannot be updated, and the rank of the person desired by the user cannot be increased.

In Patent Document 2, an image of a person taken at the same time as the time when the search query image was taken is regarded as an image taken by a person other than the person himself / herself. Then, the attribute information of the person is extracted from the search query image, and the attribute information is also extracted from the image of the person other than the person. The discrimination performance of the attribute information is evaluated based on the statistical information of the person and the attribute information other than the person. By increasing the weight of attribute information with high identification performance to perform identification, the ability to distinguish between the person and others is improved. In this method, since the statistical information of all the persons taken at the same time as the query image is used, the effect of improving the discrimination ability of the persons wearing similar clothes is not great. Therefore, the desired effect of improving the rank of the person the user wants to find cannot be obtained.

The present invention has been made in view of the above problems, and improves the discriminating ability by finding a plurality of non-identical persons from the search results and increasing the weight of the difference in the feature amounts of the plurality of persons. It is intended to provide the technology to do.

In order to solve this problem, for example, the image search device of the present invention
An extraction method that extracts the characteristics of the object contained in the input image,
An object having features similar to the features extracted by the extraction means by searching for a storage means that stores the features represented by the multidimensional vector and the image in association with each other in the object included in the image. A first search method for obtaining candidate images including objects, and
A determination means for determining whether or not the objects included in the candidate image group have individuals that can be regarded as non-identical to each other based on preset conditions.
When it is determined by the determination means that there are individuals that can be regarded as non-identical to each other, the extraction is performed in a feature space in which the weights of each dimension of the multidimensional vector are adjusted so as to emphasize the difference in features between the individuals. A second search means for obtaining a candidate image group including an object having a feature similar to the feature extracted by the means by searching the storage means, and
When the determination means determines that the individual does not exist, the search result by the first search means is output, and when the determination means determines that the individual exists, the second It has an output means for outputting the search result by the search means of.

According to the present invention, it is possible to obtain search results with higher accuracy even when a plurality of objects having similar features are present.

The system block diagram in the 1st Embodiment. The schematic diagram of the feature quantity space in 1st Embodiment. The flowchart which shows the search processing procedure in 1st Embodiment. A flowchart showing a process of creating another object group. The figure which shows the application example of the creation process of another object group. The figure for demonstrating the method of integrating the weight vector of a plurality of different target groups. Diagram to illustrate how to show the user where the differences are highlighted. The figure for demonstrating the weight vector calculation method at the time of combining a whole body feature amount and a face feature amount. The system configuration diagram in the third embodiment. Flow chart showing the detection processing procedure in the third embodiment The figure which shows the application example of another object group creation processing in 3rd Embodiment. A diagram for explaining the difference between a query person image and a similar image. Diagram to illustrate the differences between groups.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
The image search device described in the first embodiment is an example applied to a server that analyzes an image transmitted from a photographing device and presents a search result to a user.

FIG. 1A is a hardware configuration diagram of the image search device according to the present embodiment. The CPU 101 controls the entire apparatus by executing the control program stored in the ROM 102. The control program stored in the ROM 102 includes an object extraction unit 103, a feature amount extraction unit 104, a feature amount registration unit 105, and a feature amount search unit 106. The RAM 107 temporarily stores various data from each component. In addition, the program is expanded so that the CPU 101 can be executed.

The storage unit 108 stores the data to be processed and stores the data to be searched. As the medium of the storage unit 108, an HDD, a flash memory, various optical media, or the like can be used. The input unit 109 is composed of a keyboard, a touch panel, and the like, and receives input from the user, and also receives input of search conditions and a query image selected by the user. The display unit 110 is composed of a liquid crystal display or the like, and displays the search result to the user. In addition, this device can communicate with other devices such as a photographing device via the communication unit 111. The storage unit 108 does not necessarily have to be provided inside the image search device. When it is outside the image storage device, the data to be processed is acquired via the communication unit 111.

FIG. 1B shows a data flow including the configuration of FIG. 1A. The present device (image search device 120) receives the video data 112 acquired by the external photographing device 121 via the communication unit 111. The image search device 120 extracts the registered feature amount 113 from the video data 112 and registers it in the feature amount DB 114. Further, the image search device 120 searches the feature amount DB 114 for a feature amount similar to the query feature amount extracted from the query image input from the input unit 109.

In the data registration process of the present device, the video data 112 received by the object extraction unit 103 is analyzed to extract the object displayed in the video. As the object, a whole body area or a face area of a person can be used. In this embodiment, the whole body area of a person is used as an object. In the present embodiment, it is assumed that the face region is not photographed with a sufficient size. In that case, it is difficult to identify by the feature amount of the face, so the identification is performed by the feature amount of the whole body area of the person. Next, the feature amount extraction unit 104 extracts the registered feature amount 113 from the object extracted by the object object extraction unit 103, and supplies the extracted feature amount to the feature amount registration unit 105. The feature amount registration unit 105 registers the received feature amount in the feature amount DB (database) 114. In this feature amount DB 114, not only a large number of feature amounts are accumulated, but also a thumbnail image obtained by cutting out an object area from the video data 112, shooting time information, and shooting camera information are registered, and these information are registered at the time of searching. Is also available. The feature amount DB 114 may be arranged in the storage unit 108, or may be expanded on the RAM 107 in a state of being read from the storage unit 108.

In the search process of the present device, first, the query image 115 is received from the user via the input unit 109. The method of inputting the query image by the input unit 109 is not particularly limited. For example, a photograph prepared by a user showing a target person is read from a scanner. Further, a storage medium storing an image file in which the target person is captured may be set in the present device, and the present device may read the image file. In addition, an image file showing the target person may be received by using a communication technology such as e-mail. If a plurality of people are shown in the query image, one of them shall be specified by the user. In any case, when the input unit 109 inputs the query image 115, the object extraction unit 103 extracts the object reflected in the image with respect to the query image 115 as in the registration process, and the feature amount extraction unit 104 extracts the query feature amount 116 from the image of the object.

The normal search unit 117 performs a search process for a feature amount similar to the query feature amount 116 from the feature amount DB 114. The object selection unit 122 selects a plurality of objects from the search results of the normal search unit 117. The separate object group creation unit 118 determines whether or not a plurality of selected objects are the same object, and if it is determined that they are not the same object, another object is selected from the plurality of objects. Create an object group. In the case of a person, a group composed of a plurality of different people who are not the same person will be created. The re-search unit 119 re-searches the feature amount DB 114 by using the feature amounts of a plurality of persons belonging to another object group. As a result of the re-search, the re-search unit 119 outputs a thumbnail image associated with a feature amount having a high degree of similarity to the display unit 110 as a search result.

The input unit 109 and the display unit 110 may be provided in another device, or may send and receive queries and search results via the communication unit 111. Further, some of the control programs may be provided in a separate device. For example, the object extraction unit 103 and the feature amount extraction unit 104 may be provided in another server, and the present device may have only the feature amount registration unit 105 and the feature amount search unit 106. In this case, it is possible to separate the devices for the process of analyzing the video and the process related to the search, and it is possible to select the hardware specifications suitable for each process.

The object extraction unit 103 learns the characteristics of the area of the object in advance, and then searches the input image with the search window to determine whether or not each search window is the area of the object. Apply the method. As a learning method, a deep learning method such as ensemble learning or CNN (Convolutional Neural Network) can be used. The feature amount extraction unit 104 may extract the color histogram of the partial region in the area of the object as the feature amount, or may use a method of extracting by a deep learning method such as CNN.

With reference to FIGS. 12A and 12B, the outline of the effect of improving the discrimination performance between the person and the other person in the present embodiment is shown. FIG. 12A is a list of information on the appearances of three persons wearing similar clothes. The three persons are the person who is the same person as the search query, another person A who is different from the person in the query image, and another person B who is different from the person in the query image and the person A. Since the ratio of the upper and lower suit areas to the whole body image area is large, the person's image, another person A, and another person B are close to each other in the feature space. When the similarity search using the query image is performed, the score of the person image, the other person A, and the other person B is high. Even if there are differences in shirts, ties, shoes, bags, skin color, hair, etc., they make up a small proportion of the overall picture, so noise factors such as differences in shooting conditions, person orientation, and background information The score of another person may be higher than that of the person himself / herself. Therefore, even if a search is performed using a query image, a large number of others will be confused in the search results.

FIG. 12B summarizes the differences in appearance. In this example, the difference between a tie and hair is common to all three. Therefore, if the difference between another person A and another person B is emphasized, the ability to distinguish between the person and another person A is also improved. In other words, if it is possible to select a person who is different from each other from the people who are at the top of the search results and emphasize the difference in the feature amount between the people, the ability to discriminate between the people wearing similar clothes can be improved. Can be improved.

The situation will be further described with reference to FIGS. 2A, 2B, and 2C. FIG. 2A shows the distribution of the feature amount on the feature space 1 in which the first search is performed. Although it is expressed in two dimensions for the sake of clarity, the number of dimensions of the feature quantity may actually range from several hundreds to several thousand. It should be understood that the illustrations are merely examples.

The “x” mark indicated by the reference code F101 in FIG. 2 (A) is the query feature amount, the black square mark indicated by the reference code F102 is the feature amount of the person, and the black triangle mark indicated by the reference code F103 is the other person A. The black circles indicated by the reference reference numeral F104 represent the feature amount of another person B. Since they wear similar clothes, the features of the query, the person, the other person A, and the other person B are densely packed in the feature space. When the search is performed using the feature amount at the position indicated by the “x” mark F103, the person, the other person A, and the other person B are mixed in the rank. Three people are given as an example with an emphasis on easy-to-understand explanations, but in the actual search scene, a search is performed on a scale of thousands or tens of thousands of people, so more people are assigned to a specific area of the feature space. It will be crowded, and the search results will be flooded with information about others.

Here, it is assumed that the two feature quantities surrounded by the dotted line circle in FIG. 2 (A) satisfy the condition that they are other people. The feature space 2 shown in FIG. 2B is a modification of the feature space so as to emphasize the portion where the feature quantities of the other person A and the other person B have low commonality. In this space, the difference between the shirt, tie, shoes, and hair, which is the difference between the other person A and the other person B, is emphasized, so that the degree of separation between the other person A and the other person B in the feature space is higher than that in the feature space 1. FIG. 2C is a mapping of the query image and the feature amount of the person in the feature space 2. When there is a common difference between the person, the other person A, and the other person B as in the example shown in FIGS. 12 (A) and 12 (B), emphasize the difference between the other person A and the other person B. Then, the degree of separation between the principal and the other person A and the principal and the other person B in the feature space is also increased, and the search result is improved.

The processing corresponding to each step of the flowchart described in the present embodiment may be realized by software using a CPU or by hardware such as an electronic circuit. Among the above-mentioned operations, the search-related processing will be described according to the flowchart.

FIG. 3 shows the processing of the feature amount search unit 106 in the present embodiment. In this flow, S401 to S403 are the processes of the normal search unit 117, S404 is the process of the object selection unit 122 and the separate object group creation unit 118, and S405 and subsequent processes are the processes of the re-search unit 119. .. It is assumed that the search conditions have been entered in advance.

First, in S401, the normal search unit 117 accepts the input of the query feature amount extracted from the query image input by the user. Since the query feature amount is a feature amount located at a specific location in the feature space 1, it means that the feature amount has been selected. When the query feature amount is input, in S402, the normal search unit 117 calculates the degree of similarity with the feature amount stored in the feature amount DB 114. The feature amount of the target for which the similarity is calculated may be all the feature amounts in the feature amount DB 114, and when it is desired to search for a person reflected in a specific time or camera, the feature amount is collated by the shooting time or the camera name. The target may be narrowed down. The reciprocal of various distance functions can be used for the similarity of the features, and the similarity is calculated so that the closer the distance, the higher the similarity. In this embodiment, the Euclidean distance is used for the distance calculation.

In S403, the normal search unit 117 sorts the search results in descending order of similarity based on the similarity calculated in S402, and assigns a ranking. At this time, only candidates whose similarity is equal to or higher than a predetermined value are left as search results. If there are a large number (plurality) of candidates whose similarity is equal to or higher than a predetermined value, only the candidate group whose ranking is equal to or higher than the predetermined rank may be left as the search result.

In S404, the object selection unit 122 and the separate object group creation unit 118 create another object group. It is assumed that the search results are narrowed down to the top N results in S403.

Here, the process of creating another object group in S404 will be described below with reference to the flowchart of FIG. In FIG. 4, S501 to S504 and S510 to S513 are processes performed by the object selection unit 122, and S505 to S509 are processes performed by another object group creation unit 118.

In S501, the object selection unit 122 sets “1” in the control variable i. In S502, the object selection unit 122 selects the object whose search result ranking is i-rank as the first object. In S503, the object selection unit 122 sets “1” in the control variable j. In S504, the object selection unit 122 selects an object having a ranking of "i + j" as the second object.

In S505, the separate object group creation unit 118 determines whether or not the camera that captured the first object and the camera that captured the second object are the same. The separate object group creation unit 118 proceeds to S508 if the determination is Yes, and to S506 if the determination is No.

In S506, the separate object group creation unit 118 determines whether or not the camera that captured the first object and the camera that captured the second object do not share the field of view. This determination is made based on the information that defines the installation position and the imaging direction of each camera stored in advance in the storage unit 108. The separate object group creation unit 118 proceeds to S507 if the determination is Yes, and proceeds to S508 if the determination is No.

In S507, the separate object group creation unit 118 determines whether or not there is an overlap between the shooting period of the first object and the shooting period of the second object. The separate object group creation unit 118 proceeds to S509 if the determination is Yes, and proceeds to S508 if the determination is No.

When the process reaches S508, it means that the first object and the second object do not satisfy the condition that they are different objects. Therefore, the separate object group creation unit 118 does not determine that the first object and the second object are different object groups.

On the other hand, when the process reaches S509, the condition that the first object and the second object are different objects (condition that they are not the same) is satisfied. Therefore, the separate object group creation unit 118 determines that the first object and the second object are different object groups.

In S510, the object selection unit 122 adds “1” to the control variable j. Then, in S511, the object selection unit 122 determines whether or not the control variable j has reached the Nth position, which is the lowest in the ranking. The object selection unit 122 proceeds to S512 if the determination is Yes, and to S504 if the determination is No. That is, when the process proceeds to S504, the object selection unit 122 reselects a new second object.

In S512, the object selection unit 122 adds “1” to the control variable i. Then, in S513, the object selection unit 122 determines whether the control variable i has reached the N-1 position, which is one higher rank than the lowest in the narrowed down ranking. If the determination is Yes, the object selection unit 122 ends the process of creating another object group. If the determination is No, the object selection unit 122 advances the process to S502, reselects the first object, and repeats the above process.

An example in which a separate object group is created based on the separate object group creation process of FIG. 4 will be described with reference to FIGS. 5 (A) to 5 (C).

FIG. 5A is an example of the primary search result. In the figure, it is assumed that the search results having a score (similarity) of 780 points or more are extracted as the primary search results. Further, it is assumed that the cameras are monitored by two cameras (camera 1 and camera 2), and the two cameras are arranged so as not to share the shooting field of view. Since the shooting field of view is not shared, the same person cannot be shot by two cameras at the same time. In FIGS. 5B and 5C, the horizontal axis represents the shooting period of each search result, and the vertical axis represents the x-coordinate or y-coordinate of the image coordinate of the detection position of the object, and plots the image. In order to perform tracking across frames, it is necessary to determine the same person among a plurality of frames. To determine the same person between frames, a method of extracting color histogram features from image data and using a particle filter or the like can be used. It should be understood that the person whose ID is "36" shown in FIG. 5 (A) corresponds to "ID 36" in FIG. 5 (B).

According to the present embodiment, the object with ID 36 having the first rank is first selected as the first object. Here, it is assumed that the object with ID 56 having the fourth rank is selected as the second object. The shooting time of the person with ID 56 is t1, and the shooting period overlaps with the person with ID 36, which is the first object. Therefore, the persons with ID 36 and ID 56 are determined to be different object groups. These two people will be referred to as Group A. Similarly, it is assumed that the object with ID 62 having the third rank is selected as the first object. At this time, when the ID 61 of the fifth rank is selected as the second object, it is determined to be another object group because the shooting period overlaps at time t2. These two people are referred to as group B.

Returning to the description of FIG. In S405 of FIG. 4, the re-search unit 119 determines whether or not another object group has been created, and if it is determined that it has been created, the process is set to S406, and if it is determined that the process cannot be created, this process is performed. finish.

_{In S406, the re-search unit 119 uses the weight vector w = (w 1)} as a parameter group used to transform the feature space by using the feature quantities of a plurality of persons in another object group created in S404. , w _2, ..., to calculate the w _d). Here, d is the number of dimensions of the feature space, and the weight vector sets the weighting coefficient for conversion for each dimension. By multiplying each feature quantity by this weight vector, the feature space conversion can be performed.

The feature amount of the person A in the different object group is a, the feature amount of the person B in the different object group is b, and the weight vector is calculated by the following equations (1) and (2).

In the equation (1), the difference between the feature amount of the person A and the feature amount of the person B is calculated for each dimension, and the weight for each dimension is calculated. According to this equation, the larger the difference between the numerical values of the feature amount of the person A and the feature amount of the person B, the larger the weight value. The dimension in which the difference between the feature amount of the person A and the feature amount of the person B is large is considered to well indicate the feature for distinguishing between a plurality of people having an appearance similar to the query image. By increasing the weight of, the difference between the person A and the person B can be emphasized. When the person in the query image is either person A or person B, there is an effect of improving the ability to distinguish between the person and another person. Even when the person in the query image is neither the person A nor the person B, the effect of improving the discriminating ability between the person and the person A in the query image and the person and the person B in the query image can be obtained.

In equation (2), normalization is performed to obtain the final weight value. By normalizing so that the sum of the weights of each dimension is the number of dimensions d, it is possible to reduce the difference between the absolute values of the distances in the feature space before and after the conversion.

The above-mentioned discriminating ability evaluates the relative degree to which one person and another person can be expected to be discriminated from each other. For example, it can be calculated by the equation (3) obtained by multiplying the sum of squared deviations of the weights and the ratio of the effective dimensions calculated based on the effective dimensions of the weights in the equation (4).

When the weights of each dimension in the weight vector are uniform, the discrimination ability does not increase because there is no change from the original feature space even if the weight vector is multiplied. Therefore, in order to improve the discriminating ability, the weights need to be different for each dimension to some extent. Therefore, in order to express the degree of variance of the weights for each dimension, the sum of squared deviations is used in the first term of the equation (4).

On the other hand, even if there are variations in weights, if the weights are concentrated in only a small number of dimensions and most of the dimensions have weights close to 0, the number of dimensions that contribute to identification is significant. It is conceivable that the number will decrease and the distinctiveness will decrease. In order to reflect this characteristic, the ratio of dimensions in which the weight is equal to or greater than the predetermined reference value is used in the second term of the equation (4). You may simply use the number of dimensions instead of the proportion of dimensions.

On the other hand, the smaller the difference between the numerical values of the feature amount of the person A and the feature amount of the person B, the smaller the weight value. A dimension with a small difference in numerical values is considered not to contribute to discrimination between a plurality of people having an appearance similar to a query image, so the weight is reduced, and the influence of the similarity of this dimension on the search result is reduced.

In S407, the re-search unit 119 determines whether or not the number of different object groups is two or more. When the re-search unit 119 determines that the number of different object groups is one, the process proceeds to S409, and when it is determined that the number of different object groups is two or more, the process proceeds to S408.

In S408, the re-search unit 119 integrates the weight vectors for each different object group. As a method of integration, there is a method of leaving the maximum value among the values of the weight vector of each group. When the dimension to be emphasized in each group is different, the information of the dimension to be emphasized in each group can be saved by leaving the maximum value.

For example, there are two groups, the weight vector calculated in group 1 is w ₁ = (w ₁₁ , w ₁₂ , ..., w _1d ), and the weight vector calculated in group 2 is w ₂ = (w ₂₁ , w). ₂₂ , ..., w _2d ). At this time, if the weight vector after integration is w _u = (w _u1 , w _u2 , ..., W _ud ), w _u is calculated by the following equation (5).

ここで、max(…)は、括弧内に並んだ数値列の中の最大値を返す関数である。

Here, max (...) is a function that returns the maximum value in the numerical sequence arranged in parentheses.

Here, the reason why the discriminating ability is improved by integrating the groups will be described with reference to FIG. 13, FIG. 5, and FIG.

In FIG. 5, two people with ID36 and ID56 are group A, and two people with ID61 and ID62 are group B. Further, in order to facilitate understanding, it is assumed that the person with ID 62 is the person himself, the person with ID 36 is the person A, and the people with ID 56 and ID 61 are the person B. FIG. 13 summarizes the differences between the two members in group A and the differences between the two members in group B. The ○ mark is the part where there is a difference, and the × mark is the part where there is no difference. Here, when the two groups of group A and group B are integrated, if the differences between the groups are integrated so as to be preserved as described in the fourth column of FIG. 13, the groups are individually integrated. The discriminating ability after integration is improved rather than the discriminating ability.

A more specific description will be given with reference to FIGS. 6 (A) to 6 (G). In the graph of the figure, the horizontal axis represents the number of dimensions and the vertical axis represents the magnitude of the feature amount of each dimension. FIG. 6A shows the feature amount of another person A, and FIG. 6B shows the feature amount of another person B. FIG. 6 (C) shows the difference in the feature amount between the other person A and the other person B belonging to the group A by the above-mentioned formula (1). Since the color of the shoes is different between the other person A and the other person B, the difference in the feature amount of the dimension corresponding to that part is large.

Subsequently, FIG. 6 (D) shows the feature amount of the person himself / herself, and FIG. 6 (E) shows the feature amount of the other person B. FIG. 6 (F) shows the difference in the feature amount between the person belonging to the group B and the other person B by the above-mentioned formula (1) as in the group A. Since the presence or absence of a bag is different between the person and the other person B, there is a large difference in the amount of features in the dimension corresponding to that part.

FIG. 6 (G) is the result of the integration process of the weight vector of S408 of FIG. A weight vector is calculated that emphasizes both the difference in shoes in group A and the difference in bags in group B. In some cases, people wearing similar clothes can see only one of the shoe information and the bag information, so it is better to use a weight vector that can emphasize as many differences as possible. Is expensive. Therefore, it can be understood that the discriminating ability is improved by integrating the weight vectors of multiple groups.

Returning to the description of FIG. In S409, the re-search unit 119 performs an image search by calculating the similarity between the query feature amount and the feature amount in the feature amount database 114 in the feature space after conversion using the weight vector. When calculating the similarity in the feature space after conversion, the Euclidean distance may be used after multiplying the feature quantity by the weighting coefficient (after correction), or the weighted Euclidean distance may be used. The distance in the feature space after conversion may be calculated directly. The weighted Euclidean distance between the query feature amount q and the feature amount x in the feature amount database 114 can be calculated by the following equation (6). Since the weight of the dimension effective for distinguishing the person A and the person B is large, the distance is large (the similarity is small) when the difference in the feature amount of the dimension effective for identification is large, which is effective for identification. There is an effect that the distance is small (the degree of similarity is high) when the difference between the features of the same dimension is small. In addition, the dimensions that are not effective for identification have smaller weights, and the difference between these dimensions is less likely to affect the distance calculation result.

Finally, in S410, the re-search unit 119 selects a predetermined number of features with high similarity calculated in the previous S409 from the top, and outputs thumbnail images associated with the features to the display unit 110.

The weight calculation method that can be used in the embodiment is not limited to the above equations (1) and (2). In the formula (1), the ratio of the difference may be used instead of the absolute value of the difference between the two features, or the calculated weight may be a binary value of 0 or 1. When the weight is binary, the dimension having a weight of 0 does not contribute to the similarity, so that there is an effect of dimension reduction and the amount of calculation can be reduced.

Further, when the relationship between the image position and the feature dimension is clear, the user may be presented with which position of the image the feature is different and emphasized. FIG. 7 (A) shows an example in which the image is directly superimposed and displayed, and FIG. 7 (B) shows an example in which the character information is displayed. It should be understood that these images are displayed on the display unit 110. FIGS. 7 (A) and 7 (B) show an example in which the dimension in which the difference in the feature amount is large corresponds to the part of the tie, the bag, and the shoe, and FIG. 7 (A) corresponds to the part. An example of superimposing a dotted circle on the image position is shown. The dotted circle 1101 is drawn at the position corresponding to the tie, the dotted circle 1102 is drawn at the position corresponding to the bag, and the dotted circle 1103 is drawn at the position corresponding to the shoe. In FIG. 7B, a necktie, a bag, and shoes having a difference in feature amounts equal to or greater than a threshold value are displayed as text information. Both the superimposed display on the image and the display of characters may be displayed. By presenting to the user the parts where the difference is large in this way, the user can explicitly know where the difference in appearance is between the person wearing the clothes similar to the person to be searched for.

According to this embodiment, the person to be searched for can be automatically ranked in the top of the search results even under the condition that many other people wearing clothes similar to the person to be searched for are included. This is realized by finding a plurality of different objects from the search results, increasing the weight of the dimension in which the difference between the features is large, and recalculating the score to improve the discrimination performance. When a plurality of object groups are created, the rank of the search result of the person to be searched for can be further improved by integrating the weight vectors obtained in each group. In addition, by displaying the part where the feature amount is different on the display unit, the user can clearly know where the difference in appearance is between the person wearing clothes similar to the person he / she wants to find. become.

In the above embodiment, the example of two cameras has been described, but it may be applied to three or more cameras that do not share the shooting field of view with each other. In this case, the person imaged in the time zone that can be regarded as the same time between the two cameras out of the plurality of cameras may be regarded as different persons.

As described above, according to the present embodiment, by finding a plurality of different objects from the search results and increasing the weight of the difference between the features of the search results, many parts of the features are similar. It is possible to emphasize the differences that are common to each object and improve the ability to discriminate between the objects. In particular, in a person image, the weight of the portion where there is a difference between people wearing similar clothes becomes large, so that the ability to discriminate between people can be improved.

[Second Embodiment]
In the first embodiment, the whole body area of a person is targeted as an object. On the other hand, a method for improving the discrimination ability when other feature quantities of a person are used in combination will be described as a second embodiment. What is disclosed in the second embodiment is a method for improving the discriminating ability of a person who has the same clothes and belongings, such as a student.

In the second embodiment, it is assumed that the face region is photographed with a sufficiently large size (or sufficiently high resolution), and in this case, both the identification by the facial feature amount and the identification by the whole body feature amount are performed. Will be available.

The hardware configuration diagram of the target image search device in the second embodiment is the same as FIG. 1 (A) shown in the first embodiment. Further, the data flow in the second embodiment is also the same as that shown in FIG. 1 (B) shown in the first embodiment. Therefore, the description of the device configuration will be omitted, and the difference from the first embodiment will be described.

The object extraction unit 103 extracts a human face region and a whole body region from the video data and the query image. The feature amount extraction unit 104 extracts the facial feature amount from the face area of the person and the whole body feature amount from the whole body area of the person. The feature amount registration unit 105 registers both the whole body feature amount and the facial feature amount in the feature amount database 114. The normal search unit 117 and the re-search unit 119 are different from the registered feature amount 113 registered in the feature amount database 114 by the combined query feature amount that combines the whole body feature amount and the face feature amount associated with the query feature amount 116. Find the similarity. Before calculating the similarity, the whole body features and facial features extracted from the same image are converted into combined features. If only one feature can be extracted from the image, the feature of each dimension is set to 0 at the time of combination, or the similarity is calculated with either the extracted facial feature or the whole body feature. .. Alternatively, those for which only one feature quantity can be extracted may be excluded from the search target.

The flowchart of the processing of the feature amount search unit 106 in the second embodiment is the same as that in FIG. 3 of the first embodiment. An example of calculating the weight vector of S406 from the combined feature amount of the whole body feature amount and the facial feature amount will be described with reference to FIGS. 8A to 8C.

FIG. 8 (A) shows the combined feature amount of the first person in the different object group extracted from the search results, and FIG. 8 (B) shows the second person in the different object group extracted from the search results. Shows the combined features of the person. With the positions of the dotted lines in FIGS. 8 (A) and 8 (B) as boundaries, the left side is the whole body feature amount and the right side is the facial feature amount. In the example of this figure, the person in FIG. 8 (A) and the person in FIG. 8 (B) are wearing the same clothes / shoes, and the belongings are also the same. There is no difference in the amount of systemic features. On the other hand, since the facial features have a large difference, the facial features are more important than the whole body features as a weight vector. 8 (C) is a calculation of the difference between the combined feature amount of the person of FIG. 8 (A) and the person of FIG. 8 (B). It can be seen that the difference in facial features is emphasized because the difference in facial features is large.

In the second embodiment, an example of obtaining the similarity between the combined feature amount of the face feature amount and the whole body feature amount has been described, but it is also possible to search for each of the face feature amount and the whole body feature amount. A search is performed using the query features of the whole body, a different group is created from the top of the search results, and the weight vector for the whole body is obtained. Then, a search is performed using the query feature amount of the face, another person group is created from the top of the search results, and the weight vector for the face is obtained. Each weight vector may be used for score calculation of each feature.

In the second embodiment, a case where there is no difference in the amount of whole body features and a difference in the amount of facial features has been described as an example, but the applicable range is not limited thereto. For example, it can be applied to cases where there is a difference in attribute information. As attribute information, for example, gender, age, hair color, hair length, walking style (gait), etc. can be observed as differences even if the clothes and belongings are the same. In cases where the clothes are the same and the belongings are different, information about the belongings can also be used as attribute information.

In the second embodiment, since there is no difference in the whole body feature amount in the situation where a plurality of people wear the same clothes and own the same belongings, there is a difference in the facial feature amount and the attribute information. By using the feature amount and performing a re-search, the person to be searched for can be ranked in the top of the search results.

[Third Embodiment]
A third embodiment will be described below. In the first and second embodiments described above, an example of application to an image search device has been described. In this embodiment, an example in which the image search device is applied to an image detection device that detects a person registered in a list will be described.

In this case, the image detection device registers the image of the person to be detected in the list, and when a person similar to the person registered in the list is found in the video data, the user is notified in real time. Notice. Even in the image detection device, if the appearance of the person registered in the list is similar to the appearance of a person other than the person detected in real time from the video data, the degree of similarity is high, and if the threshold value is exceeded, false detection is performed. It becomes. In the case of the image detection device, since the detection result is notified to the user each time, if erroneous detection occurs frequently, the user may not trust the detection result, and even if the person is correctly detected, it may not be confirmed. By using this embodiment, it is possible to increase the positive detection rate and reduce the frequency of false detections.

FIG. 9A is a hardware configuration diagram of the target image detection device according to the third embodiment. Reference numerals 101 to 105 and 107 to 111 of FIG. 1 (A) of the first embodiment correspond to reference numerals 201 to 205 and 207 to 211 of FIG. 9 (A). Reference numeral 206 is a feature amount detection unit.

FIG. 9B shows a data flow in the present apparatus according to the third embodiment. This device is a part shown in the target image detection device 220, and is an external photographing device 221a. Video data 212a. Acquired by each of 221b. 212b is received via the communication unit 211.

In the data registration process of the present device, the object extraction unit 203 analyzes the input detection target image 215 presented by the user via the input unit 209, and extracts a person area of a specific person from the detection target image. Next, the feature amount extraction unit 204 extracts the detection target feature amount 213 and holds the extracted feature amount in the feature amount memory 214. Since real-time processing is required in the third embodiment, the feature memory 214 is composed of a RAM capable of high-speed reading. In addition, the information of the registered detection target feature amount is so that it is not necessary to input the detection target image 215 every time. It is managed as a list in the storage unit 208. The detection target feature amount registered in the list is read out each time it is started and expanded in RAM.

In the detection process of this device, the received video data 212a. The 212b is extracted by the object extraction unit 203 as in the case of the registration process, and the feature amount extraction unit 204 extracts the real-time detection feature amount 216. The normal detection unit 217 included in the feature amount detection unit 206 calculates the similarity calculation between the extracted real-time detection feature amounts 216a and 216b and the detection target feature amount 205 held in the feature amount memory 214, and is similar. When the degree exceeds a predetermined threshold value, the detection result is displayed on the display unit 210.

Next, the object selection unit 222 selects a plurality of objects from the real-time detection result by the normal detection unit 217. The different object group creation unit 218 determines whether or not the selected multiple objects are the same object, and if it is determined that they are different objects, another object consisting of a plurality of objects. Create a group. The rediscovery unit 219 performs rediscovery by using the feature quantities of a plurality of persons belonging to another object group. Specifically, the similarity is recalculated with respect to the feature amount of the detected image detected by the normal detection unit 217, and the recalculation result of exceeding a predetermined threshold value is output to the display unit 210. After that, real-time detection is performed by the calculation method of similarity used at the time of re-detection.

Among the above-mentioned operations, the detection-related processing will be described with reference to the flowchart.

FIG. 10 shows the processing of the feature amount detection unit 206 in the third embodiment. In this flow, S801 to S802 are the processes of the normal detection unit 217, and S803 and subsequent steps are the processes of the re-detection unit 219.

First, in S801, the normal detection unit 217 receives the input of the real-time detection feature amount 216 extracted from the video data 212. When the real-time detection feature amount input is executed, the normal detection unit 217 calculates in S802 the degree of similarity with the detection target feature amount 213 of a plurality of persons held in the feature amount memory 214.

In S803, the re-detection unit 219 selects a plurality of objects whose similarity exceeds the threshold value. If a person who satisfies the condition for creating another object group is included in the plurality of objects, another object group is created. If a plurality of selected features are shot by the same camera and the shooting periods overlap, it is determined that they are different objects. Alternatively, when a plurality of selected feature quantities are photographed by a plurality of cameras that do not share a field of view and the shooting periods overlap, it is determined that they are different objects, and another object group is created. An example in which another object group is created will be described with reference to FIGS. 11A to 11C.

11 (A) shows a list of detection results, FIG. 11 (B) shows the detection result and shooting time by the camera 1, and FIG. 11 (C) shows the detection result and shooting time by the camera 2. When the IDs of the persons registered as detection targets match and the shooting times match, it is determined that the appearance similarity is high and the conditions for creating another object group are satisfied. In FIG. 11A, a person with a registered person ID of 5 is detected by the camera 1 and the camera 2 at the same time, and since these two people exist at different places at the same time, different object groups. Is created. It should be noted that features that are not normally included in the search results but whose similarity is equal to or higher than a predetermined value may be added to the creation conditions of another target group.

In S804, the re-detection unit 219 determines whether or not another object group exists. The rediscovery unit 219 proceeds to S805 when it is determined that another object group does not exist, and proceeds to S806 when it determines that another object group does exist.

In S805, the re-detection unit 219 notifies a person whose similarity exceeds the threshold value in S802 as a normal detection result, and ends this process.

In S806, the re-detection unit 219 calculates the weight vector based on the equations (1) and (2) based on the difference in the feature amount between the plurality of persons determined to be different object groups. In the examples of FIGS. 11A to 11C, the weight vector is calculated because the person with ID 5 satisfies the condition of another object group. The weight vector is calculated and saved for each feature to be detected. When there are a plurality of different target groups, the weight vectors may be integrated.

In S807, the rediscovery unit 219 calculates the similarity in the converted feature space using the weight vector. The score is recalculated using the weight vector obtained in S806 and the equation (3) for the feature amount of the image for which the registered person ID is determined to be 5 in the detection result list. Hereinafter, the score of the detection target feature amount of the person whose registered person ID is 5 is calculated by using the weight vector obtained in S806 and the formula (6). The weight vector may be updated after a predetermined time has elapsed. The weight vector may not function properly due to changes in the clothes worn by the person or changes in ambient light conditions such as sunrise and sunset due to changes in time and date. Therefore, by updating at an appropriate timing, robust detection against external factors becomes possible. In the case of a feature amount such as a face feature amount that is not affected by a date change, a mechanism may be adopted in which a weight vector is provided for each time zone.

According to the third embodiment, even if the appearance of a person registered as a detection target image in the image detection device and the appearance of a person other than the person detected in real time from the video data are similar, only the person is correctly detected. It becomes possible to do.

[Other Embodiments]
The present invention is applicable not only to an image search device and an image detection device, but also to an image tracking device. When an object detected by an image detection device is tracked in a camera or across a plurality of cameras, two or more tracking candidates may be found at the same time. In this case, since the time is the same, the two or more persons can be regarded as different persons. By calculating the weight vector based on the features of these two people, calculating the score again using the weight vector, and tracking the person with the higher score, the person can be tracked accurately. ..

The present invention can also be applied to an object other than a person. Specifically, they are very similar, but there are slight differences, and although there is only one, it can be used for image retrieval. For example, pets such as dogs and cats have slight individual differences even if they are of the same type, so the present invention can be applied. When multiple owners gather to hold a pet photo session, it takes time and effort to manually extract only the photos of their pets. By applying the present invention, it is possible to automatically extract only an image of one's pet with high accuracy. The present invention can also be used for image retrieval of pottery and handmade crafts excavated from archaeological sites. Many of the pottery excavated from the ruins are underdeveloped in manufacturing technology, so although they are similar in size, shape, and color, they are not so similar that they can be said to be the same. Alternatively, crafts such as handmade vases and works made by potters have slight individual differences in color and shape, and no two have the same appearance. The present invention can be applied when searching for an image of a desired pottery or craft from a database of images of such an object. As mentioned above, there is only one, but if there are restrictions on the shooting location and shooting time, only one may exist within that range.

The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device implement the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101 ... CPU, 102 ... ROM, 103 ... Object extraction unit, 104 ... Feature amount extraction unit, 105 ... Feature amount registration unit, 106 ... Feature amount search unit, 107 ... RAM, 108 ... Storage unit, 109 ... Input unit, 110 ... Display unit, 111 ... Communication unit

Claims (8)

An extraction method that extracts the characteristics of the object contained in the input image,
An object having features similar to the features extracted by the extraction means by searching for a storage means that stores the features represented by the multidimensional vector and the image in association with each other in the object included in the image. A first search method for obtaining candidate images including objects, and
A determination means for determining whether or not the objects included in the candidate image group have individuals that can be regarded as non-identical to each other based on preset conditions.
When it is determined by the determination means that there are individuals that can be regarded as non-identical to each other, the extraction is performed in a feature space in which the weights of each dimension of the multidimensional vector are adjusted so as to emphasize the difference in features between the individuals. A second search means for obtaining a candidate image group including an object having a feature similar to the feature extracted by the means by searching the storage means, and
When the determination means determines that the individual does not exist, the search result by the first search means is output, and when the determination means determines that the individual exists, the second search means. An image search device characterized by having an output means for outputting search results by the search means of. The storage means stores the characteristics of an object imaged by a plurality of imaging devices that do not share a photographing field of view with each other.
The input image is a query image and
When there are two or more candidates that can be regarded as the same time and are imaged by different imaging devices in the candidate group obtained by the first search means, the determination means are not identical to each other. The image search device according to claim 1, wherein it is determined that the indicated group exists. The storage means stores the characteristics of one or more objects to be searched.
The input image is a real-time image taken by a plurality of imaging devices that do not share a shooting field of view with each other.
When there are two or more candidates that can be regarded as the same time and are imaged by different imaging devices in the candidate group obtained by the first search means, the determination means are not identical to each other. The image search device according to claim 1, wherein it is determined that the indicated group exists. When it is determined by the determination means that there are a plurality of combinations of individuals that can be regarded as non-identical to each other, the second search means is a feature space in which the weighting coefficients of each dimension of the multidimensional vector calculated for each combination are integrated. The image search device according to any one of claims 1 to 3, wherein the search is performed in. The object is a person
The search device according to any one of claims 1 to 4, wherein the extraction means extracts features corresponding to the whole image of a person and an image of a face. The feature space is a feature space in which the weighting coefficient of each dimension of the multidimensional vector is adjusted, and is characterized by further having a display means for displaying information for specifying a part of an object represented by a weighting coefficient equal to or higher than a threshold value. The image search device according to any one of claims 1 to 5. It is a control method of the image search device.
An extraction process that extracts the characteristics of the object contained in the input image,
An object having features similar to the features extracted in the extraction step by searching for a storage means that stores the features represented by the multidimensional vector and the image in association with each other in the object included in the image. The first search process for obtaining candidate images including objects, and
A determination step of determining whether or not the objects included in the candidate image group have individuals that can be regarded as non-identical to each other based on preset conditions.
When it is determined by the determination step that there are individuals that can be regarded as non-identical to each other, the extraction is performed in a feature space in which the weights of each dimension of the multidimensional vector are adjusted so as to emphasize the difference in features between the individuals. A second search step of obtaining a candidate image group including an object having a feature similar to the feature extracted in the step by searching the storage means, and
When it is determined by the determination step that the individual does not exist, the search result by the first search step is output, and when it is determined by the determination step that the individual exists, the second A control method of an image search device, which comprises an output process for outputting a search result by the search process of. A program for causing the computer to execute each step of the method according to claim 7, when the computer reads and executes the process.

Images