JP2002304415A - Image search device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image search device for a user to more easily understand points on a system side at searching such as what part of a key image is watched and searched. SOLUTION: Images similar to a given key image (a) are searched from a database, and the searched images are displayed as candidate images together with ranking on a display device. At that time, parts with high similarity in the candidate images provided at similarity search are marked with a border W and pats with low similarity are masked M with half-tone dot meshing. Therefore, the user can understand which part of the key image the search device watched and searched, and selection of the key image when requesting re-searching can be effectively done.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image search device.

[0002]

2. Description of the Related Art An image search apparatus used for searching for a specific image from a large amount of image databases includes:
There are the following. In other words, when registering an image in the image database, character information (title, creator, classification, ...) as a key for specifying the image is added. This is input as information, a key that matches the condition is extracted, and the matching image data is read.

As another method, there is a method in which an image is given as a search key, features of the image are extracted, and an image having similar features is detected from an image database.
The image given as the search key may be an existing image or a rough sketch drawn by the user.
And as a search algorithm based on such an image,
For example, “QBIC system” manufactured by IBM is commercially available.

In the above-described image search using an image as a key,
Since the feature values of the image data are extracted and the feature values close to each other are detected, similar images can be detected, so that an image matching the image of the user (searcher) can be searched. Further, even when only images are stored, a target image can be searched. In addition, since the corresponding image is searched from the similarity based on the feature amount calculated by the internal algorithm, for example, the images can be displayed in order from the image having the highest similarity.

[0005]

However, the above-described image retrieval system using an image as a key has the following problems. That is, although the order determined by the image search device (image search system) is known, the evaluation result in that order is often difficult to understand for a user as a user.

That is, doubts may be generated in the order of similarity, and further, a completely different human sensation such as "Why was such an image retrieved?" is there.

As an example, suppose that a search result as shown in FIG. 1B is obtained when an image of a search key as shown in FIG. 1A is given. Then, the search key is "seascape"
However, the second image may be "sky landscape", and the third image may be "shaved ice" which is not related to the landscape. In this way, if the search result is questioned, the adverse effect is that the reliability of the system itself is reduced.

[0008] Further, it is considered that the reason that the search result differs between a human and a computer (system) is that the judgment that "similarity" includes ambiguity due to human subjectivity. Moreover, the permutation of the degree of “similarity” between humans may be different due to subjectivity.
Therefore, it can be said that it is often difficult for the computer to search for the desired image desired by the user from the beginning.

Then, whether to search for a target image using the image search device (search algorithm) is determined by selecting and inputting an appropriate image as a key.
Actually, it is complicated to prepare such images. Also, the target image may be similar to the whole of the key image, but there are many cases where the user wants to execute a search by focusing on a certain part in the image. There is a problem that it is difficult to transmit the part to the system side.

[0010] If the search result is not satisfactory, the search is performed again. In this case, it is not clear what kind of viewpoint and algorithm the device has been searched for. It is not clear how to give the given key image, and after all, the target image is detected by repeating trial and error, and much time and effort is expended.

The present invention makes it easy for a user to understand the viewpoint at the time of a search on the device (system) side, such as what part of the key image the image search device searched for, and the intention of the user. It is an object of the present invention to provide an image retrieval apparatus which can easily communicate the purpose to the apparatus side. It is another object of the present invention to provide an image search apparatus that can continue a search interactively and efficiently obtain a target image.

[0012]

In an image retrieval apparatus according to the present invention, an image database storing a plurality of images and an image similar to a given key image are searched from the image database, and the search is performed. There is provided processing means for outputting the image as a candidate image to the display device. The processing means has a function of displaying at least one of a high similarity part and a low similarity part in the candidate image on the display device.

In the embodiment, both the portion having a high similarity and the portion having a low similarity are displayed. By displaying in this way, the image search device (processing means)
Can be easily understood by the user as to which region is focused or evaluated without focusing and the candidate image is determined. Therefore, even if the user is not satisfied with the search result, the next time a key image is input, the user can easily search for the target image by referring to the above result.

The processing means divides each of the key image and the image into a plurality of identical regions, obtains a characteristic amount for each of the divided regions, and calculates a characteristic amount difference between corresponding regions. The degree of similarity of each image is obtained based on the divided area, and a portion having a high degree of similarity or a part having a low degree of similarity in the candidate image can be a unit of the divided area. In this manner, the process of displaying at least one of the high similarity portion and the low similarity portion in the candidate image can be easily performed.

Further, the apparatus has an area setting means for setting an area of interest in the search for the candidate image displayed on the display device, and the processing means performs a search with emphasis on the set area. Is preferably added. Of course, such a function is not required.

As described above, by setting the area by the area setting means, the processing means can easily recognize which part of the key image the user is particularly focusing on. As a result, the search result intended by the user is easily obtained. That is, it is possible to perform an interactive search.

On the other hand, another solution is to provide an image database in which a plurality of images are stored, acquire the image information being created, and, when the pause time of the drawing becomes equal to or more than a predetermined value, And a processing unit for searching for an image similar to the key image from the image database and outputting the searched image as a candidate image to a display device. That is. In this case, as the display mode, the key image and the candidate image may be displayed side by side, or the key image and the candidate image may be superimposed and displayed on the display device.

When performing an image search based on the rough sketch drawn by the user, the search can be performed before all the images are created, that is, during the drawing, so that the target image can be easily detected. It becomes possible. In addition, image search is performed during drawing (pause) and the result is displayed, so you can check whether the input image information is in the right direction and whether there is any missing information. By inputting necessary information in sequence, a search can be performed in an interactive manner, and a target image can be quickly detected.

The above-described components of the present invention can be combined as much as possible. Each means constituting the image search device according to the present invention can be realized by a dedicated hardware circuit, or can be realized by a programmed computer.

[0020]

FIG. 2 is a hardware configuration diagram showing a preferred first embodiment of the present invention. As shown in FIG. 1, a computer 2 has a configuration in which a mouse 2 as an input device (pointing device) and a monitor 3 as an interface with a user are connected. The computer main unit 1 has an input unit 11 connected to a data bus.
The CPU 12, the memory 13, the external storage device 14, and the output unit 15 are connected.

The external storage device 14 has an image database in which a large number of image data to be searched are stored. The command information received from the mouse 2 via the input unit 11 is given to the CPU 12. The CPU 12 performs a memory 1 as a work memory based on the given information.
3 is appropriately used, an image meeting the requirements and conditions is detected from the image database, and output and displayed on the monitor 3 via the output unit 15. As is clear from the figure, the computer main body 1 can be realized by an ordinary personal computer.

Here, in the present invention, an image search algorithm is implemented in the CPU 12. First, the concept of the present embodiment will be described. A feature amount of a normal image is extracted, and a feature amount of a key image input as a search key and a feature amount of an image registered in an image database are obtained. The degree of similarity is obtained from the degree. Then, the images are displayed as candidate images retrieved from those having a high degree of similarity.
Until the candidate image is determined, a conventional algorithm can be basically applied.

In displaying the search result, information indicating which part is determined to be similar to the key image and the search is performed is also displayed. Specifically, as shown in FIG. 3, a mask M is shaded on a portion having a low similarity. As a result, a portion having a high similarity is displayed more clearly. In addition, a frame W surrounding the area is displayed in a portion having a high similarity. Using the mask M and the frame W, the user can easily understand which part the image search device (search algorithm) has determined to have a high degree of similarity and has selected it.

Further, from the layout of the area where the mask M is placed and the area surrounded by the frame W, it is also possible to estimate how the division pattern at the time of retrieval has been set. In the example shown,
It can be predicted that a horizontal division pattern is applied.

Further, in the present embodiment, when performing a rebuilding plan on the basis of the search result, it is possible to specify a target portion of the key image. In other words, the user operates the mouse 2 to specify a portion to be emphasized in the key image. In the example of FIG. 4A, the upper “sky and cloud portion” is specified. A re-search instruction is given in a state where the part to be emphasized is designated in this way.

On the basis of this, the image search device searches again, but at this time, the search is performed by obtaining the similarity with emphasis on the feature amount (color, shape, etc.) of the designated area. As a result, as shown in FIG. 4B, a result different from the first search result is obtained.

As a matter of course, the search results are different by changing the region of interest and importance. In this way, the search can be continued in an interactive manner such that the re-search request is made together with the setting of the region to be emphasized and the search result is obtained, and the target image can be detected.

From the relation of the search result corresponding to the set important area, it is easy to understand how to set the important area in order to obtain the search result required by the user. Can be set correctly, so that a target image can be detected in a short time.

In the re-search, not only the area to be emphasized is set as described above, but also the way of setting the divided area, that is, in the above example, the horizontal division is performed. Another division method such as radial division may be designated.

CPU 12 for executing the above processing
As a processing function, the flowchart shown in FIG. 5 is executed. That is, first, a key image is selected (ST1). For this selection, for example, when a sample image for search is stored in the external storage device 14, it is called and designated from the sample list. Further, the image data prepared by the user may be input, or a rough sketch created on the monitor 3 by the user using the mouse 2 may be used.

Next, based on the given key image, an image similar to the key image is searched (ST2). In this search processing, the feature amount of the key image is obtained, and the similarity with the feature amount of each image stored in the image database is obtained, and an image having a high similarity is determined as a detection candidate image. The processing is specifically executed according to the flowcharts shown in FIGS.

That is, first, the key image is divided into a plurality of areas (ST10). This dividing method is performed according to a template prepared in advance. Then, in the present embodiment, a plurality of templates for specifying the dividing method are prepared, and divided for each template. Then, an average color as a feature amount is obtained for each of the divided regions (ST11).

Next, the variance based on the average color thus obtained is obtained for each template (ST12), and a template to be used for a search is determined from the obtained variance (ST13). That is, in step 12, the average color variance of each area is obtained for each template, so that the template having the larger variance is used. Then, the average color of each area divided according to the determined template is output as the feature amount of the key image, that is, stored in the temporary storage area (ST14).

On the other hand, the most suitable use template for the key image determined in step 13 is read (ST15), and the image is read from the image database in the external storage device 14 (ST16). Then, the read image (image to be processed) is divided into regions based on the used template, and an average color for each divided region is obtained (ST1).
7).

Next, the color difference between the key image and the corresponding area to be processed is determined as the Euclidean distance in the RGB color space (ST18). The closer the distance is, the higher the similarity is. As a result, the degree of similarity for each region between the key image and the processing target image is obtained. Then, the overall similarity is obtained from the similarity of each area. For example, since the distance for each area is obtained, the average similarity (average distance) of all the areas can be calculated to obtain the overall similarity. Then, the above-described processing is performed on all the images stored in the image database.

If the distances have been obtained for all the images (Yes in ST19), the overall similarity (average distance)
(ST20). As a result, the key images are ranked according to the degree of similarity between the images. In other words, the shorter the average distance, the higher the similarity, and the higher the target image candidate.

In this embodiment, the average of the distances of the respective regions is obtained, and the similarity (overall similarity) of the entire image is obtained.
Various methods used in the recognition algorithm can be used. Further, the feature amount is not limited to the one using the average color as in the present embodiment, and a graphic (shape) and other various types can be used. In short, any method can be used as long as similarity information is obtained for each area.

Next, the search result is displayed (ST3). That is, as shown in FIG. 8, the sorting result is read, and a candidate image is determined (ST30). This candidate image is a higher-ranked image of the sorting result, and images up to a predetermined N-th image may be used as candidate images (relative evaluation), or images having an overall similarity (average distance) equal to or more than a certain threshold value May be used as candidate images (absolute evaluation), or they may be mixed.

Then, the similarity (distance) of each region is read for the upper image serving as the candidate image (ST3).
1). A region with a low similarity (long distance) is a region that has not received much attention in the determination, and is a region to which the mask M is applied. On the other hand, regions with high similarity (short distance)
The region becomes the region of interest, and is a region where the frame W is set. Note that, in FIGS.
Although it was a type, it may be divided into three or more types. For example,
A region to which nothing is added may be provided for a region having a similarity between the region to be masked and the region to be framed.

Then, the upper image is displayed on the monitor 3 (ST32). At this time, based on the information read in step 31, the mask M and the frame W are displayed in an overlapping manner. This display example is the search result shown in FIGS. In the illustrated example, the hatched mask M is used. However, the display mode is not limited to this, and the corresponding portion can be inverted,
Various expression methods such as displaying in a light color can be adopted. Similarly, a region with a high degree of similarity can be made to cooperate with various other means, such as blinking, instead of being surrounded by the frame W.

The user who sees the search result displayed on the monitor 3 judges whether or not there is a target image, and inputs the judgment result using the mouse 2 or the like. In other words, although not shown, for example, by clicking on a predetermined area displayed on the monitor 3, a determination result can be input.

Therefore, the CPU 12 waits for the input,
The presence or absence of the target image is determined (ST4). If there is the target image, the search processing ends. On the other hand, if there is no target image, a re-search process is performed. At this time, an attention area in the key image is obtained (ST
5).

That is, since the user operates the mouse 2 to specify a region of interest in the key image, the user recognizes the specified region. Then, the process returns to the search process in step 2 and the search is performed again. At this time, the search is performed with emphasis on the attention area recognized in step 5. Note that the search with emphasis is to find the distance between the key image and the image to be processed only for the region of interest, and to find the distance for all the regions. There are various methods such as increasing the weight for
Thereafter, the target image can be detected by sequentially and repeatedly executing the above processing.

FIGS. 9 and 10 show a second embodiment of the present invention. In the present embodiment, an incremental search is performed. That is, the hardware configuration is basically the same as that of the first embodiment shown in FIG. 2, and the function of the CPU 12 is different. That is, a mouse or other device capable of creating a line drawing is used as an input device. Then, the CPU 12 functions to acquire the input line drawing information being created, perform an image search based on the information, and determine and output a candidate image. Specifically, a flowchart shown in FIG. 9 is implemented.

That is, first, input of a line drawing is waited (ST
40). Then, the non-drawing time is measured, and it is determined whether or not there is no drawing for a predetermined time (for example, 1 second) (ST4).
1). If the no drawing state does not continue for a certain period of time,
In step 40, the input line drawing information is continuously obtained.

If the non-drawing state continues for one second or more, the process proceeds to step 42, where a key image is created based on the input up to the present time. Then, the key image and each image stored in the image database are subjected to image search processing by line drawing comparison, and the similarity between the key image and each image is obtained (ST4).
3). Then, the upper image having the higher similarity is displayed on the monitor 3 (ST44). At this time, various methods such as displaying only the one with the highest similarity or displaying the top N items may be used.

Next, the user determines whether or not there is a target image, and inputs the determination result using the mouse 2 or the like. The CPU 12 waits for the input and determines whether or not the target image exists. Yes (ST45). If there is the target image, the search processing ends. On the other hand, if there is no target image, the process returns to step 40, and waits for subsequently input line drawing information.

The determination of the presence or absence of the target image is as follows.
When the line drawing information is input, it may be determined that there is no target image. In other words, subsequent input of line drawing information means that a search condition has been added (key image has been changed), so that a re-search request (no target image) can be taken.

As described above, in the case of the search based on the rough sketch, the search is performed in a state where a certain amount of line drawing has been drawn, and when the search result is poor, the result (displayed candidate image) is checked and necessary conditions are determined. By inputting as a line drawing, the search can be continued in an interactive manner and the target image can be found quickly. Further, by inputting an image for each part, it is easy to convey the intention and purpose of the user to the apparatus side. The image once displayed as a candidate image may be excluded from the subsequent search processing.

An example of the operation of this embodiment will be described.
As shown in FIG. The left side in the figure is a rough sketch (key image) created by the user by operating the input device, and is a diagram when a non-drawing state continues for one second or more. The right side in the figure is the candidate image with the highest similarity retrieved based on the key image.

As shown in the drawing, in the first case, the drawing is interrupted while the outline of the cloud existing in the target image is drawn. Along with this, a similarity search using a line drawing is performed, each image stored in the image database is matched with a line drawing element, and the image having the highest similarity as a result of the matching is displayed as a candidate image. In the illustrated example, a landscape with clouds in the sky is searched.

Since this image is not the target image, the user subsequently adds a diagram of the missing part. In the example shown,
A horizontal line is written (second), and then a shoreline is written (third). As the number of line drawing elements input as a rough sketch increases, the selected candidate image (right side in the figure) also changes. And in this example, fourth,
By writing a line drawing of the appearance of surfing on the left side of the figure, the target image could be searched. According to this example, the palm tree or bottle present in the target image is not described,
The target image could be searched without detailed description such as wave shape.

In the above description of the embodiment, an example is shown in which a key image is a line drawing and matching is performed with a line drawing element of an image in the image database. However, the present invention is not limited to this. In addition, color information can be added to the key image by filling the area, and the image search can be performed more accurately and in a smaller number of times by matching the line drawing and the color. It will be easier.

As a method of displaying the searched candidate images, the key image and the candidate image may be displayed side by side, or both images may be displayed in a superimposed manner. However, when overlapping, input image (key image)
Can be distinguished from the search result (candidate image) (the input image is described in red or the like). In the case of input using a line, an input having a high affinity can be realized even if the input is made by overlapping.

[0055]

As described above, according to the present invention, the user can understand the viewpoint at the time of searching on the device (system) side, such as what part of the key image the image searching device has searched for. It will be easier. In addition, by communicating the user's intention and purpose to the apparatus side or by continuing the search interactively, the desired image can be obtained efficiently.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a conventional example.

FIG. 2 is a hardware configuration diagram showing a preferred first embodiment of the present invention.

FIG. 3 is a diagram illustrating the present embodiment.

FIG. 4 is a diagram illustrating the present embodiment.

FIG. 5 is a flowchart illustrating functions of a CPU.

FIG. 6 is a part of a flowchart illustrating a detailed processing procedure of a search processing step (ST2) in the flowchart illustrated in FIG. 5;

7 is a part of a flowchart illustrating a detailed processing procedure of a search processing step (ST2) in the flowchart illustrated in FIG. 5;

FIG. 8 is a flowchart illustrating a detailed processing procedure of a search result display processing step (ST3).

FIG. 9 is a flowchart illustrating functions of a CPU according to the second embodiment of the present invention.

FIG. 10 is a diagram illustrating a second embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Computer main body 2 Mouse 3 Monitor 11 Input part 12 CPU 13 Memory 14 External storage device 15 Output part

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification code FI theme coat ゛ (reference) G06T 7/00 300 G06T 7/00 300F F term (reference) 5B050 AA09 BA15 CA07 EA19 FA02 FA17 GA08 5B075 ND06 NK06 NK08 NK39 PQ02 PQ22 PQ46 PQ69 PQ74 QM08 5L096 AA02 AA06 BA08 FA32 FA33 GA19 JA03 JA11 KA11

Claims (5)

[Claims] An image database storing a plurality of images and an image similar to a given key image are searched from the image database, and the searched images are output as candidate images to a display device. An image search apparatus, comprising: a processing unit, wherein the processing unit has a function of displaying at least one of a high similarity portion and a low similarity portion in the candidate image on the display device. 2. The method according to claim 1, wherein the processing unit divides the key image and the image into a plurality of identical regions, obtains a characteristic amount for each of the divided regions, and calculates a characteristic amount difference between corresponding regions. The method according to claim 1, wherein a similarity between the images is obtained based on the divided regions, wherein a portion having a high similarity or a portion having a low similarity in the candidate image is a unit. Image search device. 3. An image processing apparatus according to claim 1, further comprising: an area setting unit configured to set an area to be focused on at the time of the search with respect to the candidate image displayed on the display device, wherein the processing unit performs the search with emphasis on the set area. 3. The image retrieval device according to claim 1, wherein 4. An image database in which a plurality of images are stored, wherein image information being created is acquired, and when a pause time of drawing becomes equal to or more than a predetermined value, image information input up to that time is obtained. An image search device comprising a key image, processing means for searching an image database for an image similar to the key image, and outputting the searched image as a candidate image to a display device. 5. The image search device according to claim 4, wherein the key image and the candidate image are displayed on the display device in a superimposed manner.